Getting Education Data Right: The Case of High School Admissions

With schools across the country struggling to provide virtual and limited classroom instruction because of the coronavirus, the issue of middle and high school admissions for next fall in many cities constitutes another formidable challenge. In no city is this challenge greater than in New York, home to the largest public school system in the country by far. 

In December, Mayor Bill de Blasio put to rest much confusion by explaining modifications to the calendar for high school admissions and allayed mounting concerns about segregation at middle schools by announcing the termination, for at least one year, of all admissions based on grades and test scores. The New York Times lauded the mayor in an editorial for his effort to boost the integration of middle schools but criticized him for failing to address segregation in the city’s top high schools.

Although the calendar is now at least clear, anxiety among eighth-graders and their parents persists about the selectivity of the city’s screened high schools, in no small part because of repeated misreporting by the Times and others. The paper’s coverage—exemplified by a 2017 piece headlined “Couldn’t Get into Yale? 10 New York City High Schools Are More Selective”—has even been blamed for fueling the segregation by discouraging students from underrepresented neighborhoods from applying to many screened high schools on the grounds that admission seems nearly impossible. But the tendency to exaggerate the selectivity of certain schools is ubiquitous in media, ensnaring not just the Times but also The New York Post and, to a lesser degree, The Wall Street Journal, The Philadelphia Inquirer, The Washington Post, and The Times-Picayune 

The failure to get the admissions data right constitutes a significant reporting problem. The city’s screened high schools are nowhere nearly as selective as the press has claimed, part of a wider problem of journalists placing unqualified faith in data issued by government agencies, think tanks, and intergovernmental organizations. The result often amounts to a dramatic story but one that can be quite misleading.

The trouble with the story about high school admissions begins with official data. The admissions numbers in the annual high school directories published by New York City’s Department of Education are indeed alarming. Eight consecutive schools in the 2019 directory, for example, exhibited daunting odds: Bard High School Early College, 30 applicants per seat; Baruch College Campus High School, 44; Beacon High School, 19; Business of Sports School (BOSS), 13; Central Park East High School, 37; Chelsea Career and Technical Education High School, 14; City College Academy of the Arts, 22; and The Clinton School, 21. These odds translate into acceptance rates ranging from 2.3 percent, in the case of Baruch, to 7.7 percent, in the case of BOSS. 

But these students are not applicants in the conventional sense. They are students who rank a school by order of preference as one of up to 12 with which they would like to match. This process—introduced in 2004 and derived from the National Resident Matching Program for doctors introduced in 1952—employs an algorithm allowing only one match. Accordingly, if every eighth-grader in New York City exercised his or her right to list 12 schools, each school, on average, could in turn accept only one of 12 students, or 8.3 percent of applicants.

Sign up for CJR's daily email

While the Times published a detailed article about how the algorithm works in 2003 and another in 2014, the newspaper has nevertheless year after year used the data published in the directories to describe the selectivity of schools.

I began encountering this reporting problem in 2005, when the Times published an article on then-Mayor Michael Bloomberg’s plans to create several new high schools to address the surplus demand for seats in exam and screened schools. The Times reported that Beacon had 6,000 applicants for 250 seats the previous year, meaning an acceptance rate of 4.2 percent.

As a teacher at Beacon at the time, I knew the admissions process from the inside and emailed a correction to the paper: 6,000 students ranked Beacon as one of up to 12 schools in which they were interested; about 1,800 students submitted the requisite portfolio of their best work and visited the school for the mandated interview; and approximately 500 offers were made to fill 250 seats. This meant an acceptance rate of about 28 percent if all 1,800 applicants ranked Beacon first, which is highly improbable, given that approximately 50 percent of applicants to Beacon today who fulfill application requirements rank the school first. But that correction went nowhere, and I resigned myself to explaining the numbers to anxious parents fretting that their children had no chance of getting into Beacon given what they had read in the Times.

Following the 2017 article about 10 of the city’s high schools being more selective than Yale, I wrote a letter to the Times. As that letter went unacknowledged and as the newspaper did not run another letter to elucidate the process, I published a critique on the Web site of a research center I run at Teachers College, Columbia University. That critique led to an article published by Chalkbeat and another by The Grade, hosted by Phi Delta Kappan, which interviewed Alvin Roth, a professor at Stanford who shared the Nobel Prize in economics in 2012 for work decades earlier on market design and who, with two other economists, Atila Abdulkadiroglu and Parag Pathak, developed the algorithm used by the DOE. Roth explained that the Times had indeed greatly exaggerated the number of applicants because the algorithm pulled students from the applicant pool once they were matched. “If I applied to you as my seventh choice, and I got accepted by my first choice, I wasn’t rejected by you,” Roth said. “You never saw me.”

With a matching algorithm, the closest one can truly get to an acceptance rate is a match rate through adding the number of students who matched with a particular school to the number of students who matched with a school they ranked lower than that school and then dividing the number of matches by that sum. While these numbers are not available in the city’s high school directory, they may be obtained through a research request with the city’s DOE or Independent Budget Office. For example, in the case of Goldstein High School, which, unlike Beacon, bases admissions solely on grades, test scores, and attendance, 4,947 students ranked the school in 2016, 512 received offers to fill 250 seats, and 2,262 matched with a school they ranked lower. Accordingly, 18 percent (512/2,774) got matched, rather than 5 percent (250/4,947) accepted, as claimed by the Times in its 2017 article. If the DOE employed a conventional process, the denominator would have been 4,947, but the numerator could have far exceeded 512 because many accepted students would have likely had several offers from other schools.

In November 2019, I got a call from the head of admissions at Beacon, a history teacher named Bayard Faithfull who had been at the school since its founding in 1993 and was equally frustrated with how the Times told the admissions story. Faithfull said he had learned that the Times was about to publish another article comparing Beacon’s selectivity to Yale’s and wanted to know if I could email someone at the Times to explain the numbers again.

“It takes years to build trust with guidance counselors so students apply,” Faithfull said. “But that’s been hard to do with The New York Times making the odds appear so bad.” Faithfull explained that he had been doing outreach at middle schools in the Bronx for years and had to repeatedly extinguish fears expressed by guidance counselors that Beacon was out of reach.

Olga Ramos, the director of admissions for Bard High School Early College in Queens, another screened school, echoed Faithfull. “Over and over,” she said, “I’ve heard from qualified students interested in Bard, ‘I spoke to my guidance counselor, and she said Bard’s too hard to get into.’” 

In fact, these observations comport with the findings of a 2016 IBO report documenting “a striking difference in the choices made by high-achieving students applying from low-performing middle schools … from the choices of high-achieving students from high-performing middle schools.” The former lag significantly behind the latter in applying to selective high schools.

At Faithfull’s request, I emailed a contact at the Times to relay his concerns and urged this person to have the reporter for the story call Faithfull or me. Neither of us heard anything. The story, published a week after Faithfull called me, focused on the popularity of Beacon and the long lines for the school’s two open houses. The reporter claimed that with 5,800 applicants for 360 seats the previous year, “Beacon’s admissions rate is roughly akin to Yale’s.”

Soon after, the Times gave this myth even more life in republishing these numbers for Beacon in a story about a walkout at the school to protest insufficient diversity and in a story several months later about pressure on then-Chancellor Richard Carranza to discard the screened process of admissions.

When asked for this article why the Times persisted in describing the application numbers in this manner, a spokesperson replied, “The Times stands by its use of Department of Education data on high school admissions.” Despite repeated efforts seeking comment from someone at the DOE about why the Office of Student Enrollment continued to present application numbers in its annual high school directory in the same manner, I got no response.

Roth, the Nobel Prize winner from Stanford, sees the problem as twofold. Reporters have  been insufficiently inquisitive, he said in an interview. “Good journalism,” he said, “requires interpretation of data, not its repetition.” More fundamentally, Roth said, there is so much turnover in the central offices of big city school systems that people in charge of enrollment often do not know how to present the data correctly.

In addition, our obsession with status—epitomized by the popularity of the U.S. News & World Report college rankings, initiated in 1983—appears to have bred forced conceptions of selectivity. Whether from fear of falling or fixation on distinction or some combination, a need to stratify seems widespread.

What is nevertheless certain is that the algorithm developed by Roth with Abdulkadiroglu and Pathak has significantly streamlined the enrollment process in New York. The three economists developed the algorithm, they wrote in a 2005 article published in the American Economic Review, to “relieve the congestion of the previous offer/acceptance/wait-list process” that conferred “some students multiple offers” and “multiple students … no offers.”

The algorithm indeed relieved such congestion. Before its introduction, year after year, 30,000 of 80,000 middle school graduates ended up at high schools they had not chosen. Of students today who rank 12 schools, 97 percent get an offer from one of their choices, according to the city’s 2019 directory. Furthermore, nearly half get their first choice while approximately three-fourths get one of their top three choices. Last year, for example, according to an article published by Chalkbeat, 46 percent matched with their first choice while 73 percent matched with one of their top three choices.

That the algorithm works so well should be the story, not the challenge of getting a spot at a particular school.

ICYMI: Weighing different paths to funding local news

This story has been updated to include a more complete statement from The New York Times.

Has America ever needed a media watchdog more than now? Help us by joining CJR today.

Samuel E. Abrams is the director of the National Center for the Study of Privatization in Education at Teachers College, Columbia University, and the author of Education and the Commercial Mindset. He was previously a high school history teacher for 18 years.