Still, their limitations are legion. First, only a fraction of a district’s teachers are included—only those who teach reading and math; there are no standardized tests for other subjects. No allowance is made for many “inside school” factors, such as the effect of team teaching, after-school tutors, substitute teachers, a child or a teacher who is absent for long periods of time, or an unstable school environment—a new principal, a violent incident, a district overhaul. And finally, critics ask: Since the number is based on manipulating one-day snapshot tests—the value of which is a matter of debate—what does it really measure?
Research experts weighed in from all directions in response to the Los Angeles Times project, some saying value-added rankings may be flawed, but they are better than nothing; others said the numbers are only reliable enough to evaluate schools, not teachers. In February, two University of Colorado, Boulder researchers caused a dustup when they called the Times’s data “demonstrably inadequate.” After running the same data through their own methodology, controlling for added factors such as school demographics, the researchers found about half the reading teachers’ scores changed. On the extreme ends, about 8 percent were bumped from ineffective to effective, and 12 percent bumped the other way. To the researchers, the added factors were reasonable, and the fact that they changed the results so dramatically demonstrated the fragility of the value-added method. Times editor Russ Stanton disputed the claim that the Colorado study discredited his paper’s effort, saying it “shows only that their analysis, using somewhat different data and assumptions than we used, produced results somewhat different than our own.” Thrusts and parries like these were not likely to help parents and readers make sense of it all.
But here is perhaps the most telling observation: nearly every economist who weighed in agreed that districts should not use these indicators to make high-stakes decisions, like whether to fire teachers or add bonuses to paychecks. The numbers, they said, can’t carry that kind of weight. By last summer, it should be noted, Michelle Rhee had already fired twenty-six DC teachers based in large part on low value-added scores. And New York City wants principals to use them immediately for tenure decisions.
The first installment of the Los Angeles Times’s series featured a photo of a fifth-grade math teacher named John Smith, chalk in hand, standing before his class of mostly Latino students. One child is raising his hand, expectantly. The caption told the story: “Over seven years, John Smith’s fifth graders have started out slightly ahead of those just down the hall but by year’s end have been far behind.”
Smith’s silver hair and stern expression became the poster image for the series. The sixty-three-year-old was meant to illustrate the potential of these numbers to expose some teachers as failures (and others as triumphs), offering parents knowledge they never before had. As for the schools, some questions remained unanswered. Did these calculations add to what was already known? Hard to tell; the principal had refused to talk. In the end, readers only really knew what the numbers revealed—Mr. Smith’s students slide 14 percentage points in math during the school year on average, compared to their peers across the city.
President Obama’s education secretary, Arne Duncan, weighed in the following day endorsing the newspaper’s move, later calling on all school districts to consider making such data public. Surprisingly, the Los Angeles Times’s own editorial page took a less hawkish tone, criticizing federal policies for pushing these numbers too far.
Policy aside, has the series helped LA’s parents? Principals worried that hundreds of them might demand to transfer their children into the top-rated teachers’ classrooms. That didn’t happen. So far, said Felch, it has been mostly non-minority, middle-class parents, already engaged in the schools, who have contacted him with comments—some grateful, some skeptical. It seems that the stories have yet to reach many low-income parents. One reason for that may have to do with language. It took about a month for the Times to translate the series into Spanish, the language used by many parents of the district’s majority Hispanic population.