The cover story for CJR’s March/April issue—“Tested,” by LynNell Hancock—explores the nationwide effort to “reform” education, and what happens when reporters get their hands on data about local schools and teachers that can be both controversial and difficult to contextualize.
If you like that story, you might be interested in another excellent cover story, an article by Jessica Lussenhop in Minneapolis’ altweekly City Pages. Her feature, “Inside the Multimillion-Dollar Essay-Scoring business,” investigates another dark side of the education industry: the standardized test essay-scoring business. (Article recommended by the great longform journalism aggregation website Longreads.)
The standardized testing business is, as you’d imagine, a growing industry; Lussenhop reports that President Bush’s No Child Left Behind Act tripled the demand for testing companies, which can barely keep up with the flow. Hence, these companies hire more and more temporary workers, rush them through the system with less training, and encourage them to score faster and faster. One ex-scorer Lussenhop interviewed describes having to crank through 200 essays in a day; with only seconds to decide whether each one should be deemed “excellent” or “good” or “adequate,” how could the ranking process not become relatively arbitrary?
Lussenhop speaks to Dan DiMaggio, who worked for one such company for three years, and who learned from his first day what a slippery thing “standardization” actually is, when it came to grading essays via “rubric”:
DiMaggio’s question concerned an essay titled, “What’s your goal in life?” The answer for a surprising number of seventh-graders was to lift 200 pounds.
Although DiMaggio had been through a training process, he found himself tripped up as he began scoring the essays. What made the organization “good” as opposed to “excellent”? What happens when the kid doesn’t answer the question at all, but writes with excellent organization about whatever the hell he wants? Did it matter that it was insane for seventh-graders to think they’d be benching 200 pounds?
DiMaggio had good reason to worry. His score could determine whether the school was deemed adequate or failing—whether it received government funding or got shut down.
Many of the companies that dominate the industry—places like Scantron, and National Computer Systems—are located in the Twin Cities. (“From March through May, hundreds of thousands of standardized test essays will pour into the Twin Cities to be scored by summer,” Lussenhop writes.) But this story with a Minneapolis peg also obviously has huge national implications, as well:
Eventually, DiMaggio got used to not asking questions. He got used to skimming the essays as fast as possible, glancing over the responses for about two minutes apiece before clicking a score.
Every so often, though, his thoughts would drift to the school in Arkansas or Ohio or Pennsylvania. If they only knew what was going on behind the scenes.
“The legitimacy of testing is being taken for granted,” he says. “It’s a farce.”
Most shocking, perhaps, is the manipulation of the scores by the companies’ supervisors, seemingly at whim. Supervisors are implicitly encouraged to “fudge” the scores within their groups, in order to expedite consensus on essay scores—bell curves are optimal, and deviations bring demotions. But when a representative from one state’s Department of Education complains that too many children in her state are getting “1”s and “2”s, that, too, is just another anomaly to adjust for:
The scorers would not be going back to re-grade the hundreds of tests they’d already finished—there just wasn’t time. Instead, they were just going to give out more 3s.
No one objected—the customer was always right.
Read Lussenhop’s whole story—worth more than a scan—on the City Pages site, here.
Lauren Kirchner is a freelance writer covering digital security for CJR. Find her on Twitter at @lkirchner