Another quibble that some social psychologists have with these journalistic experiments is the use of the word “experiment” to describe them in the first place. To a dyed-in-the-wool researcher, an experiment involves comparing a control group with an experimental one, in which a single condition has been varied so that any changes in the outcome can be clearly attributed. Practically no journalistic “experiment” meets this standard, but many golden-age experiments didn’t either, strictly speaking. In addition, practically every journalistic experiment includes a disclaimer that its results are decidedly unscientific.
Wendell Jamieson, city editor at The New York Times who assigned the subway-experiment story, chafes at calling the exercise an “experiment,” pointing out that it was conducted in connection with another article about the original experiment. “It’s just a fun way to take a different approach to a story,” Jamieson says, comparing it to when he was at the New York Daily News and sent a reporter to Yankee Stadium during a subway series dressed in Mets regalia. “It’s tabloid trick two-hundred and fifty-two.” Bill Wasik, the Harper’s editor who started flash mobs, points out that using the word “experiment” is a way for journalists to appropriate the “alpha position” of science, lending their endeavors a sort of added legitimacy. “The piece is wearing a lab coat,” Wasik says of his own article, which repeatedly describes flash mobs as an experiment, “but it’s not entirely scientific by any means.”
Perhaps no media outlet has tried harder to achieve uniformity in conducting its experiments than Reader’s Digest. Detailed instructions for how to conduct its “studies” are distributed to researchers in more than thirty cities around the world to ensure that their results will be comparable. For the courtesy tests, researchers were told how long dropped papers were to be left on the ground, how far to walk behind people entering buildings to see whether they would hold the door, and what sort of demeanor to adopt when speaking with clerks who were being tested to see whether they would say “Thank you.” Nonetheless, despite all the careful planning, New York City’s courtesy title may need to be affixed with an asterisk. Robert Levine, a social psychologist at California State University, Fresno, did a series of helpfulness experiments in the early nineties in which New York City placed dead last out of thirty-six United States cities. While this doesn’t necessarily contradict the Reader’s Digest result, in which New York was the only U.S. city tested among a global selection of cities, Levine points out that all the Reader’s Digest New York tests were carried out at Starbucks, yielding a potentially skewed sample. What if Starbucks employees and customers are simply more courteous than New Yorkers as a whole? “I’m not saying they screwed up,” says Levine, “but that was certainly a flag that was raised for me.”
So maybe journalists can and should be more careful in how they design experiments, but that debate, in many ways, is beside the point. The best examples of the genre are undeniably good journalism, and the lesser lights, for the most part, amount to innocuous entertainment. Indeed, my hope is that some enterprising reporter is even now hatching a plan to find out whether Joshua Bell really would draw such a big crowd outdoors on a sunny day in D.C.