Others are frustrated by the premium journalists place on appealing to a mass audience. Duncan Watts, a Columbia University sociologist, designed an experiment for Primetime to test Milgram’s small-world theory—commonly known as “six degrees of separation”—that people divided by great social or geographical distance are actually connected by a relatively small number of links. In the experiment, two white Manhattan residents competed to connect with a black boxer from the Bedford-Stuyvesant neighborhood of Brooklyn using the fewest links, then the boxer had to connect with a Broadway dancer. All three connections were made using at most six links. Watts says that after the segment aired in late 2006, he received an e-mail from its producer, Thomas Berman, saying that its ratings had been poor. (An ABC spokeswoman insists that the network was satisfied with the ratings.) “One of the limitations of this model is that it’s crowd-driven, it’s about entertainment,” says Watts. “It’s a bit of a Faustian bargain.”

Another quibble that some social psychologists have with these journalistic experiments is the use of the word “experiment” to describe them in the first place. To a dyed-in-the-wool researcher, an experiment involves comparing a control group with an experimental one, in which a single condition has been varied so that any changes in the outcome can be clearly attributed. Practically no journalistic “experiment” meets this standard, but many golden-age experiments didn’t either, strictly speaking. In addition, practically every journalistic experiment includes a disclaimer that its results are decidedly unscientific.

Wendell Jamieson, city editor at The New York Times who assigned the subway-experiment story, chafes at calling the exercise an “experiment,” pointing out that it was conducted in connection with another article about the original experiment. “It’s just a fun way to take a different approach to a story,” Jamieson says, comparing it to when he was at the New York Daily News and sent a reporter to Yankee Stadium during a subway series dressed in Mets regalia. “It’s tabloid trick two-hundred and fifty-two.” Bill Wasik, the Harper’s editor who started flash mobs, points out that using the word “experiment” is a way for journalists to appropriate the “alpha position” of science, lending their endeavors a sort of added legitimacy. “The piece is wearing a lab coat,” Wasik says of his own article, which repeatedly describes flash mobs as an experiment, “but it’s not entirely scientific by any means.”

Perhaps no media outlet has tried harder to achieve uniformity in conducting its experiments than Reader’s Digest. Detailed instructions for how to conduct its “studies” are distributed to researchers in more than thirty cities around the world to ensure that their results will be comparable. For the courtesy tests, researchers were told how long dropped papers were to be left on the ground, how far to walk behind people entering buildings to see whether they would hold the door, and what sort of demeanor to adopt when speaking with clerks who were being tested to see whether they would say “Thank you.” Nonetheless, despite all the careful planning, New York City’s courtesy title may need to be affixed with an asterisk. Robert Levine, a social psychologist at California State University, Fresno, did a series of helpfulness experiments in the early nineties in which New York City placed dead last out of thirty-six United States cities. While this doesn’t necessarily contradict the Reader’s Digest result, in which New York was the only U.S. city tested among a global selection of cities, Levine points out that all the Reader’s Digest New York tests were carried out at Starbucks, yielding a potentially skewed sample. What if Starbucks employees and customers are simply more courteous than New Yorkers as a whole? “I’m not saying they screwed up,” says Levine, “but that was certainly a flag that was raised for me.”

So maybe journalists can and should be more careful in how they design experiments, but that debate, in many ways, is beside the point. The best examples of the genre are undeniably good journalism, and the lesser lights, for the most part, amount to innocuous entertainment. Indeed, my hope is that some enterprising reporter is even now hatching a plan to find out whether Joshua Bell really would draw such a big crowd outdoors on a sunny day in D.C. 

If you'd like to help CJR and win a chance at one of 10 free print subscriptions, take a brief survey for us here.

 

Daniel Weiss is a freelance writer based in New York City.