The internet has largely weighed in on Popular Science‘s sudden decision to shut off its comments section earlier this week. The responses have been, for the most part, positive, as battle-weary writers leap to the defense of a publication looking to preserve the integrity of its work from biased trolls. (Except for Mathew Ingram, who devoted a PaidContent post and an extended Twitter beat-down to the new policy.)
But PopSci is justifying its new policy by arguing that silencing commenters preserves the integrity of science, as evidenced by a study published earlier this year in the Journal of Computer-Mediated Communication, part of a string of commenting-centric research. In the study, a team of researchers had a sample of just over 1,100 subjects read a blog post on nanotechnology, followed by a set of comments that were alternately laudatory or incendiary. The readers’ opinions of the merits of the technology, PopSci reports, were strongly affected by the negative comments, a result PopSci editor Suzanne LaBarre deems dangerous: “If you carry out those results to their logical end–commenters shape public opinion; public opinion shapes public policy; public policy shapes how and whether and what research gets funded–you start to see why we feel compelled to hit the ‘off’ switch.”
But the study doesn’t prove or demonstrate that comments have nearly the scope of influence that LaBarre attributes to them. The problem with her extrapolation of the study’s logic was summed up by Nature editor Noah Gray:
@fmanjoo @mathewi @jaredbkeller To me, it's ironic that they're using a narrow reading of science to save the public's perception of science
— Noah Gray (@noahWG) September 24, 2013
In a blog post, Marie-Claire Shanahan, the chairwoman of science education at the University of Calgary, agreed, commenting on the “large gap” between how PopSci characterizes the study results versus what the results actually say. Eighty-three percent of the influence of the piece was based on factors that had nothing to do with the comments, explains Shanahan, and the fact that “the civility of the comments had “NO SIGNIFICANT DIRECT EFFECT on readers’ perceptions of nanotechnology,” a fact left conveniently out of LaBarre’s analysis of the study.
LaBarre also failed to communicate the study’s main finding: That the comments had the most sway over people who were strongly pro or against uses of nanotechnology–not undecided readers. “So among those who already held strong views, the uncivil comments tended to polarize them a bit further,” writes Shanahan.
There’s also research suggesting that lively commenting can positively add to the debate, says Shanahan, citing her own research surveying comments left on health stories in The Globe and Mail. After Shanahan and a grad student completed the study they found–surprise–that the comments section did exactly what it was supposed to. “Extensive contributions were made by parents, patients and people with medical expertise. Questions were asked and clear thoughtful answers were often given,” she wrote. (PopSci doesn’t include Shanahan’s research in its policy defense, and as any competent science writer understands, two pieces of research does not consensus make.)
We’ve written often about the dangers of adding false balance to stories by lending uninformed sources equal weight when the research community has, more or less, reached consensus. Though the comments section doesn’t bear the same weight as what a writer includes in a piece, there’s reason to speculate that the comments section might have more influence on readers when they examine more familiar topics.
In January, Mother Jones published a more thorough post on the paper cited by PopSci, musing that the already fraught emotions of climate-change skeptics might exacerbate the effects of comments:
As the researchers note in the paper, compared with climate change, relatively few people know much about or have strong feelings about nanotechnology. When it comes to climate change, in contrast, “the controversy that you see in comments falls on more fertile ground, and resonates more with an established set of values that the reader may bring to the table,” explains study co-author Dietram Scheufele, a professor of science communication at the University of Wisconsin-Madison. If commenters have stronger emotions and more of a stake, it stands to reason that the polarizing effect of their insults may be even stronger–although, to be sure, this needs to be studied.
As MoJo points out, the verdict’s still out. Over at Slate, technology writer Will Oremus has some acrimonious words for PopSci’s argument that removing comments is necessary to preserve the purity of reasoned science, a move he points out is “antithetical to the spirit of free inquiry that has always driven scientific discovery.”
It’s a solid point that scientific discovery is based in this kind of collective accumulation of knowledge, with the assumption that pseudoscience will be filtered out through the hive mind. And there are ways of improving the baseline of opinion: Many companies, like the notoriously troll-heavy YouTube, are counting on the idea that lessening the anonymity of their comments by linking users to their social media profiles will create accountability and less bad content, instead of doing away with the section entirely.
And as the debate surrounding PopSci’s new policy has demonstrated, one publication’s interpretation of a study doesn’t necessarily convey the most accurate information. The internet picked up PopSci’s slack in probing the nuances of the paper in bits and pieces throughout the Web and on Twitter.
Too bad there isn’t a place to gather all of this analysis in one location, proximate to the article in question. Oh, wait…
Alexis Sobel Fitts is a senior writer at CJR. Follow her on Twitter at @fittsofalexis.