First there was the wild speculation about the discovery of extraterrestrial life. Then came widespread, sometimes misguided, coverage of the real news: discovery of a bacterium than can substitute arsenic for phosphorus, one six elements considered essential for life (which may, perhaps, expand the scope of humanity’s search for life beyond this planet).
Now comes the third installment in the commotion-filled saga: widespread criticism of the paper detailing the discovery, published last Thursday in Science, and an apparent snubbing of the media by the paper’s authors and NASA (which helped fund the research), who rebuffed journalists’ requests for a response to the criticism.
The latest round of controversy began on Saturday when University of British Columbia microbiologist Rosie Redfield penned a scathing critique of the paper, “A Bacterium That Can Grow by Using Arsenic Instead of Phosphorous,” on her personal blog. One of the key findings of the study was that the bacterium, discovered in Mono Lake and cultured in a lab, incorporated arsenic into its DNA. But Redfield argued that the paper didn’t “present ANY convincing evidence” that that actually happened (for DNA or any of the bacterium’s other biological molecules).
Redfield criticized a number of the research team’s methodologies. For instance, she argued that scientists did not correctly purify the DNA in order to remove any arsenic that might simply have been sticking to the DNA. She also said that the bacterium was grown in an environment with enough phosphorous (even at such a depleted level) to account for the observed bacterial growth, which was attributed to arsenic. The “bottom line,” Redfield wrote, was “lots of flim-flam, but very little reliable information.”
Redfield’s post immediately caught on in the blogosphere. Science reporter David Dobbs highlighted it on his Wired Science blog, Neuron Culture, observing that although he wasn’t familiar with Redfield, “her opinion was quickly seconded in the blog’s comments and on Twitter by many sharp scientists.”
Things got even more interesting when NASA rebuffed the Canadian Broadcasting Corporation after it sought comment on Redfield’s critique. According to its article, published Monday evening:
When NASA spokesman Dwayne Brown was asked about public criticisms of the paper in the blogosphere, he noted that the article was peer-reviewed and published in one of the most prestigious scientific journals. He added that [Felisa] Wolfe-Simon [the paper’s lead author] will not be responding to individual criticisms, as the agency doesn’t feel it is appropriate to debate the science using the media and bloggers. Instead, it believes that should be done in scientific publications.
“Any discourse will have to be peer-reviewed in the same manner as our paper was, and go through a vetting process so that all discussion is properly moderated,” wrote Felisa Wolfe-Simon of the NASA Astrobiology Institute. “The items you are presenting do not represent the proper way to engage in a scientific discourse and we will not respond in this manner.”
[Update, Dec. 8: Despite her initial reluctance to respond to the criticism, Wolfe-Simon posted the following comment on her blog on Wednesday:
One can't really blame the science community to be reluctant to dive into popular controversy on a blog or in a newspapers. Journalists and bloggers are often given to bloviate on just about any subject, whether or not they have any knowledge or expertise. Journalists are notorious for running with sensational stories that misrepresent the research. After all, they are the ones that ran with the extra-sensational alien stories. That doesn't bode well for their ability to handle the story any further.
And then you've got the rightwing loonies, who automatically became PhDs in climate science, epidemiology, social science, and psychology whenever it suits the agenda, whether or not they have a high school diploma, to the point of having one's emails and correspondence stolen, tapes of one's lectures selectively edited, and so on, in order to have one's entire career and character slandered and impugned.
If the research is faulty, then let qualified scientists try to replicate the experiment and write about their results. Either they can replicate it or not. It's fine to criticize the methodology and the interpretation of results -- that is what science is, after all -- but one can't really blame responsible sponsoring organizations for holding out for additional research, rather than diving into a comment stream on some blog to respond to criticism.
#1 Posted by James, CJR on Tue 7 Dec 2010 at 08:25 PM
I think this is a really interesting look at the limbo-position scholarly communication is in this days: The scholarly journal is being challenged not only by open access, but by experts in the field writing on their own, and the academy still isn't certain how to deal with new outlets.
On the one hand, Redfield has a very important point, that she is an expert in her field (she wears the robes) and it shouldn't matter in what forum or place she speaks. Whether her criticism of the experiment is published in an established journal or on her own site, it's the same criticism. However, the traditional scientific process does demand the experiment be replicated before any true rebuttal of the findings can be made.
James, you talk about blogs without seeming to make the distinction between research blogs, written by people in the field who are part of the academy, and journalists' blogs or amateurs' blogs, which are not. The distinction is important, I think, and will be a part of how scholarly communication is redefined from this time onward.
#2 Posted by laura k, CJR on Wed 8 Dec 2010 at 10:36 AM
Mankind as a whole is responsible for the directions of technologies. Love for war will help war machines and genuine desire for peace will help future of life on the universe.
#3 Posted by M. shahjahan Bhatti, CJR on Wed 8 Dec 2010 at 11:23 AM
@laura k:
I take your point about distinguishing between research blogs and others. Redfield's criticisms are interesting, but ultimately someone will need to try to replicate the findings, or not. I'm just a little cautious about generating popular controversy prematurely; it can seem like unsavory headline-mongering, and can be misused by "scientists" with no expertise in the particular field. The pitfalls of that kind of blog-driven controversy are fraught.
It isn't fair to try to discredit someone's work to the public without having to make a rigorous case for it, and gaining at least some support for one's doubt among one's colleagues.
Remember how the Hopkins study on Iraq was "discredited" by some off-the-wall physicist in Britain, as if he knew anything about population research. The problem is that journos and bloggers generally cannot distinguish between quacks and real scientists, and are apt to pick up and sensationalize a bogus story. I'm not saying that Redfield's criticism is bogus, just that I'm not persuaded that it is the most appropriate venue for scientific criticism, if one wants to advance the science. If one wants to advance one's name recognition, and that's a legitimate aim, then that's another matter.
Cheers,
James
#4 Posted by James, CJR on Wed 8 Dec 2010 at 12:58 PM
The bottom line here is that this paper from NASA makes an extraordinary claim, a claim was made with great fanfare in a press conference, and one that is poorly supported by their evidence. I would have expected rock solid evidence for an arsenate diester within DNA. Period. You don't go to press until you have solid factual evidence for your science.
#5 Posted by Rob, CJR on Wed 8 Dec 2010 at 03:14 PM
I would like to pose a simple question for everyone here. Suppose this were a medical finding. And suppose NIH came out with a press release and said "A remarkable new study has major implications for preventing heart disease". We will hold a press conference about this tomorrow. And then they hold the press conference and NIH and the authors make grandiose claims about a new paper in a high profile medical journal about a relatively simple means to prevent heart disease. All the news stories report this great finding and then multiple people out there in the "blogosphere" start to point out flaws in the paper and also flaws in the statements made to the press. I think this would be a great service to the world - and to medicine and to science.
Certainly - one should make such critiques carefully and should avoid personal attacks on the people involved. But if the press coverage and the paper itself have potential flaws, I think it is imperative to point those out sooner rather than later. Though this study was not about heart disease, the claims made by the authors and NASA were very "big" - suggesting all textbooks should be rewritten and implying clearly that this should influence decisions about searching for life elsewhere in the universe. In other words, the claims had potential to influence some major decisions immediately. I believe the engagement of scientists in discussing the press coverage and paper was a good thing. Again, headline mongering and sensationalism happened on all sides here - and that is too bad - but this should not deter scientists from attempting to participate in the active, lively, ongoing discussion of science that is going on on the web.
#6 Posted by Jonathan Eisen, CJR on Fri 10 Dec 2010 at 12:34 PM
I just doubt that one can have much of a fruitful discussion on research methodology in the larger blogosphere. I'm uncomfortable with this:
On Monday, [science journalist Carl Zimmer] contacted a dozen experts for an analysis published by Slate. “Almost unanimously,” he found, “they think the NASA scientists have failed to make their case.”
First of all, is Slate magazine a good place to discredit the work of a microbiologist or other scientist? Slate, whose columnist Christopher Hitchens, of all people "discredited" the fine work of the Hopkins epidemiologists in the 2006 Iraq study, published in Lancet. Please read it here Epidemiology meets moral idiocy. - By Christopher Hitchens - Slate Magazine Of course, Hitchens knows, and knew, nothing whatsoever about the perfectly legitimate methodology of that study, which is widely used internationally to study prevalence in developing countries, nor did he have the knowledge to interpret the results -- had no clue what confidence intervals were, nor did he bother reviewing the definition of "excess death." But this didn't stop Mr. Hitchens, whose agenda was to discredit any and all critics of the Iraq War. And so he did. That's the kind of "science reporting" that you get in Slate magazine.
And here we have the same magazine, Slate, doing some kind of "analysis" by anonymous "experts" that purportedly discredits the NASA scientists by stating that they didn't "make their case." I make no judgment on whether their methodology was flawed, but I think part of the problem here was how the press misinterpreted and ran with the sensational distortion.
As an epidemiologist, the scenario you put forth on a possible "breakthrough" in heart disease happens all too frequently. JAMA and Lancet and NIH are not immune from publishing bad research, or more commonly weak research, and touting it in press releases -- far from it. Perhaps even more commonly, health journalists are wont to grab on to a published paper with preliminary research and oversell the results. Popular misconceptions about medical treatments and health conditions because of overselling by the authors or over-interpretation by health journalists are probably far, far more common in health research than in the field of microbiology.
It's fine to have a discussion in the blogosphere about faulty medical research and draw attention to some of the flaws, have a debate about the interpretation of the findings and so on. But ultimately, the better, more productive discussion, IMHO, occurs at conferences and in the peer reviewed journals. The science ultimately needs to be replicated and expanded and the scientist must go back and review the methods and the data, and at least in my field there are no harsher critics of sloppy research than one's colleagues.
The NASA scientists perhaps should give Ms. Redfield and others the chance to replicate the findings. The controversy stirring the the blogosphere generates a lot of heat, but very little light. That, to me, is the crux of the problem.
#7 Posted by James, CJR on Fri 10 Dec 2010 at 02:16 PM