Last month John Bohannon, a biologist and science writer, published the kind of piece in Science magazine that defies classification. One part sting operation, one part creative investigation, the article, “Who’s Afraid of Peer Review?” chronicled Bohannon’s attempts over the course of a year to factcheck the peer review process of a number of open-access scientific journals. Bohannon’s piece is by far the most ambitious attempt to check the rigor of peer review in open-access publishing, but perhaps because of the heated charge behind the topic, press coverage of his project was out of sync with the larger conversation happening in the ivory tower.
One of the most debated topics in science publishing, open-access journals don’t charge subscription fees, making it easier for the public (and universities) to access scientific literature. They’re hailed by advocates as the answer to the skyrocketing fees of the traditional journals, which is causing a “serials crisis” in university libraries, where diminishing budgets mean that administrations can no longer afford to keep bundled publications fully stocked. But the price of publication must shift somewhere: to foundations, donations, or, in some cases, charging the scientists themselves for the right to publish. Critics complain that open-access models have led to a plethora of second-class publications skimping on peer-review costs, or, in the case of fee-charging open access journals, create financial burdens for researchers, and making an incentive for publications to accept papers of any quality.
After receiving a note from a scientist in India complaining of the fee attached to publishing her latest paper, Bohannon, who regularly covers science in the developing world, conceived of an ambitious plan to test the fee-based, open-access journals’ rigor. Under the pseudonym “Ocorrafoo Cobange,” Bohannon wrote a mock study in which he found a new anti-cancer substance, with “wonder drug” potential, in a substance extracted from lichen. Bohannon documents the study’s many flaws in his article. “Any reviewer with more than a high-school knowledge of chemistry..should have spotted the paper’s short-comings immediately,” wrote Bohannon, who promptly sent the article to 304 open access journals, including every one that charged a fee.
If a journal rejected the paper, that was the end of the line. If a journal sent review comments that asked for changes to layout or format, I complied and resubmitted. If a review addressed any of the paper’s serious scientific problems, I sent the editor a “revised” version that was superficially improved–a few more photos of lichens, fancier formatting, extra details on methodology–but without changing any of the fatal scientific flaws.
The journals didn’t reject his paper. Over half (157) the journals accepted it outright; 97 rejected it. The rest were a mix of nonexistent or abandoned publications or short-staffed ones, where the paper was still under review by the time Science published the sting. The media’s reaction to Bohannon’s project was positive. National Geographic used the piece to prompt a discussion of open access. While Bohannon was hailed for his creativity in quick-hit coverage by NPR, The Scientist, and The Discovery Channel’s news site. (The Chronicle of Higher Education was a rare outlier, unpacking Bohannon’s piece, and its critical reception, more fully in its coverage.)
But in their laudatory coverage of the piece, reporters missed the heated criticism launched at Bohannon by the academic world.
“Bohannon’s sting has some major problems itself–most glaringly, the fact that he didn’t submit the paper to any traditional subscription journals,” wrote Michael Eisen, an associate professor of molecular and cell biology at Berkeley. The absence of what Eisen calls a “control group” means that Bohannon’s study “gives no sense of how blameless traditional publishing is on these counts.” The Database of Open Access Journals called Bohannon “racist” for fixating on journals based primarily in the developing world. (That’s a claim that’s easily debunked by looking at the plotted locations of the journals on a map Science published; the United States is the second most-populous home base.) In a blog post, Peter Suber, the director of the Harvard Open Access Project, called the piece “unjustified and invidious.” “It exposes a problem with low-quality or dishonest journals, not with OA journals as such,” he wrote.
“The one misunderstanding I’ve seen on a lot of the social media posting about this is that I had a small sample and that it’s not random and that it’s uncontrolled,” Bohannon told me, when I reached him by phone on his way home from Canada, where he was administering the “Dance Your PhD” interpretive dance competition. “If my goal was to do a comprehensive story about the open-access business model of academic publishing, then this was a small sample.” The difference, Bohannon says, is he wasn’t trying to compare open-access to traditional journal models; he was trying to evaluate fee-based publications. “I didn’t do a small sample of those; I did all of them,” he said. “In all the hot air on this, I did all of them. Every single one.”
But some of the language of the piece (the words “open access” are used frequently, almost interchangeably with “fee based”) and a subsequent press release, reprinted on Eisen’s site, give a different impression. According to the press release, the article “exposes the dark side of open-access publishing”…revealing “a “Wild West” landscape that’s emerging in academic publishing, where journals and their editorial staffs aren’t necessarily who or what they claim to be.” Though Bohannon says he regrets not clarifying his fee-based intentions, he still thinks the sting reflects accurately on the open access model. “I have shown a very high frequency and incidence of fraud within this new and fastest growing section of academic publishing,” he says.
Much of the language used to criticize Bohannon’s piece is scientific jargon; this makes sense, because most of his critics are scientists. But critics of his work have held it to a higher standard of research than most works of investigative journalism–did Watergate require a control group? Perhaps because his original piece is so scientifically rigorous (and open access so hotly contested), he’s been held to a higher standard of proof. “I deliberately mixed styles between traditional narrative investigative journalism and science communication, because my readership is readers of science,” he said. “I wanted to spell out rigorously how I generated the paper, what the flaws were. I wanted to be more clear about the methodology of things than one typically is in journalism.”
Regardless of the criticism, more surprising is the fact that the piece has largely failed to have the rippling effects that Bohannon envisioned when he conceived of the project. Though one journal that accepted the paper was ultimately shut, public criticism hasn’t touched the others that accepted the piece in a significant way. Even the findings that reflect positively on open-access have been largely unacknowledged: Along with the journals that Bohannon debunked, the often criticized journal of the Public Library of Science, PLOS ONE, came off looking vindicated when they rejected the article in just two weeks.
If scientists can write off the results of such a sting with hedging about caveats, then Bohannon worries about the future of rigor in open-access publishing. “Shouldn’t you feel as surprised and worried as I felt when I saw these results?” Bohannon said in an interview this week. “I can only assume that people are either far more cynical than I am about academic publishing, or they just haven’t thought through this all clearly yet.”
Alexis Sobel Fitts is a senior writer at CJR. Follow her on Twitter at @fittsofalexis.