Last month John Bohannon, a biologist and science writer, published the kind of piece in Science magazine that defies classification. One part sting operation, one part creative investigation, the article, “Who’s Afraid of Peer Review?” chronicled Bohannon’s attempts over the course of a year to factcheck the peer review process of a number of open-access scientific journals. Bohannon’s piece is by far the most ambitious attempt to check the rigor of peer review in open-access publishing, but perhaps because of the heated charge behind the topic, press coverage of his project was out of sync with the larger conversation happening in the ivory tower.

One of the most debated topics in science publishing, open-access journals don’t charge subscription fees, making it easier for the public (and universities) to access scientific literature. They’re hailed by advocates as the answer to the skyrocketing fees of the traditional journals, which is causing a “serials crisis” in university libraries, where diminishing budgets mean that administrations can no longer afford to keep bundled publications fully stocked. But the price of publication must shift somewhere: to foundations, donations, or, in some cases, charging the scientists themselves for the right to publish. Critics complain that open-access models have led to a plethora of second-class publications skimping on peer-review costs, or, in the case of fee-charging open access journals, create financial burdens for researchers, and making an incentive for publications to accept papers of any quality.

After receiving a note from a scientist in India complaining of the fee attached to publishing her latest paper, Bohannon, who regularly covers science in the developing world, conceived of an ambitious plan to test the fee-based, open-access journals’ rigor. Under the pseudonym “Ocorrafoo Cobange,” Bohannon wrote a mock study in which he found a new anti-cancer substance, with “wonder drug” potential, in a substance extracted from lichen. Bohannon documents the study’s many flaws in his article. “Any reviewer with more than a high-school knowledge of chemistry..should have spotted the paper’s short-comings immediately,” wrote Bohannon, who promptly sent the article to 304 open access journals, including every one that charged a fee.

If a journal rejected the paper, that was the end of the line. If a journal sent review comments that asked for changes to layout or format, I complied and resubmitted. If a review addressed any of the paper’s serious scientific problems, I sent the editor a “revised” version that was superficially improved—a few more photos of lichens, fancier formatting, extra details on methodology—but without changing any of the fatal scientific flaws.

The journals didn’t reject his paper. Over half (157) the journals accepted it outright; 97 rejected it. The rest were a mix of nonexistent or abandoned publications or short-staffed ones, where the paper was still under review by the time Science published the sting. The media’s reaction to Bohannon’s project was positive. National Geographic used the piece to prompt a discussion of open access. While Bohannon was hailed for his creativity in quick-hit coverage by NPR, The Scientist, and The Discovery Channel’s news site. (The Chronicle of Higher Education was a rare outlier, unpacking Bohannon’s piece, and its critical reception, more fully in its coverage.)

But in their laudatory coverage of the piece, reporters missed the heated criticism launched at Bohannon by the academic world.

“Bohannon’s sting has some major problems itself—most glaringly, the fact that he didn’t submit the paper to any traditional subscription journals,” wrote Michael Eisen, an associate professor of molecular and cell biology at Berkeley. The absence of what Eisen calls a “control group” means that Bohannon’s study “gives no sense of how blameless traditional publishing is on these counts.” The Database of Open Access Journals called Bohannon “racist” for fixating on journals based primarily in the developing world. (That’s a claim that’s easily debunked by looking at the plotted locations of the journals on a map Science published; the United States is the second most-populous home base.) In a blog post, Peter Suber, the director of the Harvard Open Access Project, called the piece “unjustified and invidious.” “It exposes a problem with low-quality or dishonest journals, not with OA journals as such,” he wrote.

Alexis Sobel Fitts is a senior writer at CJR. Follow her on Twitter at @fittsofalexis.