A review of nearly 1,500 health-medical articles over the last five years has found that while journalists are nailing a few key categories of quality reporting, they’ve been falling down on the most important ones, like the costs, harms, and benefits of care.

HealthNewsReview.org—a website that reviews news stories about specific treatments, tests, products, or procedures—released the “scorecard” on the occasion of its fifth anniversary this week. The site uses a standardized, satisfactory/unsatisfactory rating system based on ten criteria to evaluate articles from a roster of the country’s top news agencies.

It used to review the top twenty-five newspapers by circulation, plus Time, Newsweek, and U.S. News & World Report, the Associated Press, and ABC, CBS, and NBC news stories. In December 2009, it cut back to the top papers by circulation (excluding the New York Post and Daily News in the name of geographic diversity), dropped the TV networks, and added CNN, NPR, MSNBC, HealthDay, Reader’s Digest, Reuters Health, and WebMD. It does not follow hard copies or broadcasts, and instead relies on the websites of all the outlets.

Tabulating the results of 1,488 reviews, HealthNewsReview found that more than two-thirds of the articles received a satisfactory score in areas like explaining the availability of a treatment or procedure, explaining the novelty of a treatment or procedure, avoiding disease- or fear mongering, and avoiding overreliance on a news release. Yet only forty-one percent appropriately covered alternatives to a treatment or procedure, and just over half did an acceptable job quoting independent sources and letting the public know about potential conflicts of interest. And journalists fared worst in the most vital categories, with only about three out of ten articles adequately describing the costs, benefits, harms, and evidence underlying a treatment or procedure.

“We think all ten criteria are important, but clearly some are more important than others and they’re the ones that, sadly, do the worst,” said Gary Schwitzer, the founder and publisher of HealthNewsReview.

Asked why that might be so, Schwitzer added that the most crucial criteria are undoubtedly the toughest to get right. In an effort to help reporters overcome that challenge, HealthNewsReview just released a “first stab” at an online list of resources to help journalists explore the costs of treatments and procedures. The list is the latest addition to a fourteen-part toolkit designed to help journalists improve their health-medical coverage, which also includes primers on medical devices, phases of drug trials, animal and lab studies, and absolute versus relative risk, among other subjects.

Schwitzer stressed that his site’s mission is not to criticize, but rather improve the accuracy of health and medical reporting (in addition to helping consumers evaluate the evidence for and against screening tests, drug therapies, surgeries, nutritional advice, et cetera). In an interview, he mentioned a 2006 Associated Press article about the launch of HealthNewsReview, which quoted Cristine Russell, a veteran health reporter and CJR contributing editor, saying she hoped the project would not “end up being another media-bashing exercise.”

“I worked really hard to overcome that idea and frame this [website] as exactly what it is, an attempt at constructive outreach,” Schwitzer said. “I wouldn’t have given up a tenured position—I wouldn’t have been doing this every day for the last six years—if I didn’t think that it was helping people,” he added, referring to the job he left last year teaching health journalism at the University of Minnesota.

For her part, Russell agrees that the site has not indulged in any gratuitous media bashing—far from it. “HealthNewsReview has played a constructive role in setting the bar for what good health journalism should be,” she wrote in an e-mail. “It has set tough standards that may be hard for some journalists and organizations to meet, particularly at a time of limited staffs and added pressure to feed the blogs and websites. But at least they know where to aim. And HNR plays an important watchdog role when it calls out bad practices and ethical lapses.”

Curtis Brainard is the editor of The Observatory, CJR's online critique of science and environment reporting. Follow him on Twitter @cbrainard.