the second opinion

Breaking down the numbers in CNN’s controversial report on a Florida hospital

July 2, 2015

On June 1, CNN aired a year-long investigation that made alarming claims about high mortality rates and other complications in the pediatric cardiac surgery program at St. Mary’s Medical Center in West Palm Beach, Florida. Published under the headline “Secret Deaths,” the report landed with a splash—and soon brought a backlash. The hospital sent out its own set of numbers, calling CNN’s calculations about the hospital’s mortality rate “deeply flawed” and the comparisons to a national average “completely erroneous,” and declaring that the hospital “will consider all options to ensure that the network, its reporters and editorial staff will be held accountable.” A state healthcare agency in Florida criticized what it called the network’s “sensationalized reporting.” Meanwhile, CNN fired back, publishing a string of follow-ups and touting the impact of its original report.

As The Palm Beach Post wrote in a bit of local pick-up on June 8, what “started out as a heart-breaking story about dead babies…devolved into a he-said, she-said dispute between a national cable news network and a renowned trauma hospital, with an assortment of voices—including an irate state agency—stirring the pot.”

In an effort to navigate the back-and-forth—and to search for broader takeaways about covering hospital safety and transparency—CJR reached out over the past few weeks to several of the key figures in this story, including CNN reporter Elizabeth Cohen, representatives for St. Mary’s, and the Society of Thoracic Surgeons (STS), which hosts a database on surgical outcomes. We also spoke with Jeffrey Jacobs, a professor of cardiac surgery at Johns Hopkins University and the chairman of the STS workforce on national databases, who participated in a 2014 panel review of St. Mary’s pediatric cardiac surgery program. And for some outside perspective, CJR contacted Ashish Jha, a professor of health policy at Harvard and a national expert on hospital safety. Though there are many twists and turns in this dispute, we chose to focus on the main points contested by St. Mary’s, which deal with CNN’s representations of the mortality rate at the hospital’s pediatric cardiac surgery program.

The bottom line: Though CNN should have used more care in how it presented and described its data, given the information that was available, its method for calculating the mortality rate at St. Mary’s and comparing it to the STS national average was basically sound. At the same time, the dispute about mortality rates highlights just how hard-to-parse health statistics can be, and how a lack of transparency in the healthcare field creates challenges for reporters and makes these fights harder to resolve. Meanwhile, the data debate has distracted attention from the most journalistically compelling part of CNN’s original report: the unsettling findings of that 2014 review, which the network pried loose from the state over the hospital’s objection.

The debate over the data

Throughout its coverage, CNN has reported that from 2011 to 2013, St. Mary’s had a mortality rate for pediatric open-heart surgeries of 12.5 percent—which it says is more than three times the national average of 3.3 percent, as recorded in a database maintained by the Society of Thoracic Surgeons. St. Mary’s has challenged this claim on two main points: First, that any comparison must apply a “risk adjustment” to take into account the fact that some hospitals see sicker patients than others; and second, that CNN compared data for a subset of the hospital’s surgeries to a different set of surgeries in the national database.

Risk-adjustment. When it set out to determine the mortality rate in St. Mary’s program, CNN faced a reporting problem. According to the network, St. Mary’s wouldn’t discuss its numbers, except to tell CNN that its calculations were wrong. And St. Mary’s was not among the minority of hospitals that allow STS to publish its data. So, the network turned to other sources. CNN pulled state records to determine that the hospital had performed 48 pediatric open-heart surgeries from late 2011, when the program began, through the end of 2013. Separately, Cohen and colleagues confirmed six deaths in that time period after pediatric open-heart surgery at St. Mary’s by interviewing parents. That’s the basis for the reported mortality rate of 12.5 percent.

Sign up for CJR's daily email

Those numbers represent what’s called the raw mortality rate. But as St. Mary’s has noted, the Society for Thoracic Surgeons, in a discussion of its public reporting database, says that to “allow for a fair comparison of hospitals,” it applies a risk adjustment that takes into account patients’ age, chromosomal or anatomic abnormalities, and other factors. And in response to the CNN report, the hospital announced that—take a big breath here—its STS-calculated risk-adjusted mortality rate for congenital heart surgery patients between mid-2011 and June 2014 was 5.3 percent. That is not a “statistically significant difference” from the national average, St. Mary’s noted. On June 30, St. Mary’s released an updated statement saying that according to the latest data its overall risk-adjusted mortality rate is 4.7 percent, “within the average range” of hospitals that participate in the database.

There’s merit to the argument that adjusting for risk offers a fuller picture of a hospital’s performance, and CNN’s coverage should have been clearer on that point. In its initial on-air story, the concept of risk-adjustment is not mentioned. In the accompanying text version, it comes up only in a quote from the hospital’s CEO, who is elsewhere portrayed in an unsympathetic light. Cohen or a neutral expert could have explained instead why risk adjustments are used, and Cohen could have explained more fully, in the initial report, why CNN chose to use the raw rate. The raw mortality rate CNN calculated, which was teased in the headline and featured prominently in a graphic, could have been presented more cautiously.

However, the raw rate does have meaning—it is “a reasonable measure” to discuss and is “not irrelevant,” said Ashish Jha, the Harvard professor. As CNN has noted, some hospitals do make raw mortality rates public, either through the STS or on their own websites. CNN’s method for calculating the rate for St. Mary’s was reasonable. And Cohen reported the data that were available.

Jha, who is an advocate for greater public reporting of medical outcomes, had little patience for St. Mary’s complaint about CNN’s use of raw mortality data, since the hospital had chosen not to make its adjusted rate public. “It’s hard to blame CNN for not being as complete and thorough as ideal when it’s the hospital that has not made data available,” he said. “They blame CNN for not reporting all the data. It’s great to have your cake and eat it, too.”

Which surgeries, which patients? So regardless of whether it was fair for CNN to use the raw rate it calculated, does risk-adjustment make most of the gap with the national average go away? Actually, we still can’t say, because the hospital’s adjusted number covers a different, larger set of surgeries than CNN evaluated, and not just because of the slight difference in time span. That brings us to the other point of dispute.

St. Mary’s says that CNN’s calculation is “based on an incorrect assumption of volume,” because “the 48 cases they used are only open heart cases” while “the STS database includes both open and closed cases”—closed cases meaning those performed without a cardiopulmonary bypass machine. In its replies, St. Mary’s has focused on the overall national average and its own risk-adjusted rate for all congenital heart surgeries eligible for reporting in the database. (Nationwide, a small portion of “congenital” surgeries are also performed on adults.) For the period from July 2011 to June 2014, the hospital says, that encompasses a total of 70 patients.

In its comments, the hospital has drawn comparisons between that set of surgeries and the overall national average. “It is incorrect and misleading for CNN to compare a raw mortality rate of only open procedures to the STS calculated national average,” a St. Mary’s spokesperson told CJR. (In this paywalled Palm Beach Post story , St. Mary’s acknowledged seven deaths among those 70 patients.)

It’s true that CNN has been clear throughout its coverage that it is focusing on pediatric open-heart surgeries. And it’s true that if you go to the STS website you can find national mortality data by age group, but not broken down by open or closed surgeries. At first, this looks like a problem for CNN.

However, the society’s underlying data does distinguish between open and closed surgeries. And Cohen says she was mindful of this concern and took steps to get a proper comparison.

“We felt it was very important to get this specific information—the national pediatric, on-pump surgical mortality rate for 2011-2013—so that it would match the information we had available for St. Mary’s Medical Center, which is pediatric on-pump surgeries for 2011-2013. We wanted to do an apples to apples comparison,” she said. “So we asked a highly respected source with access to the STS data to run the numbers for us, and that’s where we got our national rate for pediatric open-heart surgeries of 3.3 percent.”

CNN’s method is not necessarily replicable for someone who wants to check what the network did. That was not made clear in the initial report, the text version of which says, imprecisely, that the 3.3 percent figure for pediatric open-heart surgery was “cited by the Society for Thoracic Surgeons.” It should have been clearer.

Still, assuming the analysis was done correctly, it should give CNN the national average for pediatric, open-heart surgeries—the same set of operations it was tracking at St. Mary’s.

In the course of reporting this story, CJR reached out to Dr. Jeffrey Jacobs, the chairman of the STS workforce on national databases—and also a practicing pediatric surgeon at another Florida hospital, who plays a role in CNN’s story. Jacobs emphasized that he could not comment on St. Mary’s. But, he said, Cohen’s use of the national aggregate data was “reasonable, complete, and accurate.”

Where should the focus be?

If you’ve followed us through all that, here’s the unfortunate thing: The mortality rate data are in many ways not the heart of CNN’s contention that there was cause for concern at St. Mary’s.

From an emotional perspective, that heart is the interviews with parents of dead or paralyzed children. And from an analytical perspective, it is the reports filed with the state Department of Health by a team of doctors from other hospitals who conducted a site visit and evaluation in April 2014. CNN obtained the reports via a Freedom of Information request over St. Mary’s objection, and posted them online along with its story. Though the documents contain some praise for aspects of the program, they make for worrisome reading.

The reviewers cited apparent shortcomings that might be of interest to any parent faced with the prospect of heart surgery for a child. With today’s journalistic emphasis on numbers and data, qualitative information is often treated as supplementary. Here, with the data somewhat limited, the reports could have anchored the story—and they were public records under state law, though it’s doubtful anyone had seen them before.

“The echocardiogram reports are uniformly inadequate,” wrote one reviewer. “The institution does not currently have pediatric electrophysiology expertise nor a pediatric electrophysiological laboratory,” wrote another. Some of the reviewers noted concerns about the low volume of patients at the hospital, which could mean the surgical crew wasn’t getting enough opportunities to develop skills for complex operations. The bluntest report came from Jacobs, the Johns Hopkins professor. He wrote that it “is common knowledge that multiple pediatric cardiac surgeons … have expressed serious concern” about the program; that the low volumes and the death of several children who transferred to other hospitals after complex surgery at St. Mary’s represent “not the failure of any one individual [but] the failure of the entire team and system”; and that the existence of other pediatric cardiac surgery programs in the region, “coupled with the very low volumes (and suboptimal outcomes)” at St. Mary’s, “makes this reviewer challenge the need for [the program] to exist.”

Many of those quotes did appear in the text version of CNN’s initial story, and the review was also discussed at length on air. But in both cases, they are supplementary to the mortality rate data. Had the reviews been in the foreground, with the mortality data treated more cautiously in the background, the report could have been just as compelling—and, perhaps, some of the ensuing controversy avoided.

Asked about the decision to highlight the mortality rate calculations instead, Cohen said it stemmed in part from concerns about narrative structure, and in part from the report’s overarching emphasis on transparency. “It’s hard to have an educated, empowered discussion about where to take your baby for heart surgery without having data, and we wanted to make data available and shed light in dark places,” she said.

That focus on data transparency is important, and the larger effort in CNN’s reporting to push more hospitals to publicly disclose more data is valuable. But in the case of this story about this program, the reports were the stronger material.

There’s one other thing CNN might have done before going to air, which was to ask St. Mary’s if it had conducted a “root cause analysis,” or a systemic inquiry to determine the cause of the deaths. Cohen told me, “We did not ask them specifically if they had conducted any internal investigations of the deaths.”

When I asked St. Mary’s if it had conducted such an analysis, the hospital repeated its claim that the mortality rate is “in line with the national average” and added, “we continually evaluate our performance to identify opportunities for improvement”—a boilerplate reply.

But, notably, even as it criticized CNN, the hospital did announce earlier this month a “comprehensive review” including external experts, and a pause in scheduling elective pediatric congenital cardiac surgery cases. According to this CNN tracker, the hospital has also begun reporting the overall volume of surgeries it performs—a step toward more transparency.

Earlier this year, ProPublica’s Charles Ornstein noted at a meeting of the Association of Health Care Journalists that hospital safety coverage may be fading. Given the difficulties Cohen encountered, and the criticism she has faced, other journalists might look at this case and decide it’s best to focus on easier material. That’s not the answer.

Medical errors remain a concern throughout the healthcare system, and despite the efforts of experts like Jha, public reporting often remains limited. As CNN’s broader coverage makes clear, and as Jha emphasized to me, the lack of the transparency the network ran into is not unique to St. Mary’s. And the reformers are right that greater transparency could both provide more information to patients and create pressure to improve outcomes.

We need aggressive coverage that both identifies specific concerns and sets expectations for robust reporting and transparency about outcomes. If the press fails to push for more transparency, who will?

Trudy Lieberman is a longtime contributing editor to the Columbia Journalism Review. She is the lead writer for CJR's Covering the Health Care Fight. She also blogs for Health News Review and the Center for Health Journalism. Follow her on Twitter @Trudy_Lieberman.