PRAIRIE VILLAGE, KS — “Bottom line, Rep. [Tim] Huelskamp appears to be utterly full of hooey,” Charles Gaba of ACASignups.net told CJR in May, after the bombastic Tea Party congressman had claimed that more people in his home state of Kansas were uninsured under Obamacare than before.
Our story also noted that Washington Post factchecker Glenn Kessler had awarded his worst rating of “four Pinocchios” to Huelskamp’s claim—and that Kessler’s critique had spurred some critical coverage of the congressman here in Kansas.
Those criticisms were warranted, based on the data available at the time. This week, though, a surprising poll result made the situation a bit more complicated. The latest numbers in the Gallup-Healthways Well-Being Index show a staggering 5 percentage point increase in Kansas’ uninsured rate—from 12.5 percent at the end of 2013 to 17.6 percent now.
That would make the state a massive outlier from the rest of the country, where the uninsured rate has clearly gone down. (The Gallup poll found statistically insignificant increases in the latest period in two other states, Virginia and Iowa.) And if the uninsured rate has in fact gone up in Kansas, that should prompt questions about the policy decisions the state has made—not to expand Medicaid, and not to set up a state-based exchange. States that took those steps saw the greatest decreases in their uninsured rates, according to Gallup’s findings.
Still, as Gaba, a noted Obamacare blogger and “numbers geek,” wrote on his blog Wednesday: with respect to the factual claim, “if the current Gallup survey is accurate, it appears that all three of us”—himself, Kessler and CJR—“owe Rep. Huelskamp an apology.”
But that turns out to be a pretty big “if,” and unpacking the reasons why may offer some insights on how to report on surprising survey data.
Gaba, for his part, remains skeptical. Based on enrollment numbers from the Department of Health and Human Services, the Centers for Medicare and Medicaid Services, and the GOP-led House Commerce Committee, he writes, “a total of at least 172,000 people … had to somehow have lost their insurance in the first half of this year” in the state for the Gallup number to add up. This would be all the more puzzling given that a state insurance official told Kessler in April that the number of Obamacare-related cancellations and non-renewals in Kansas was “low.” And although the state’s economy has been stagnant in the past year, it has not seen massive job losses or sudden demographic shifts that would account for such a spike in the number of uninsured. (Kessler is out of the country and unavailable for comment this week. Huelskamp, who narrowly beat back a primary challenge on Tuesday, also did not respond to a request for comment.)
Kansas Insurance Commissioner Sandy Praeger was skeptical of the Gallup result too, telling the Kansas City Star that the number “appears to be an anomaly that needs more review.” She said the state’s uninsured rate has remained steady for years and that such a dramatic one-year uptick would be “unprecedented.”
So what does Gallup have to say? Turns out they’re surprised, too.
“It’s a weird one,” Dan Witters, research director for the Gallup-Healthways Well-Being Index, acknowledged in an interview on Wednesday. “Every other state is significantly lower or flat.”
The discrepancy prompted internal discussion before the results were released. “The Gallup.com team and our editorial review team spent a lot of time talking about Kansas last week because it’s a significant outlier,” Witters said. But ultimately, he added, “It’s not like we’re not going to report Kansas just because it’s an outlier.”
So if the number of uninsured Kansans hasn’t actually spiked, what’s another explanation for the unexpected result?
One possibility offered by Witters is that the 2013 baseline of 12.5 percent uninsured might have been a bit low to start with. In fact, that figure marked a significant decrease from Gallup’s estimates for each of the prior four years, which never strayed more than about a point from the average of 15.0 percent. Starting from that average, the 17.6 percent figure in the latest survey is no more anomalous than 12.5 percent in 2013.
The recent pattern, then, as Witters put it, is consistent with a “lower-end-of-the-bell-curve estimate in 2013 followed by a higher-end-of-the-bell-curve estimate the year after.”
In fact, the earlier, lower figure might be more reliable than the newer, higher figure given the respective margins of error for the two results. The mid-year surveys, Witters told me, are conducted with much smaller sample sizes than the year-end surveys. The sample size in Kansas for this latest survey was 845 respondents, he said, which is half the size of a year-end sample. This means a greater margin of error—3.1 percent, according to Witters.
Ultimately, Witters said, the latest release is “just a mid-year snapshot” whose validity will be tested in late January with the release of a more comprehensive survey, and again in future surveys to come. “We will need to get the full-year 2014 estimate, followed by the 2015 estimate, to have a good handle on the 2014 situation in Kansas.”
This long view is also encouraged by Brendan Nyhan, a Dartmouth political scientist and former CJR contributor.
“My general advice to journalists has been to beware statistical anomalies and outliers in cases like this and consider the broader array of evidence instead,” Nyhan told me in an email Wednesday. “We know that there’s some probability of getting highly inaccurate results in one poll due to random chance, so when expert judgments and experience in other states all point in the other direction it’s important to be cautious and look for confirmation across multiple measures rather than blindly chasing what could be statistical noise.”
Coverage of the survey in Kansas this week did, appropriately, point to the questions around the new figure—though the uncertainty could have been better captured in some headlines.
What about the reporting tasks ahead? The most important story is capturing the full picture of what Obamacare means for Kansans. Future Gallup surveys will help there, as will other data sources, and of course qualitative reporting. If Kansas does, in the end, turn out to be a major outlier, there’ll be a good story to dig into.
In the meantime, if Huelskamp (or anyone else) points to the Gallup survey in support of his earlier claims, an enterprising reporter might want to ask what he makes of the survey’s results as a whole—and underscore that there’s uncertainty here. There’s now a bit of support for the congressman’s claim, but we’ll hold off on the apologies until further data comes in.
Deron Lee is CJR’s correspondent for Iowa, Missouri, Kansas, and Nebraska. A writer and copy editor who has spent nine years with the National Journal Group, he has also contributed to The Hotline and the Lawrence Journal-World. He lives in the Kansas City area. Follow him on Twitter at @deron_lee.