In the course of surfing the Web last week, you may have come across some polling data showing that large numbers of Republicans believe some pretty scary things about the president. The figures, provided by Harris Interactive, seem to have been first reported by Daily Beast contributor John Avlon in an item posted at midnight Tuesday. The story, titled “Scary New GOP Poll,” cited details including:
45 percent of Republicans (25 percent overall) agree with the Birthers in their belief that Obama was “not born in the United States and so is not eligible to be president”
38 percent of Republicans (20 percent overall) say that Obama is “doing many of the things that Hitler did”
Scariest of all, 24 percent of Republicans (14 percent overall) say that Obama “may be the Antichrist.”
Ok, maybe that last line is not meant to be taken seriously, but still, pretty crazy, right? And Avlon wrote that “the full results of the poll… are even more frightening.” But those results wouldn’t be available until Wednesday, meaning there was no way for readers to see them—and, more importantly, no way for skeptics to examine the poll’s methodology—as the early numbers filtered through the media.
But soon enough, the results did come out, and they were promptly and persuasively skewered by, among others, Gary Langer, the polling director for ABC News. Langer’s post clearly summarizes various shortcomings of the poll, from its approach to sampling to the way it persistently pushed respondents toward more extreme positions through “a highly manipulative approach to questionnaire design.” The obvious response is that, whatever the design, 14 percent of respondents really did say that Obama “may be the Antichrist.” But as Langer writes near his conclusion, good polling involves more than simply asking a question:
Unless carefully crafted, with balance and an approach that encourages due consideration and probes for meaning, simply asking the question can turn into little more than the old reporter’s trick of piping quotes. It’s a shopworn use of true/false and agree/disagree questions, one long overdue for retirement.
Langer’s post doubles as a helpful primer on polling methodology, but even for those not interested in the arcane, there were plenty of red flags here. As Avlon wrote in his item, the poll was “inspired in part by my new book Wingnuts.” (The full title is Wingnuts: How the Lunatic Fringe is Hijacking America.) While Avlon apparently did not pay for the poll, the opening of the Harris press release cited his book by name, and stipulated that the purpose of the poll was “to measure how many people are involved” with right-wing extremism.
It sounds not so different from a similar poll commissioned by Daily Kos founder Markos Moulitsas earlier this year, which, he wrote on his blog, was meant to provide support for “certain claims about Republicans” he was making in his book, American Taliban. (That poll, too, came under scrutiny from experts in the field: see here, here, and here.) Most inquiries begin with propositions to test, of course, but in these cases there seem to be clear incentives for all involved are to generate as many scary-sounding responses as possible. (Harris has responded to criticisms of its poll’s methodology, and the motivations behind it, in a Q&A at its site.)
What are the consequences of this? Well, for one thing it seems likely to, if anything, entrench the environment of “fear and hate” these writers are decrying: even as Avlon worries over “hyper-partisanship,” he’s feeding his readers tenuous numbers that give them permission to believe their worst fears about conservatives.
From a media perspective, meanwhile, the existence of these polls makes it harder to communicate good information. That’s partly because any storyline can be hard to dispel once it’s accepted, but there’s something more going on. In this case, we really are in a period in which conservative elites—both elected officials and members of the media—have been using inflammatory rhetoric. It’s reasonable to conclude that this rhetoric has contributed to the incidents of harassment and vandalism we’ve seen in the wake of the health-care vote. For a number of reasons, it would be journalistically valuable to try to deduce the broader state of opinion among conservative voters, and both polling and old-fashioned reporting can play a role in that process.
But in order both to be credible and to be perceived as credible, that undertaking has to come from a place of open inquiry, not from an expectation that the results will boost book sales, create a news cycle, or confirm a political narrative. The whole appeal of polling is its promise (sometimes oversold) to produce objective, almost scientific data that tells us something about the state of our politics. When that enterprise itself becomes an exercise in political gamesmanship, it may foster cynicism about the utility and quality of any effort to collect information. If any given piece of news is valuable only insofar as it advances a particular political view, then they are all in a sense equivalent, which is to say worthless.
That’s not to say that political perspectives don’t have a place in journalism (they do) or that our political views won’t shape the way we understand information (they will). But we still need to protect a place for facts and evidence in our political debate, and to do that we need to push back against rhetorical opportunism and statistical sloppiness on all sides. Journalists can do this by not cutting corners when reporting and compiling data—but also by making the case for why methodological rigor is valuable, and by providing readers with the tools to evaluate information themselves.
This, in the end, is the real value in Langer’s post—it explains why the Harris poll is flawed in a way that will hopefully prompt closer scrutiny of storyline it perpetuated. While this type of direct engagement between media outlets is somewhat unusual, Langer has a clear incentive here: as a pollster, he needs to uphold the integrity, credibility, and rigor of his profession. It’s a lesson for other members of the press to keep in mind.
Update: For another close look at the Harris poll featuring thoughts on how to design a more valid survey, see Mark Blumenthal’s “Mystery Pollster” column for National Journal.
Update 2, 3/30: James Taranto’s Wall Street Journal piece last Thursday also raises good questions about the sampling and survey design.
Greg Marx is an associate editor at CJR. Follow him on Twitter @gregamarx.