Polls taken after the Iowa caucus and before the New Hampshire primary consistently showed Barack Obama beating Hillary Clinton—by as many as 13 percentage points. Today, not so much.
“It’s a really big deal. I’ve never seen anything like it and I think anyone would be hard pressed to find another collapse this big,” says Gary Langer, the director of ABC’s polling unit—which didn’t poll before New Hampshire. CJR asked him what went wrong, how pollsters might solve the riddle, and whether or not this incident might change the way journalists treat polls.
CJR: What could possibly explain this? You have nine different organizations with nine different screens.
GL: There are a variety of possibilities—in the quality of the sample, in overstatements of enthusiasm from Obama supporters
CJR: So you mean more people identifying themselves as Obama voters than actually voted?
GL: Which is possible. But the blame for that, if that was the cause, doesn’t go to the respondents for being enthusiastic. It goes to the likely voter modeling, for failing to accurately select true likely voters.
There are other possibilities. These polls were done largely over the weekend. Saturday is a dreadful day for conducting survey research. Sunday is not great, during the day—Sunday night is a good night for interviews. Two-day polls aren’t great in terms of methodology.
CJR: But all those things were true of the Republican polls too.
GL: That’s true. The Republican race looks to be pretty good. So I’m not sure what went on in the Democratic race. All of the estimates in the Democratic race—there were nine polls released on Monday and Tuesday—everyone of them had Obama in the lead.
There was even a national poll by Gallup which, son of a gun, showed Clinton and Obama tied. That’s the best Obama’s ever done in a national poll. So there was something going on in Obama’s favor.
CJR: Unless that poll’s wrong too.
GL: [Chuckles] That’s possible. There looks to have been sort of a systemic failure. Now we’ve got to take some real careful evaluation to get this figured out.
CJR: So it really is a mystery.
GL: At the moment, it is. But look, there’s a lot of data, and there’s a lot of good analysis that we’ll be able to do around this.
CJR: So where should we be looking?
GL: Another blog I saw today suggested it was the late deciders. That’s a common answer of faulty final pre-election polls, and it’s one I don’t buy, and it’s one I certainly don’t buy in this case. Because if we look at the exit poll results, indeed if we take out everyone who decided on Election Day, we get a result of Clinton plus four, which is of course, exactly what her margin was. If we look at who did decide on Election Day, it’s Clinton plus three, which is within polling tolerances.
Polling can go bad for a variety of reasons, particularly pre-election polling, where you are trying to measure an unknown population. People haven’t voted yet. You have to estimate who is going to turn out. And when pre-election polls are wrong, the first place I would suggest to look is in the likely voter estimates, in the way they tried to decide who was going to turn out.
It might have perhaps been over-done enthusiasm for Obama voters. After the win in Iowa, a lot of people were so charged up to vote for him but maybe didn’t really make it. Maybe it unanticipated a particularly active get-out-the-vote drive by Clinton’s supporters. It’s hard to say. It’s going to take a careful review.
CJR: Where would you begin?
GL: By looking at the estimates of turnout that each of these polls reached—what sort of electorate did they anticipate would vote at the primary, and how did that actually compare with the reality? Then look at the individual figures: what share of the Democratic turnout did they anticipate—older men, and younger people, and all the various groups? Looking at the subgroups will move us a long way towards understanding what went wrong.