An interesting debate about polling samples is underway this afternoon in the wake of a very encouraging new set of figures for President Obama.
The figures come from a new and widely cited AP-GfK poll released today that shows the president’s approval rating hitting its highest point in two years (on the AP-GfK poll): 60 percent. That’s a significantly bigger bump (from 53 percent in March) than has been noted by several other polls and fits more in line perhaps with conventional thinking about the kind of boost the president would receive after giving the order to take out Osama bin Laden. (The Times polling guru Nate Silver has a nice summary and discussion of Obama’s post-Osama poll numbers here.)
Here’s how the AP presents the latest poll findings in its report:
President Barack Obama’s approval rating has hit its highest point in two years—60 percent—and more than half of Americans now say he deserves to be re-elected, according to an Associated Press-GfK poll taken after U.S. forces killed al-Qaida leader Osama bin Laden.
In worrisome signs for Republicans, the president’s standing improved not just on foreign policy but also on the economy, and independent Americans—a key voting bloc in the November 2012 presidential election—caused the overall uptick in support by sliding back to Obama after fleeing for much of the past two years.
Comfortable majorities of the public now call Obama a strong leader who will keep America safe. Nearly three-fourths—73 percent—also now say they are confident that Obama can effectively handle terrorist threats. And he improved his standing on Afghanistan, Iraq and the United States’ relationships with other countries.
The report has been doing the rounds, from NPR to Gawker. But conservative online media outlets have locked on to something that may discredit the gleaming 60 percent headlines: the AP-GfK’s sample was heavily skewed toward Democrats and few of the reports on the approval rating are noting as much.
Here’s what conservative blogger Ed Morrissey had to say in a Hot Air post headlined, “And the award for most ridiculous poll sampling goes to…”
The Dem/Rep/Ind breakdown in this poll is 46/29/4, as AP assigned most of the leaners to the parties. That is a 17-point gap, more than twice what was seen in the 2008 actual popular vote that elected Obama. It only gets worse when independents are assigned properly. When taking out the leaners, the split becomes—I’m not kidding—35/18/27. Oh, and another 20% “don’t know.” That’s significantly worse than the March poll, in which the proper D/R/I was 29/20/34, and far beyond their post-midterm sample of 31/28/26. It’s pretty easy to get Obama to 60% when Republicans are undersampled by almost half.
At National Review’s Campaign Spot blog, Jim Geraghty asks: “With a poll sample that has a 17-percentage-point margin in favor of the Democrats, is anyone surprised that these results look like a David Axelrod dream?”
These pugnacious bloggers have a point.
Compare the samples in the May poll and the March poll and indeed the numbers in the later poll favor Democrats by a significant degree—46 percent lean Democrat, 29 percent Republican, and 4 percent independent for May; for March, 45, 33, and 4. For context, in May 2010, the divide was 45, 39, and 2.
Now, not being Nate Silver or a poli-sci wonk of any stripe, it’s difficult to say how significant these kinds of sample differences are, both in reading the latest results in isolation, in reading them against previous AP-GfK polls, or reading them against other polls quantifying what’s being called the “Osama bump.” And perhaps the direction in which respondents decide to tell interviewers that they lean is as contingent on elements like an “Obama bump” as is the way they decide to respond on the question of presidential approval (and thus, the sample make-up might actually be as reflective of the national mood—and not just the Democratic mood—as the approval rating it provides.)
No, Joel. You are mistaken. Please brush up on your use of the word "significant."
46% vs. 45% percent Democrat is NOT a significant difference in a poll like this;
29% vs 33% percent Republican MAY be a significant difference, just barely, in a poll like this (you should check with Roper before you claim it is significant)
4% vs 4% independent is not a significant different in a poll like this.
What is certain is that you probably need to check with a statistics expert and issue a correction of this post.
If you look at the self-reported ideology identification, there is not much difference there, a lot of noise in the "moderate" category. And if you look at the confidence in each party's ability to solve problems, there has been a very noticeable erosion of respondent's perception of the Republicans to solve America's problems -- terrorism, the economy, Medicare.
What these numbers tell me is that the Republicans are losing substantial support in pushing their anti-American agenda -- abolishing Medicare, destroying the middle class, pushing their forced childbirth agenda. And an uptick in American's confidence in Obama's ability to solve problems. That's not surprising what with the bin Ladin thing, his recent speeches assuring Americans that he won't allow the Republicans to destroy Medicare and Social Security, and his successful appearance at the WH Correspondence Dinner.
Please note that this poll includes 30% cell-phone users -- an appropriate proportion of the population in 2011. This differs from the cheap robo-polls that Rasmussen and CNN run, which skew heavily towards older, conservative, suburban and rural landline users. These robopolls are what the rightwing ideologues love to see, and these are the cheap, quickie polls that innumerate journos run with. These robo-polls are no longer representative of American opinion.
Please note, as well, that these numbers for party identification are for all respondents, and not registered voters or likely voters. Registered voters and likely voters, as political journos are used to seeing, have a different kind of distribution of opinion -- you can see there the difference in party "don't know" between likely voters and this poll.
You been spun, dude. Do the additional work necessary to write accurately on polling of American opinion, please.
#1 Posted by James, CJR on Wed 11 May 2011 at 06:46 PM
James, Thanks again for the comments. As noted, not claiming to be a polling expert, but my comparison and use of the term "significant" was not regarding the differences between the percentages of Democratic-leaners in the March and May poll, but the difference between Democrats and Republicans in both polls (46-29 in May, 45-33 in March). Noting the numbers from the previous polls was to show that—in my view—the difference between the percentage of Dems and Reps in the May poll sample was even more "significant" than it has been in the past. If this was unclear, I apologize. I still say however, that with figures that glowing for the pres, it's important to note, for context, that 45 percent of respondents self-identified as leaning toward his party and 29 percent self-identified as leaning toward the other guys. Have at me.
#2 Posted by Joel Meares, CJR on Thu 12 May 2011 at 01:53 PM
Well, look, Mr. Meares,
I've always thought that the most basic requirement of becoming a political writer is taking a basic statistics class. Sadly for readers everywhere, there is no such requirement.
You said that "These pugnacious bloggers have a point" and "What is certain is that the sample difference in this instance between Republicans and Democrats is significant." Those are strong statements of support for the views of these two rightwingers. I'm telling you they are wrong and don't know what they are talking about. I appreciate that you attempted to provide context, but I don't agree with how you did it.
First of all, the word "significant" has a very specific meaning in statistics. It means that -- if there was no bias in sampling -- that the difference is attributable to something other than chance. Secondly, both samples had a margin of error of 4.2%. That means the 29% could be a little better than 33% or a little less than 25%. What *that* means is that the margin of error is such that the distribution of party ID numbers may not be significant -- it's a close call. So you shouldn't use the word. A noticeable difference, an eye-popping difference, quite a difference would have been preferable.
The bloggers don't have a point. They don't like the results of the poll, that's what's going on here. This is a sample of public opinion which has been obtained by valid population-based sampling methods. They are claiming that the sample was obtained by respondents' political affiliation, when actually they were sampled and asked that question, among other questions. There's a big difference there. I pointed out other questions that support my contention that the decline in GOP party identification appears to be something other than biased sampling.
I'll note here that the MAIN sample has a margin of error of 4.2%. When you are breaking down to categories and comparing them as you have done with two different surveys, the MoE goes up quite a bit. If you have 28% of 1001 (the main sample number), that's down to 280 respondents. If you are comparing 280 respondents to 330 (33%) respondents, you've got a MoE of 6% or greater. So that difference is *really* not significant.
I also pointed out some evidence that there might be some noise -- random variation -- in the party and ideology questions. If one attributes those differences to noise, then the difference would be *not* significant. I also pointed out that the sample results were those of the general public, and not registered or likely voters, which is an appropriate population for these questions. Political journos are not used to seeing opinions of the general population, because most political polling is of registered voters, and Rasmussen, with their cheapo robo-polling, uses some weird model of likely voters to get the results that they want, i.e., profit from. Same with CNN's polling outfit. Their main goal is not information, but headline-generation. So don't be fooled. Please.
However, these questions were asked independently of the question of Obama's approval. The margin of error for Obama's approval is 4.2% -- could be 56% or 64% -- and I see no evidence presented that would suggest a biased sample. Can you?
A better approach, if I may suggest it, would have been to consult the polling outfit or an independent polling expert for interpretation of these numbers, rather than just weighing in on a subject for which you have no expertise, and claiming that the points they make are valid.
NOTE: As an epidemiologist, I make a living designing population-based surveys on health- and behavior-related subjects, managing, analyzing, and interpreting the data, and presenting it in reports, publications, professional conferences, and peer-reviewed journal articles.
.. What qualification does Michelle Malkin and Ed Morrisey have for weighing in on the validity of the poll?
#3 Posted by James, CJR on Thu 12 May 2011 at 05:10 PM
For all the display of scholarly apparatus above, the poster above does not persuade. The AP poll was striking because it is an outlier among polls, putting Pres. Obama's approval rating much higher than anyone else does. Gallup has Obama at 46% today, for example. The margin-of-error argument does not persuade because 56% is still much higher than anyone else has for an approval rating. The skewed sampling is a legitimate argument, whether Obama supporters want to believe it or not.
Don't just take my word for it. Anyone want to bet that the next AP poll will have a different composition in its sample?
#4 Posted by Mark Richard, CJR on Mon 16 May 2011 at 07:36 PM
More James Statistics 101:
The Margin of Error calculation is the interval around which, assuming no bias, one has 95% confidence (as in this particular survey) that the true population proportion is within the interval. Or put another way, if one took 100 different samples of the same population, at least 95 of them would fall within the margin of error interval. So, that doesn't preclude the possibility that the survey is an outlier. There is no evidence, I contend, that the sample itself is biased, as the seething, angry rightwingers (who have never completed a statistics class) are claiming.
One should be very cautious about comparing approval and job ratings across different surveys. Questions asked in different ways, at different junctures of the survey, will produce different numbers. As I noted above, this particular sample included 30% cell phone users, a remarkably different demographic than the landline population. Most surveys still use 20-25% cell phone users, except Rassmussen and CNN, who use convenience samples of people sitting beside their landline phone watching television on a weekday evening.
And don't confuse job rating with approval rating. Approval ratings traditionally run much higher than job approval ratings. That's true for all presidents, not just Obama. This was an approval rating. The Gallup daily tracking poll is job approval, a wholly different animal.
But thanks for reading!
#5 Posted by James, CJR on Mon 16 May 2011 at 09:45 PM
To James, the problem is, as I wrote, that the AP poll put Obama approval rating much higher than anyone else had it. You don't have to be a seething, angry right-winger, or to have completed a statistics course (which I have, mean-square weighted deviations and all, though I don't seethe, am not angry, and would only be considered a 'right-winger' by strict ideological drillmasters of the far Left) to take note of the anomaly.
#6 Posted by Mark Richard, CJR on Tue 17 May 2011 at 12:35 PM
Mr. Richard, I wasn't really referring to you with the "seething, angry" bit, and I apologize if I gave that impression. It's true, you often make interesting points and offer thoughtful analysis in a respectful and rational manner.
Certainly, the AP poll may be an outlier, or it may reflect an indication of temporary euphoria consistent with a big news story like this. After all, approval and disapproval are fluid opinions, which is why they are surveyed, as opposed to other, more stable population characteristics. The bloggers in question were asserting that the sample was intentionally biased toward Democrats for whatever reason, and Mr. Meares appeared to agree, so I thought it worth the time and effort to educate on the subject of population sampling, the interpretation of statistics and the limitations thereof.
Peace?
#7 Posted by James, CJR on Tue 17 May 2011 at 02:56 PM
As stated, James, eminently reasonable. Peace.
#8 Posted by Mark Richard, CJR on Wed 18 May 2011 at 12:31 PM