Who will win the presidential election next Tuesday? Until recently, the market for analysis of questions like these has been dominated by mainstream political reporters and commentators. Their style leans heavily on qualitative impressions and hazy narratives. But as the audience for quantitative analysis of politics has grown, the establishment analysts have become increasingly defensive about their status.
The most well-known quantitative analyst of politics is Nate Silver, whose FiveThirtyEight blog now appears on the New York Times website. Even though his black box statistical models have not been publicly disclosed or scientifically validated, Silver’s analyses have acquired a talismanic quality among political junkies, particularly with nervous liberals who are reassured by his forecast of a likely Obama victory (current estimate: 72.9 percent).
Unfortunately, Silver has become the target of a vitriolic backlash from innumerate pundits whose market dominance is under threat as well as ill-informed conservative commentators who think Silver is somehow skewing the polls for partisan reasons.
The problem with these objections is best summarized by Elspeth Reeve’s headline at The Atlantic Wire: “People Who Can’t Do Math Are So Mad At Nate Silver”. Most amusingly, on PBS David Brooks said forecasting the expected outcome based on polls makes you some sort of “wizard”:
Because, if you tell me you think you can quantify an event that is about to happen that you don’t expect, like the 47 percent comment or a debate performance, I think you think you are a wizard. That’s not possible. The pollsters tell us what’s happening now. When they start projecting, they’re getting into silly land.
In his piece on the controversy, Politico’s Dylan Byers took the jeremiad against the laws of probability even further, suggesting that Silver would be discredited if Romney won:
So should Mitt Romney win on Nov. 6, it’s difficult to see how people can continue to put faith in the predictions of someone who has never given that candidate anything higher than a 41 percent chance of winning (way back on June 2) and—one week from the election—gives him a one-in-four chance, even as the polls have him almost neck-and-neck with the incumbent.
But as The Weekly Standard’s Jonathan Last pointed out, Silver’s estimate doesn’t mean that there is no chance Romney will win: “a .250 hitter [in baseball] gets on base once a game, so you’d never look at him in any given at bat and think there was no chance he’d get a hit.”
Ultimately, however, the debate over both Silver himself and the specifics of his model misses the point. The best available evidence from both statistical forecasting models and betting markets suggests that Obama remains the favorite in the election. These charts below, for instance, present the estimated probabilities of an Obama victory and current Electoral College forecasts (where available) from the political scientists Jay DeSart and Tom Holbrook, Stanford’s Simon Jackman, Emory’s Drew Linzer, Silver, Princeton’s Sam Wang, the British sports book Betfair, and the Intrade futures market (here and here):

The FiveThirtyEight forecast is not an outlier. Silver’s estimates are generally somewhat more conservative than the other statistical forecasting models and only moderately more confident in an Obama victory than the betting markets. Whatever objection pundits or conservatives may have is with the state of the publicly available evidence or the way in which forecasters and bettors translate that evidence into probabilities, not with Silver or his methods.
It's so cute that you only listed left-leaning models and purposefully avoided the University of Colorado model which puts Romney at 330 votes in the Electoral College.
http://www.colorado.edu/news/releases/2012/10/04/updated-election-forecasting-model-still-points-romney-win-university
#1 Posted by Will Spencer, CJR on Tue 30 Oct 2012 at 03:17 PM
330 for Daddy Warbucks? Dream on whack job!. Blacks and Latinos vote, despite your party's best efforts.
#2 Posted by Kevin, CJR on Tue 30 Oct 2012 at 03:24 PM
@Will Spencer
It's even cuter that all the models in this article are up-to-date unlike the article you've posted that was published about a month ago.
Checking the dates is important unless you own a Delorean.
#3 Posted by Burak, CJR on Tue 30 Oct 2012 at 03:29 PM
Will: That study is a month old and based purely on economic factors, not polling data, so there really isn't a comparison to be made.
If there is other statistical analysis of poll numbers from others out there, we would all love to see them.
#4 Posted by Brian Miksic, CJR on Tue 30 Oct 2012 at 03:35 PM
I've been reading 538 for years. Anyone who thinks that Mr. Silver doesn't explain, in detail, the model obviously does not read his writing. You can also go back to his previous independent blog (prior to his NYT time) which offers further breakdown of the modeling. Casting Mr. Silver as biased because of the current model predictions is insane to me given his past modeling, especially in 2010.
The goal of a pundit is to espouse a specific opinion. But the goal of quantitative analysis is to provide an accurate statistical portrait of a given situation and project that into the future. Analysis of a presidential race is not a study that occurs in a vacuum, rather everyone will know whether predicted outcomes came to fruition in a week.
I think arguments concerning the use of polling to shape the narrative regarding the state of the race miss that only a small class of political nerds like myself (on both sides) really look at this stuff.
What I am most curious about but haven't seen addressed is the selective poll inclusion by RCP. Why do TPM and 538 include polls that RCP leaves out? How does RCP determine which polls to include and/or keep track of on its site? Comparing quantitative analysis only works if everyone is using the same base set of data. To the extent that there is data selection occurring I wish that was discussed further.
#5 Posted by JB, CJR on Tue 30 Oct 2012 at 03:40 PM
The Colorado model is based on economic factors only.
Other than that, yes, totally the same thing as the rest of these.
#6 Posted by chris, CJR on Tue 30 Oct 2012 at 03:44 PM
Okay, so even if you fold in the Colorado model, you have 6 predicting an Obama victory to 1 forecasting a Romney win. Brendan's thesis still holds: Silver is not an outlier. If anything, the Colorado model is the outlier.
I don't understand why partisans find it fruitful to attack forecasting models, particularly by social scientists. What would we stand to gain from being wrong? Why do you assume we're toiling away in liberal Ivy Towers trying to cook the numbers? It's just silly.
#7 Posted by Brice, CJR on Tue 30 Oct 2012 at 03:51 PM
It makes sense for partisans to attack forecasting models and surveys that show their side losing. Being an apparent winner supposedly conveys a certain amount of momentum. Being the apprent loser may discourage campaign workers. And, supposedly, some undecided voters will prefer to be on the winning side.
#8 Posted by David in Cal, CJR on Tue 30 Oct 2012 at 06:18 PM
As George Soros points out, systems involving people are unlike other physical systems. Other systems are not self-reflexive. In other words, they do not read what is written about them and change their behavior because of what they have read.
Perhaps sites like 538 are taking this into account.
If Romney were to win, how could you judge statistical analyses? If they do give him a 25% chance of winning an event that will only happen once, does his winning or losing tell you anything?
#9 Posted by Steve13565, CJR on Tue 30 Oct 2012 at 07:23 PM
Paraphrasing: If Silver gives Romney a 25 percent chance of winning an event that happens only once, does Romney winning or losing tell you anything about Silver's model?
Actually, it does tell you something because the model's results are more detailed than Romney or Obama winning or losing. On the presidential race. Silver has specific, numeric probabilities for the vote in every state (or at least the battleground ones where there's regular polling). He also is doing separate projections for all of the contested Senate races and for the Senate as a whole.
There will thus be many, many numbers and results after the election to compare with his posted projections state by state, in both the presidential and Senate races. Recall that he correctly called 49 of 50 states in 2008 (thus, state by state) and also had very strong predictions in 2010 for the House races.
#10 Posted by Fairfax Voter, CJR on Tue 30 Oct 2012 at 08:56 PM
The problem with Nate Silver's work is the same one quantum physicists have - a variation on the observer effect.
Ideally, Nate's little black box would hum along in a vacuum, pulling in the data, weighting it appropriately, adding in whatever other factors Nate felt appropriate, and spitting out a reasonable projection of the outcome of the election. Keen-eyed readers will spot large elements of Nate's selection bias in the structure of the black box, and that of course is where many who haven't been on paid retainer to the Obama campaign during their careers get suspicious, but all that's essentially moot: because Nate's black box isn't in a vacuum at all, it's in a goldfish bowl.
Mr. Silver finds himself the sine qua non of projection models - an authority, in fact. Authorities that make assumptions about the world tend to influence other authorities to make the same assumptions, and everybody tends to start to see the world through the prism of those authorities - this is how the Pope had everyone believing the world was flat and the sun orbited around it, although there was that pesky Galileo and his "eppur si muove" schtick. While there are certainly plenty of people merely disgruntled at Mr. Silver's conclusions, a decent number of his critics instead cast doubt on his assumptions - the built-in bias for Mr. Obama's legendary ground game, his incumbency advantage, the disregard for party affiliation in polling samples (yes, I've read Silver's explanation that this is apparently so volatile that we can treat an electorate that claims to be 40% Democratic as if it's the one that will show up at polling time, even though there is a dramatic difference between Democratic and non-Democratic support for the President). This doesn't make us mathematical dunces, any more than the comforting support of people who share Mr. Silver's assumptions makes him a prophet.
#11 Posted by Mojo, CJR on Tue 30 Oct 2012 at 09:42 PM
Let me get this straight, some people look at national polls and are happy that Romney is performing reasonably well and conclude that he has a good shot to win the election. When somebody takes those polls, and more importantly state polls, into account and concludes that Obama is a favorite to be reelected they get angry that anybody would dare draw conclusions based on polls?
That doesn't make any sense. 538 predicted a healthy R win in 2010 and was more or less correct. Where was the "bias"? Where was the obvious liberal slant? Oh, so in 2010 it was obvious, but now he is biased?
538 is nothing more than an attempt to digest as much information as possible and put a percentage on what is likely to happen. How likely is it to rain tomorrow at your house? There are a million variables and ultimately it is impossible to know for sure. But some people have made a living out of estimating the best they can. This is a similar endeavor.
If you choose to get bent out of shape about it that says more about you than about fivethirtyeight.com
#12 Posted by Elliot, CJR on Wed 31 Oct 2012 at 01:46 PM
Mojo:
1) Please help this "less keen" observer understand what the selection bias rather than simply referring to it vaguely. If it is a black box at 528, which I think it is, how does one go about observing bias in the model?
2) One way or another, we will find out who is right/wrong on 11/7.
#13 Posted by Niels, CJR on Wed 31 Oct 2012 at 02:16 PM
@mojo
There is no selection bias on 538. That's B.S. from those that choose to not believe the probabilities and want to sound like they know something. Again, it's pure nonsense disguised as intelligence or inside information.
Nate Silver's model was designed in a vacuum, using poll aggregation and economic factors, poll weighting based on past performance, and other factors Nate is happy to supply. The model pre-existed Obama even running for president, back in 2007. I distinctly recall feeling nervous when the model showed McCain leading back on the old blog site.
The only people who distrust Silver's model and are ready to discount it if Romney wins, are either innumerate, partisan, or purposely attacking Silver's character to prop up their own false beliefs. I suspect a combination of all three, and part of a coordinated effort to prevent an Obama landslide when last minute voters usually decide to side with the winner.
#14 Posted by Magic Underwear Mocker, CJR on Wed 31 Oct 2012 at 04:11 PM
Silver might be a target because the vote machines might be rigged in favor of Romney & other Republicans. Not saying they are, just that they might be. And if they are, then it would have to be part of the Republican strategy to cast doubt on honest polls and poll aggregators like Silver. Otherwise, it'd be too easy to cast doubt on the actual results when they defy all the polls.
Current Harper's has a piece on the voting machinery, its ownership, its suseptibility to tampering and the impossibility of talking about all this in the MSM. If anyone's interested in exposing yourself to "conspiracy" charges, I recall Ronnie Dugger was digging into this back around 1992.
#15 Posted by Edward Ericson Jr., CJR on Wed 31 Oct 2012 at 04:59 PM
"If Romney were to win, how could you judge statistical analyses? If they do give him a 25% chance of winning an event that will only happen once, does his winning or losing tell you anything?"
Here's a simple analysis:
Take the vote tallies in each State predicted by the model and subtract the actual results.
Divide these numbers by half the width of the 68% confidence interval given for that state's results (this is, for exampe, the 3.2 in +/- 3.2 quoted for Florida, for example)
If the model is unbiased and estimating uncertainties properly, the distribution of these 50 residuals should have a mean near 0 and an RMS spread near 1. (Here "near" means within 0.14 or so). In a statistically well-behaved perfect world, it would fit to a Gaussian curve with mean 0 and width 1 very nicely.
Unfortunately we only have 50 states to work with, or else you could get a clearer picture. If you add the senate races you could get more information, but the senate models are different if I recall correctly. In any case, this is the simplest such analysis you could do.
#16 Posted by Brian Hamilton, CJR on Wed 31 Oct 2012 at 05:01 PM
More interesting yet to compare an econometric model with a model based on analysis of probabilities,
#17 Posted by Ximena, CJR on Wed 31 Oct 2012 at 05:52 PM
As Simon Jackman (HuffPost Pollster) has pointed out, the poll consensus may be off, but they are just as likely to be off in either direction historically. Accounting for this possibility reduces Obama's odds from 9:1 favourite to 7:3:
http://www.huffingtonpost.com/simon-jackman/converting-a-poll-average_b_2044222.html?ncid=edlinkusaolp00000008
At this point the most interesting intellectual exercise to determine which quant model got the election right 'first':
http://evmodcast.wordpress.com/2012/11/01/model-differences/
#18 Posted by belegoster, CJR on Wed 31 Oct 2012 at 08:41 PM
Nate's methodology is pretty well explained on Wikipedia. And probably in his book too which I have not read yet. However the database of his coefficients is not publicly available and he is loath to give it out, putatively because of copyright worries, but possibly other reasons - perhaps there are signs of partisanship in there, perhaps he sees it as a valuable trade secret that he wants to keep. Certainly if you could download the whole framework kit-and-kaboodle, them people would do so and get busy tweaking the coefficients to find a model more to their liking.
I imagine a tweak-able "Open Source Election Model" is inevitable now. The next election will probably have dozens of "Nate Silvers", most - if not all - of them considered partisan. And maybe that is a good thing since Nate's model takes a bit of the suspense out of this game - at least until we have a case where it is systematically and spectacularly wrong which seems rather unlikely.
#19 Posted by MikeWise, CJR on Thu 1 Nov 2012 at 03:36 AM
It isn't Pundits vs Probabilities, it's Polls vs Probabilities.
What do viewers/readers want to know? "Who's winning?"
Every Big Media outlet shoves its Poll data at the viewers. They do so in a percentage format, like probabilities. Note that when Nate's numbers show a 2% difference in the poll numbers, its a 45% difference in the Probabilities. Using Poll % to answer a probabilities question is patently misleading. And misleading to make races look closer and more dynamic than they really are. To sell newspapers, to sell TV minutes.
No media outlet (except the NYT) builds models to turn poll numbers into probabilities. They meekly hide behind the "We report, not predict" Yet the polls don't REPORT public sentiment, they use math models and smart people to turn 800 responses into PREDICTIONS on how 100 Million voters will vote in the future. Making a model which yields Probabilities, not "most likely popular vote" is simply just another model.
This isnt pundits vs Probibilities. It's Big Media (except NYT) packaging polling numbers in order to make people think a 40% difference race is a 2% race in order to sell more tickets to the horse race.
#20 Posted by TerryWynn, CJR on Thu 1 Nov 2012 at 10:51 AM
It will be interesting if, after the election, the same disgruntled pundits who are criticizing Silver now will make public mea culpas when he is right, again. It's doubtful since nobody in the MSM ever thinks of calling out these blowhard know nothings.
#21 Posted by Joseph Brown, CJR on Thu 1 Nov 2012 at 02:10 PM
Mojo's pulli pseudo-ideas out of his butt. William's just the usual (sadly, it's now "usual") wingnut troll who comes to CJR.
Seriously, William, is there so much conservative SuperPAC money sloshing around that people like you get paid to monitor websites like this and try to "work the refs"? #fail
#22 Posted by socraticGadfly, CJR on Thu 1 Nov 2012 at 10:41 PM
Oh, and that's not even a poll that Will posted. It's simply a statistical modeling. Double #fail.
#23 Posted by SocraticGadfly, CJR on Thu 1 Nov 2012 at 10:47 PM
Neither Obama nor Romney winning has anything to do with Silver being 'right' or 'wrong'. In fact, building statistical models isn't about being right or wrong about a particular outcome, it's about using mathematics to accurately and precisely gauge the likelihood of a particular outcome. Thus, even if Obama wins (or loses), Silver's model will be 'wrong' in the sense that it will always need correction and refining. To show this, though, you'd have to challenge him on the algorithms he uses in his model--i.e., the variables, weights, polls, and other information he includes. Romney winning will not necessarily mean anything to Silver other than he owes Joe Scarborough some cash.
#24 Posted by Seth, CJR on Fri 2 Nov 2012 at 10:20 AM
Using the words 'right' and 'wrong' to describe Nate Silver's approach, analysis and predictions forces his work into a dichotomy - which it is not. Mr. Silver carefully reports probabilities, which are continuous, and on a ratio scale. Most election viewers and participants ask "who is going to win?" not "who is more likely to win?" Consequently, they accuse Silver of equivocating and vagueness. Or they infer a yes/no prediction, then accuse him of bias or deny the prediction inferred from his results. "Media" reporters would do everyone very well if they would understand what a probability means, and help their audience draw an inference from probability to a dichotomous outcome. Mr. Silver will certainly make more predictions (he called 2 big ones in a row; one more and he'll be labeled a genius!); I anticipate that a lower probability candidate will sometimes win. But I won't throw out his model or his work.
#25 Posted by Jay Warner, CJR on Tue 13 Nov 2012 at 10:40 PM