Factchecking made great strides during the 2012 campaign, but were those advances compromised by the pressure to maintain partisan balance?

Two respected Washington think tank scholars say yes. Thomas Mann of the Brookings Institution and Norman Ornstein of the American Enterprise Institute, who have recently argued that Republican extremism is to blame for many of the pathologies of Washington, told The Huffington Post’s Dan Froomkin last week that journalistic factchecking of the 2012 campaign may have been counterproductive:

Mann and Ornstein said that in practice, the fact-checkers may have made things worse rather than better.

“We had these little flurries of fact-checking—which I found not worthless, but not a substitute for coherent, serious reporting—and most of the time it just got stuck in the back of a news organization’s output and there was no cost to a candidate of ignoring it,” Mann said.

And then there was this terrible irony: “Fact checkers almost seemed obliged to show some balance in their fact checking.”

“There was some damn good stuff done, and stuff that really did hold Romney to account,” Ornstein said. But no fact-checker intent on “appearing to be utterly straightforward, independent, and without an axe to grind, is going to actually do the job of saying that we’re going to cover 20 fact checks on one side, to three on the other.”

So, Ornstein concluded: “If you looked at where the scales should have been, and where they were, they were weighted. And they weren’t weighted for ideological bias. They were weighted to avoid being charged with ideological bias.”

However, Mann and Ornstein’s factual premise about the factcheckers seems to be flawed. As Mark Hemingway pointed out in The Weekly Standard, PolitiFact actually has tended not to be balanced in the ratings they assign; neither has Glenn Kessler in his Washington Post factchecker column. Both have tended to give more negative ratings to Republicans (and have been accused of having a liberal bias for doing so!). Perhaps Mann and Ornstein believe that accurate factchecking would be even more asymmetric, but their statements suggest incorrectly that the ratings were artificially balanced.

The underlying problem here is the difficulty of definitively establishing which side is “worse” when it comes to taking liberties with the truth. While there are statistical methods we can use to compare polarization in Congressional voting between Republicans and Democrats, it is less clear how to determine whether one side is more irresponsible when it comes to factual accuracy. Some have tried to do so using ratings from the factcheckers. However, the process by which statements are selected to check is not random, so the evidence is only circumstantial—differences in average ratings between the parties don’t necessarily establish that one side is worse. In addition, while the factcheckers work quite hard to apply their ratings consistently, the categories are inherently subjective, which makes it difficult to rely on them as comparable indicators of accuracy. And even if one accepted both the factcheckers’ selection process and their ratings, it would still be reasonable to argue that statements in question and the issues they concern vary in their importance.

More fundamentally, it’s not clear what Froomkin or Mann and Ornstein would want factcheckers to do differently. Specialized factcheckers should not be expected to address the question of which side is generally less accurate—the question is simply beyond the scope of the format, which focuses on assessing the accuracy of discrete statements, not making sweeping generalizations about the state of American politics. Even if the factcheckers agreed with claims like those made by Mann and Ornstein (which they’ve generally disavowed), it’s not clear what they could or should do differently. Factcheckers are being asked to maintain a commitment to following the facts wherever they take them while also maintaining a reputation as non-partisan and fair. Of course, they should be fearless in following the facts in any given case, but focusing on asymmetry would destroy their credibility as honest brokers while adding little to the value of their work. Indeed, it has the power to create perverse incentives—should factcheckers stop factchecking whichever side is less bad at any given moment? Why?

Brendan Nyhan is an assistant professor of government at Dartmouth College. He blogs at brendan-nyhan.com and tweets @BrendanNyhan.