Jon Cohen does that sort of screening for polling and survey data that journalists at the Washington Post wish to cite.
The Post maintains a short black list of pollsters—which Cohen would not disclose—who, by Cohen’s judgment, operate with serious methodological deficiencies, or have committed “serial misanalysis,” on the scale that their data is banned from publication. Other blue-chip organizations’ polls are trusted enough to be allowed into print without additional vetting. For pollsters in between, Cohen maintains a checklist on the paper’s intranet that helps reporters determine poll quality, and he is often called upon to decide whether data meets the paper’s standards.
“All the methodological concerns aside, at the end of the day, like any business, there are honest and dishonest brokers, and it takes a bit of time to figure it out,” says Cohen.
The Washington Post’s unit—in some ways modeled after Langer’s ABC shop, where Cohen used to work—also designs and commissions the paper’s own polling, where, since the paper can control every decision and step, even more exacting standards are imposed than those the Post applies to third party research.
AAPOR, one of the polling industry’s most prominent trade organizations, has wrestled since its founding—and largely passed on the opportunity—to set or impose standards, according to Peter Miller, a Northwestern professor who recently served as AAPOR’s president. Instead, they’ve opted to encourage polling organizations to release information about how they conducted their polls—information that not only would give poll consumers a basis for evaluating their work, but that theoretically could be used to replicate their results.
Miller stepped down in May, but is still helming AAPOR’s “transparency initiative,” the organization’s newest effort to encourage such disclosure.
The planned initiative is a step beyond AAPOR’s long standing code of ethics. This document calls on polling organizations to release for any public poll fifteen points of information upon the poll’s publication, or within a month of any request. This includes the survey questionnaire, information on sampling, the study’s funder, and screening procedures.
But there are very rarely any consequences for failing to live up to the code’s requirements.
“By our rules, we respond only when a complaint is made,” says Newport, the association’s current president. ”We don’t have a committee that sits and judges all the polling out there.”
While Miller says that many complaints on incomplete compliance with the code’s disclosure requirements are settled behind the scenes, the organization has only publicly censured three pollsters for lacking disclosure in the last twenty years or so—in the 1990s, Frank Luntz; and in 2009, an Iraqi mortality researcher and Strategic Vision.
“It’s not a lot,” concedes Miller.
The transparency initiative seeks to reverse that order. Rather than wait passively for complaints about pollsters’ non-compliance with AAPOR’s transparency requirements, and issue rare censures, AAPOR will instead allow pollsters who proactively disclose key aspects of their methodology to claim that they are operating in accordance with the initiative—offering something of a seal of approval for disclosure, if not for methodology or results. Miller is hopeful that by setting up an infrastructure that makes disclosing the information easier, it will encourage more pollsters, especially those who haven’t done so regularly in the past, to regularly share information about their methodologies.
“If you do sponsor a poll, one of the things you could for is look at our participating organizations,” says Miller. “If they are one of the group who haven’t signed up with the transparency initiative, you might encourage them to do so, or you might think about what they are putting out in another way.”
The transparency initiative is a long way from being ready. At one point Miller had hoped to have it running in time for the 2010 elections—now the target date is next May’s AAPOR annual meeting.
Miller envisions a Web form that participants can use to upload their information to a central public repository. (They’ve reached out to the University of Connecticut’s Roper Center for Public Opinion Research, which stores polling data from a variety of pollsters, but no agreement has been reached.)