“All the methodological concerns aside, at the end of the day, like any business, there are honest and dishonest brokers, and it takes a bit of time to figure it out,” says Cohen.
The Washington Post’s unit—in some ways modeled after Langer’s ABC shop, where Cohen used to work—also designs and commissions the paper’s own polling, where, since the paper can control every decision and step, even more exacting standards are imposed than those the Post applies to third party research.
AAPOR, one of the polling industry’s most prominent trade organizations, has wrestled since its founding—and largely passed on the opportunity—to set or impose standards, according to Peter Miller, a Northwestern professor who recently served as AAPOR’s president. Instead, they’ve opted to encourage polling organizations to release information about how they conducted their polls—information that not only would give poll consumers a basis for evaluating their work, but that theoretically could be used to replicate their results.
Miller stepped down in May, but is still helming AAPOR’s “transparency initiative,” the organization’s newest effort to encourage such disclosure.
The planned initiative is a step beyond AAPOR’s long standing code of ethics. This document calls on polling organizations to release for any public poll fifteen points of information upon the poll’s publication, or within a month of any request. This includes the survey questionnaire, information on sampling, the study’s funder, and screening procedures.
But there are very rarely any consequences for failing to live up to the code’s requirements.
“By our rules, we respond only when a complaint is made,” says Newport, the association’s current president. ”We don’t have a committee that sits and judges all the polling out there.”
While Miller says that many complaints on incomplete compliance with the code’s disclosure requirements are settled behind the scenes, the organization has only publicly censured three pollsters for lacking disclosure in the last twenty years or so—in the 1990s, Frank Luntz; and in 2009, an Iraqi mortality researcher and Strategic Vision.
“It’s not a lot,” concedes Miller.
The transparency initiative seeks to reverse that order. Rather than wait passively for complaints about pollsters’ non-compliance with AAPOR’s transparency requirements, and issue rare censures, AAPOR will instead allow pollsters who proactively disclose key aspects of their methodology to claim that they are operating in accordance with the initiative—offering something of a seal of approval for disclosure, if not for methodology or results. Miller is hopeful that by setting up an infrastructure that makes disclosing the information easier, it will encourage more pollsters, especially those who haven’t done so regularly in the past, to regularly share information about their methodologies.
“If you do sponsor a poll, one of the things you could for is look at our participating organizations,” says Miller. “If they are one of the group who haven’t signed up with the transparency initiative, you might encourage them to do so, or you might think about what they are putting out in another way.”
The transparency initiative is a long way from being ready. At one point Miller had hoped to have it running in time for the 2010 elections—now the target date is next May’s AAPOR annual meeting.
Miller envisions a Web form that participants can use to upload their information to a central public repository. (They’ve reached out to the University of Connecticut’s Roper Center for Public Opinion Research, which stores polling data from a variety of pollsters, but no agreement has been reached.)
But before any pollster could start inputting and disclosing, there are a score of outstanding issues that AAPOR members who are serving on the volunteer committees tasked with organizing aspects of the initiative must sort out, including fundamentals like exactly what information participating pollsters will be required to disclose, and how quickly they will have to do so.
While Miller says the existing AAPOR code’s disclosure requirements provide a “template” for what the initiative could require, he says they may “go beyond that.” And in the wake of the Strategic Vision episode, Miller has decided that it will not be adequate to take pollsters at their word that their disclosure information is accurate. He plans on setting up some system to verify, perhaps through spot checks, what pollsters are self-reporting.
The goal of the transparency initiative is to bring more information to the polling market, not only so people can more easily judge the quality of conducted polls, but so that commissioning organizations can more easily compare what they might buy.