What went wrong with '08 primary polls
University survey experts working with the American Association for Public Opinion Research have identified several reasons polls picked the wrong winners in the 2008 presidential primary.
The study is believed to be the most comprehensive analysis ever conducted of presidential primary polls.
"The most jarring element of the presidential primary polling was that polls picked the wrong winner in New Hampshire," said polling expert Michael Traugott, who chaired the AAPOR committee composed of leading academic and private sector experts in public opinion and survey research.
The committee's analysis shows that a handful of methodological missteps and miscalculations combined to undermine the accuracy of predictions about presidential primary winners in New Hampshire and three other states.
One source of error the researchers were able to eliminate was the so-called 'Bradley Effect,' in which people say they support a black candidate in order to appear unbiased, but then cast their ballots for a white candidate in the privacy of the voting booth.
"Many New Hampshire polls predicted Barack Obama would beat Hillary Clinton in that state," says Traugott. "So when Clinton won, some people pointed to latent racism as the reason. But in the data we have from a wide variety of New Hampshire pre-election and exit polls, we found no evidence that whites over-represented their support for Obama."
For the report, supported in part by a grant from the Institute for Social Research (ISR), Traugott and colleagues analyzed individual, household-level response data provided by seven polling organizations.
"This analysis suggests some important explanations for the errors in the 2008 New Hampshire presidential primary and raises significant questions about pre-election polling methods," said Richard Kulka, AAPOR president.
"The materials we received from polling organizations showed that there was much more variation in the methodology of pre-election polls than I ever imagined there would be," Traugott said.
The committee analyzed poll performance in four primary states: Wisconsin, South Carolina, California, and New Hampshire. Although the limited data they received made it impossible to conduct definitive tests of all likely sources of different poll performance, the following factors were identified as the most likely reasons the polls got it wrong:
• The New Hampshire primaries occurred only five days after the Iowa caucuses, truncating the polling period in New Hampshire.
• Most commercial polling firms conducted interviews on the first or second call, but respondents who required more effort to contact were more likely to support Clinton. Instead of reworking their initial samples to reach these hard-to-contact people, pollsters typically added new households to the sample, skewing the results toward the opinions of those who were easy to reach on the phone, and who typically supported Obama.
• Non-response patterns, identified by comparing characteristics of the pre-election samples with the exit poll samples, suggest that some groups that supported Clinton were under-represented in pre-election polls, possibly because they were more difficult to reach.
• The order of candidate names on state primary ballots likely contributed to increased support for Clinton in New Hampshire, where her name appeared near the top of a long list of names and Obama's appeared near the bottom.
Traugott noted that the analysis also revealed wide variation in the primary question respondents are asked the so-called trial heat question about which candidate they prefer in the coming election.
In New Hampshire, there were 11 different question wordings used in the Democratic primary, and 10 different wordings used in the Republican primary. In some versions, the candidates' names were not mentioned at all. In others, only the "major" candidates were named. Some polls randomized the candidates' names.
ISR survey research expert Robert M. Groves was a member of the AAPOR committee.