Quick Question

Are phone polls accurate?  

The quick answer:  They are accurate for those that respond, but the crucial question is how many actually responded.

The longer answer:   Today, telephone polls have become a standard part of news reporting. Every scandal, major news event, or change in direction is heralded by an “instant analyses” of who is up, who is down and “What America thinks”. Often, the findings from these polls take on a life of their own, framing what people think about an event and shaping their response to it. Yet by social scientific standards these findings are often of poor quality, or worse, are misleading, representing what an outspoken minority feels rather than mainstream America.

Information from these polls is popular because it is relatively quick and inexpensive way to put the survey sponsors in the position of being knowledge brokers. In effect they have “created news.” It is in their best interests to present the information in a way that shapes headlines. The results are typically reported in a pseudo-scientific fashion, stating that the poll was based on 1000+ adult respondents in a national sample with a margin of error of plus or minus 3%.  

What they don't tell you

The American public has been trained to assume that a national poll of thousands with a small margin of error accurately represents the opinions of the whole society, but is that true?  In fact, the sample size and margin of error give no indication as to the more important information about whether the results accurately reflect the population it has surveyed.  The margin of error simply reflects how well the responses cluster around an average score. 

To determine how well the study results describe the target population, one needs to look at the response rate.  The response rate refers to the total number of persons contacted to obtain the reported number of valid results. A high response rate in a random sample offers higher confidence in the results because the answers are more likely to be representative of the population. In the social sciences a 85-90% response rate is considered desirable.  Confidence in the results declines proportionally as the response rate gets lower. Below 50% and the survey is considered very dubious since it means that 50% of the people contacted refused to participate in the survey.   

I recently came across a poll published by LeMoyne College and Zogby International, a reputable polling firm. The report stated that “One in five Catholics say a priest in their local diocese has been accused of committing child sexual abuse”[1].  Curious about the reliability of the data I tracked down the Director of Communications for Zogby and asked him about the response rate for the study.

The communications director reported that to get the 1,508 completed responses in the study they called 41,033 people. In other words, the response rate was roughly 1/3 of one percent. Even assuming that only 25% of the people called were Catholic, and therefore able to provide valid responses, in the social sciences this outcome would be considered garbage. The respondents to these polls hardly represent the average American, rather they represent the tiny minority of people who do not hang up on those annoying telemarketers who call you during dinner. 

It makes you wonder, whose opinions do these polls really reflect? While not all telephone polls are equally unreliable one rule of thumb is that those who do not report a response rate should be treated with a high degree of skepticism. Yet until the public stops believing in them, telephone polls will be tempting to those who, for a few thousand dollars, wish to see themselves as knowledge brokers.

**********
[1] LeMoyne College.2002. “Spring 2002 LeMoyne College/Zogby International Contemporary Catholic Trends Poll Report”. www.lemoyne.edu/academics/spring02poll.htm Accessed 1/13/03

This research note was written by Patricia Chang, Assistant Director of the Boisi Center for Religion and American Public Life and Associate Research Professor of Sociology, Boston College.  More information on Dr. Chang.

For more information on statistical reporting, visit our statistics and poll data page in the Congregational Resources section of this web site.

Return to the Quick Question archive