Polling Methods

By Sally Raskoff

Are polls getting less reliable? Some say so. Our changing technology, including social media and the 24-hour news cycle, can have an effect on opinions and behavior. People who hear that a candidate already has a good lead might change their opinion or not show up to vote. On the other hand, some estimates of the error rates of polls suggest that they are somewhat stable as pollsters change their methods to adapt to society's dictates.

Or are polls still pretty accurate, taking the pulse of the people? Some say so. If the difference between opinions is huge, then a poll can certainly pick that up. However, if opinions are just a few points apart, polls may not be able to show those differences accurately.

Basic knowledge of social research methods and statistics can help us sort through these opposing views. Karen Sternheimer's blog on the Margin of Error gets us started in how to interpret the numbers they report. She discusses how polls are taken from a smaller group (a sample) who are actually surveyed to estimate what the larger group of voters (a population) might do.

Here are a list of things to consider about the quality of the sample and how confident we can be that it does actually represent a larger population.

1. Consider the source. Who did the poll? Are they a reputable polling agency? Are their research methods visible and available to assess? If not, beware.
2. Is the sample drawn from a random sampling process? If it is not random, one cannot assume any findings can be generalized to the larger population. Truly ensuring random selection means that every person in that larger population has an equal chance of being selected into the sample. Random sampling occurs in one's everyday life with the shuffle function on an iPod or mp3 player. Every song in your playlist might be played when you push the shuffle button. In research, we do the same thing – a random digit generator gives us those people to contact to ask about taking our survey.
3. What type of sampling did they do? Did they (randomly) sample cell phones? Landline phones? A mailed survey? Was it an online survey or face-to-face interviews? If online or in person, how did they find them? Standing in a mall with a clipboard and asking every fourth person who walks by is not a random process. If you're talking a political poll, how did they find "prospective voters"?
4. Assuming random sampling was used, was the sample big enough? The law of large numbers states that the larger the sample, the better the mean of the sample will represent the mean of the population. The margin of error gives us an estimate of what the population figures might be – and it is based partially on the sample size.
5. What is the response rate? How many people contacted actually answered the survey? The response rate for polling is down to 10 percent or less, compared to approximately 30 percent from a couple of decades ago. If weighted and/or proportionate sampling is done, where the sampling is done to purposely represent groups in society who may be missing, many agree that that can save the poll from being non-representative. According to the Pew Research Center, the participation rate is low and declining. This suggests even more the importance of random sampling and getting a large enough sample even as a larger number of people will have to be contacted to get a sample large enough, thus increasing the costs of polling.

If those who answer the survey are similar to those not answering, then the poll results will be representative. However, if they are not, the results can be problematic.

How will we know this without a keen look into the methods? With political polls, after the election! After which, by the way, some might have been persuaded to change an opinion or behavior due to erroneous poll results.

Keep your eye on the methods used and whether or not you can have confidence in how representative the survey results may be. And try not to let poll results change your mind or actions.

Comments

[paul]

Thanks a lot for the insightful narrative the most important part is and i quote
'Who did the poll? Are they a reputable polling agency? Are their research methods visible and available to assess? If not, beware'. very true because some agency are being bribe to say what is really not on the ground.

[Geoffrey]

Informative article!!

[Petero]

Polls can be misleading if not done professionaly