Thinking Like a Sociologist: Deconstructing Polls

By Karen Sternheimer

A recent episode of The Daily Show satirized both left wing and right wing cable news shows for using questionable poll data in their broadcasts. Regardless of your political orientation, this clip is a great introduction to some of the core concepts sociologists use in research methods and statistics: survey construction, sampling, and margin of error. Think about these issues as you watch the video below.

 

The Daily Show With Jon Stewart	Mon - Thurs 11p / 10c
Poll Bearers
www.thedailyshow.com
Daily Show Full Episodes Political Humor Healthcare Protests

1. Survey construction

Did you notice the wording of the questions in the polls? Good surveys should not suggest how respondents should answer them, but some of the polls included wording that could certainly bias responses. Note how some of the questions in this clip tend to encourage one response or another, particularly the questions about taxes. Most people would respond negatively to a general rise in taxes. But there are a wide variety of taxes (sales and income taxes to name a few), and rates vary based on a variety of factors (such as location, income level, and family size). By contrast, the tax that excludes most potential respondents (raising taxes only on a small percentage of Americans) is more likely to generate favorable results.

A good survey question is worded as neutrally as possible and is specific so that respondents are very clear about the meaning of their response. A good question about taxes would be clear on what specific tax rate would change, by how much, and for whom. The word “tax” is so loaded that another word might even be used in its place, such as fee or levy.

2. Sampling

Let’s say we’ve written good survey questions. Our next step is to administer our survey to an appropriate sample.

What is a good sample? That depends on which group we are trying to learn about. If we want to get the opinion of sociology majors from across the country, then we certainly wouldn’t want to poll people who pass through the center of a couple campuses. For one, we would get a lot of non-sociology majors, and secondly, a small number of campuses would not adequately measure the nation’s sociology majors.

Many polls reported on the news imply that they represent views of Americans. That might seem like a daunting task, but it is possible to get a sense of public opinion this way.

You might be thinking that with a population around 300 million, it is nearly impossible to sample a thousand or so people and claim to know what Americans are thinking. To do so, pollsters need to construct a probability sample, which means that every American should have an equal opportunity to be chosen at random to be part of the sample.

Hypothetically, you can think of a random sample as similar to putting everyone in the population’s name in a huge hat which is shaken so anyone’s name can potentially be pulled.

But there isn’t a hat that big, nor is it practical for someone to write everyone’s name down on a slip of paper. Pollsters often rely on measures which are admittedly imperfect. One of the most common is random digit dialing, where a computer calls random phone numbers in hopes of getting respondents.

Have you ever gotten one of these calls? If you’re like me, you get a lot of calls that interrupt your busy day or evening, and caller id allows you to ignore the numbers you don’t recognize. This has created sampling challenges: if there are consistent patterns in who is more or less likely to participate then your survey results might be skewed.

So there are no perfect samples, but there are some really imperfect samples, like the ones in the Daily Show clip above. Some internet and television polls make no attempt to create a probability sample, so their responses are limited to the demographic already watching or logging onto their site.

There are also other factors which might make someone more or less likely to participate. For instance, if a television show asks you to text your response, viewers might be more likely to participate if their cell phone plan has generous texting privileges. Beyond that, some people are more motivated to respond than others. So these polls might be fun, but they are not very useful gauges of public opinion. That’s what “results not scientific” means if they flash it on the screen. But often shows do not.

3. Margin of Error

Even the polls with well-written questions and the best possible samples are approximations of public opinion, rather than a perfect reflection of what people really think about an issue. While we might not ever be 100% certain that a probability sample reflects the true opinion of a population, thanks to probability theory, we can be 95% confident that the actual opinion lies within a certain range.

Here’s an example. Gallup regularly conducts job approval surveys of political leaders. In an August 2009 survey, they found that 31% of Americans approve of the job Congress is doing. As you can see from the graph below, the approval rating appears to have fallen from previous months.

But every poll has a margin of error, since surveys are approximations rather than perfect reflections of a population’s beliefs. The margin of error is a measure that takes into account the sample size and the amount of variation of responses. In this case, the margin of error is ±4, meaning that we can be 95% certain that between 27% and 35% of Americans approve of the job that Congress is doing.

Now, if I were a member of Congress I wouldn’t be too excited about this range of scores, but because it is within the range of scores from earlier months we probably cannot say with confidence that Americans’ approval of Congress is declining. But no matter how you look at it, it’s pretty low.

You might notice that news shows reporting polls rarely discuss the meaning of the margin of error, if they mention it at all. Keeping this important measure in mind, you can use your sociological imagination to think critically about the next poll you hear about in the news.

Comments

[ostrix]

Unfortunately, I didn't manage to open the video, even when I tried from the site itself... I agree that margin error is an important factor that is rarely emphasized. While reading this post, I thought to myself that it would be good to have read it when I was attending the course of Statistics in the university :) You won't believe just how many people watch these polls and haven't got the slightest clue about methods of statistical calculations, margin errors and things like that.

[amanda funaro 3]

I agree with ostrix about how many people do not have any clue about how polls work but I also want to point my finger at the media. If these big news companies and assuming the people who are conducting the polls are aware of their sampling errors then why air a poll that has 100% of the people agree? Is it to sway our opinion? Or is there really no one who disagrees that town halls are making a difference? Whatever the case may be Americans watching should be asking themselves how accurate these polls really are. Also they should ask themselves why do these news anchors look so "surprised" when 100% of the people who took this poll agree? Lastly, I agree that the wording of these polls are poorly constructed and the answers are pretty much bias.

[Emily]

Excellent post. I'm taking a sociology class and a lot of what I read applies to what I'm learning at the moment.
I conducted my own little expieriment a while ago and made a sample survey. It really is suprising how the wording of a survey can influence a persons decision, whether intended or not.
It's unbelievable how easily people can fall into these false ratings. What's even worse is that the media is the one who reports them. Does anyone do research to see if these polls are accurate?

[soc]

Way to really emphasize how much the media really skews things. I wasn't able to watch the video, but I was still able to get the general idea of what you were saying. This post related a lot to a survey I just did in my sociology class, dealing with margin of error and how much the wording of a question can effect the responses. Really interesting to see how small things that most don't notice can make such big differences, and it's sad to see the media rely on tactics like that to get ratings.

[Paulus]

I'm glad you addressed this issue in your writing. It is not only inaccurate over-all for people to try to influence the answers of people taking their polls,but it can often be down right deceitful. When a person creating a survey uses the wording of the question to suggest the answer that they want people to choose. I wish more people understood the significance of taking a poll, and that their answer should be 100% of their own will. Thanks for enlightening us on this issue.

[Julia]

Topic:

Hi, I am a high school sociology student just learning about all the meaning behind the numbers. Your post about the research methods of sociologists really helped me to understand the problems sociologists face as they try to formulate and conduct surveys to support a study. The example of the tax questions posed by the political parties portrayed how the format of a question can manipulate and skew the results, intentionally or unintentionally. I knew that the wording of a question could sway participants to answer one way or another, but I liked how you pinpointed the connotations of the very word “tax” itself. In this economic climate, any word associated with paying money has very heavy weight indeed.

[Jaclyn]

It seems to me that the media in most ways minipulates a lot of our thoughts whether it be through newspaper articles to even those that the government puts out for society to intake on their everyday lives. Simple things are very easily twisted to make the consumer misinterpret the actual study. "Surveys" are misread very often if not at all specific. Plus, it seems that they are misinterpreted when a person may hear or see one thing in the survey they do not like, and not really look at the true questions that are being compared or asked.

[Matt]

A good sociologist doing research will create an accurate survey with relevent questions. The polling audience should cover many ages and both genders. The wider the audience the more accurate your answers will be.

[Kristen]

This article gave good insight on how you should construct a poll, and how many companies DO construct polls the 'wrong' way. I like that it brings to your attention that polls are not always exact, and it also shows you how you can make them more exact. After reading this article I now realize that just because I hear a new statistic on TV or read one in a magazine it doesn't mean that it is accurate. If you want to find out if it is an accurate stat, you should find out what the "sampling" was and how the questions were asked.

[Missy]

This was an excellent post, I'm glad that you brout up wording of a survey, this can greatly affect a persons decision. I even learnd some things i didnt know, like a lot of theese posts I'm in a sociology class. I will defenintly benefit from reading this excerpt, thanks so much.

[alena]

Great post! i'm taking a sociology class right now and just finished a lesson on statistics and sociologists doing research. You made alot of excelent points in your article that alot of people probably never thought about. I think alot of people are unaware of how much effort is needed in creating an acurate survey, and you explained how important that is.

[Sam]

Wow! I want to just point out two things.
1. People need to learn to make polls without scewing other peoples answers BEFORE putting them on the air. It just makes them look like idiots and I don't think I will be able to trust a poll like that in a long time. and..
2. Jon Stewart is halarious.
The wording of a poll also has alot to do with someones answer. I'm glad you pointed that out because it's a need to know. Great job on this post Karen. :)

[Brian Clark]

I would have to agree. I am surprised at the data that news networks and publications use. Some of this data is collected by people who try to say things that changes people's actual answer. I'm in a high school sociology class and one of our projects was to survey ten different people and one problem I ran into was word-for-word asking the same question to every person, or elaborating for someone who didn't "understand" the question. Things like this can easily skew the data for any kind of survey.

[Katelyn]

This blog entry does an excellent job at illustrating how important the set up, and use of surveys are. If one word is off, or a certain group excluded, the survey can display completly different results. This is why when reviewing a survey, one must look over the details.

[Sally]

This blog entry shows how things can go good or bad in a survey. Sociologists have to be careful how they word answers or show emotion to one answer rather than the other. It was really interesting to see how the media does their polls, especially the one where people said 100% yes to a question but the text number was the same. I'm happy I decided to view this because now when I do surveys I will be extra careful not to give any kind of influence.

[Shelby]

Great post! I'm a high school student that's taking a Sociology class. So I know about polls and surveys already with that class and other classes I've taken in the past. But this really shows me how the results can be effected by the littliest things such as the way you word it. Also, it shows me how survey or poll results can never really be 100% accurate.

[Aaron]

This really shows how easily survey results can be skewed by the wording of the question. I'm in a sociology class right now, and just learning how to conduct an effective and accurate survey. These guidelines have to be followed if you want an accurate picture of a group's general opinion.

[Abbey]

This post was very informative and interesting. As a high school sociology student learning about surveys I found this blog entry to be very eye opening. It showed me how hard it is to actually create a good poll with no biases. It even comes down to little details like word choice-using "tax" or "fee" can potentially change the way someone might answer a question. The results can become very skewed if you don't follow the guidelines listed above.

[Jennifer Martinez]

It is important to use your sociological imagination to think critically about poll data you see featured in the news so that one could see how certain worded questions lead to certain answers; even though some questions have the same topic. It also helps to see how people react to the questions. The poll data may reflect one thing but it does not provide enough information on what the options are for answering the questions. Most questions are close- ended questions or leading questions, so the response is limited. The Daily Show touched upon three main concepts that sociologist use in research methods and statistics: survey construction, sampling, and margin of error. These concepts are well explained by the author of the blog. It helped to think of the topics while watching the clip because that is how a noticed the wording of the questions and the results of the responses. I also agree that it is not reliable when TV shows ask to answer a question by text because not everyone have access to answer the questions and not everyone is seeing the show at that precise hour. So the sampling may vary amongst the audience and their technology.

[Luisa Ter-Adamyan]

In my opinion,It is important to use our sociological imagination to think critically about poll date featured in the news, because it makes us understand the questions more clearly and also it helps us examine in a more complex way, then we would do if we were home just blankly staring at the television screen. Also, the poll data might not be formated clearly and therefore it helps us see that the questions aren't stated very well and it affects the answers we will give. The Daily Show clip touched up on the three core concepts that were already mentioned in the articles and they are, survey construction, sampling and margin of error. It's good that the article mentioned these three concepts before the clip started, because it helped me realize how the news anchor formulated his questions and the results of the poll.

[Jana Bangayan]

It is important to use your sociological imagination to think critically about poll data you see featured in the news because it helps polltakers understand questions more. By rewording the question or asking a question in a different way tends to change the answer of the polltaker. Whether the question is answered with a yes or no the questions asked does make a change in the survey results. Although questions may seem similar, by rewording the questions, people might have a different answer. The three core concepts that sociologist use in research methods and statistics that the Daily Show clip touched upon was: survey construction, sampling, and margin of error. I like how the article explained how wording questions is important! There are many surveys that are bias these days.

[Janet Leos]

It is important to keep in mind the sociological imagination when viewing polls because some polls are bias and we as people can be bias when answering questions. Society affects an individual and vice versa, therefore the manner in which a question is worded may resonate one way with one individual and differently with another. There are many factors that contribute to the already existent margin of error that may occur while taking a poll, and better understanding your audience can decrease the errors in polling. The three core concepts the Daily Show touched on was: surveys, margin of error and sampling.

[Nancy moxica]

It is important to keep track of everything you do in the sociology world all details and places are important in this field wording questions are also important to have and the poll is something reasonable to me.

[Estelle Robinson]

When viewing polls and survey data, it is important to think critically. For one, the idea of being able to think from both sides is an advantage that will in fact help to understand the possibilities better. You should always think about the source. Are they conservative or liberal? Which side are they more inclined to agree with? How could their political stance influence the polls? If you understand and ask these questions, you are more likely to think critically instead of going along with the data just because it sounds good.

[George Moore]

Real excellent writing on the subject. Sociology is a very important subject and the polls regarding it are all important as it addresses some great issues. I think it will be great for more people to understand the significance of taking a poll. In sociology it is important to keep track of everything you do.

[Polls and surveys]

An informative article which is so important to know..!! I hope that this is the post which is so helpful to us. I am really so pleased to get this post article very much. Keep it up..

[Jeffrey Haro]

It is clearly important to use one's social imagination to think critically about poll data that we see featured in the news because as explained in the video, news broadcasting is shaping the public's opinion. The questions that they ask are worded in a certain way in order to make the audience agree with the point of view that the news station believes. It is also important to use our social imagination when analyzing polls in order to come up with our own opinion.

The three core concepts that sociologists use in research methods and statistics are survey construction, Sampling, and margin of error. Primarily Survey construction is when the wording of the question in order for a survey to be good the question should be as neutral as possible. Some of the questions in the clip tended to encourage one response or another. Secondly is Sampling, in order for a survey to be accurate it needs to construct a probability sample. A probability sample should give every American an equal opportunity to be chosen at random to be part of the sample. In the clip what was happening was that the viewers were already fed their version of the truth by the media. So when the question is asked only the viewers of the show are responding so it creates inaccurate information. The third thing that sociologist look at is the Margin of Error, even if the poll has a well written question and the best possible samples polls are still only approximations of public opinions. Using the probability theory a poll can be up to 95% accurate. The margin of error is a measure that takes into account the sample size and the amount of variation of responses. When watching the clip I noticed that none of the polls listed the margin of error.

[Naeri Hakopian]

I really enjoyed this article because it pointed out some things that people don't usually think about. The three main points, survey construction, sampling and marginal error are something people don't consider where they see those statistics on the News. I think the only reason their statistics are so skewed is because they have a specific audience who agree with them and watch their show, and these people are all going to agree with one another.

[Amber Santos]

It is important to use your sociological imagination to think critically about poll data you see featured in the news because they usually give their point of view and try to persuade you to agree with them. It's better to research the information before just listening to someone's beliefs.
The three core concepts that sociologists use in research methods and statistics does the Daily Show clip touch upon was surveys, margin of error, and sampling.

[Aparrett]

It is important use sociological imagination to think critically because this enables to us to see both sides of the story and remain neutral. You need to think about who is actually participating in the polls. For example on taxing the wealthy...I wonder who mostly watches this show, and participates in the poll? Clearly it was almost unanimous that they wealthy should be taxed more....in what financial bracket do the watchers do the Daily Show fall? Are the viewers mostly middle class? The Daily show best demonstrates how inaccurate polls are due to the inaccuracy of the people taking them, as well as the companies/parties providing the polls. For example, the insurance poll provided my an insurance company, obviously they would ask a question in such a way that they would benefit from - get good results. I also noticed results varied by network. Regarding the use of social networking...keeping an un biased mind - While I don't care for political compaigns, or the media for that matter using social media to advance themselves, we have to be honest...Social media is where it's at. More people are on social media more than any other means. Thanks to cell phones people have access to social media /news/ campaigns 24/7 no matter what you are doing - no matter where you are.

[Hannah Wilson]

In my opinion, I think its important to sociology imagination to think critically about polls in the news because it makes us understand the questions more clearly. One concept that the show represents would be survey. And polls just don't reflect public opinion, it shapes it too. We try to learn who commissioned the poll and see if they are promoting anything.

[Hannah Wilson]

Amber you made it very clear about sociology imagination and the polls. And you were right about the concepts in the show.

[Sarah White]

Amber points out that using sociological imagination is essential when forming opinions about what the media presents. I agree. I think often people do not take the time to research the information and as a result are often led in directions they may not ordinarily go due to factors such as leading questions. I think often, based on the media's agenda-who they support, what issues they support, ratings, etc.-that the media leads the public with questions to shape the view of the public in their favor.

In the daily clip, it is evident that polls constructed are not following scientific method protocols. It is clear that the survey construction is in question. As aforementioned, questions that lead the public are dangerous, as it leads responses in a favorable direction to those that constructed it-that is if they benefit from such methods which happens in the media. Moreover, questions are often vague. "Tax" can imply a number of meanings used in general terms.

As mentioned above, I believe news agencies, political campaigns and media use social media to their favor. It is the agenda of the news/media to get ratings. What sells? Who benefits? Who commissioned? Because social research is not absolute and it is humans that create and parrticipate in it, it can be left wide open to interpretation. Due to this, it is easy to manipulate/present info from a bias perspective that benefits the presenter(s). Bottom line- research information presented, find out who commissioned the information and who the population sampled is/was before forming your own thoughts.

[Brooke Green]

After watching this video clip it, I really understood why it was so important to use your sociological imagination for critical thinking. When watching news on poll data, you have to keep a very open mind. Seeing poll data is very neat and interesting way to see how the rest of the world stands on a certain position in politics. It allows you to compare your own opinion to the rest of the worlds.

I think the video clip really demonstrated the survey method well. It really got behind the whole concept and method of the survey method and shows a different take and angle on it.

When it comes to the media and political campaigns, it's hard to decipher between what's real and what's not. Obviously social media is very bias towards one particular side, it doesn't matter what type of situation it is. It is however a very useful tool in social research. It allows researchers to gather information on what the world is thinking on certain situations.

[Brandon Brichetto]

I agree with Brooke when she states "When watching news on poll data, you have to keep a very open mind." In order for someone to keep an open mind; you have to use your sociological imagination to think critically. The polls are formatted in a way that steers the public to vote in the way that the news station supports. You also have to keep in mind that there is no proof that the data is not tinkered with to show a more unbalanced poll in favor of one side, rather than what the people actually voted for.

The Daily Show clip gives a good illustration of the representative sample The representative sample in the polls would be citizens of the United States that watch the Daily Show.

New agencies, political campaigns, the media, etc. all use social research. They use the concept known as applied research to try and apply the poll to help them succeed. For example, a political campaign can try to gain knowledge of what voters what to see and apply what they learn to try and use the knowledge of the public gained to make some sort of change.

[Hannah Wilson]

When you watch the news on poll data you have to keep an open mind. And in order to do that you have to use your sociological imagination to think critically. One concept from Chapter 2 that they use is survey. News agencies, political campaigns, and the media use social research. They use social research to try and apply the poll to help them succeed.

[Jane Stuckey]

I think people are going to any anything, you have to look at reality. What politicians say they are going to do and what actually happens.

[Jane Stuckey]

I also agree with Brandon, I think they are only goinh to let us know what they want us to know, we have to have an open mind, and go with the facts