April 19, 2018

The Art and Science of Survey Writing

Headshot 3.13 cropcompressBy Karen Sternheimer

Recently, the news of a question about citizenship in the 2020 Census has made news. Critics are concerned that such a question might lead to a lower response rate, most notably by immigrants.

While a census by definition is distinct from a survey, which seeks out a representative sample of a population, both types of research tools rely on good question construction to get the most accurate results. Not only should the questions be written clearly, but ideally they should be written in such a way that brings you closer to learning more about the population.

Writing a survey requires a great deal of thought about the wording of individual questions, the ordering of the questions, and which questions are necessary to learn what the researchers want to find out about the sample or population. For most surveys, length can be a problem as respondents will likely only be willing to devote a short amount of time—if any—to answering the questions.

Writing survey questions is something that students in the social sciences often struggle with. The most common mistake I observe is that new students simply pose their hypothesis in the form of a question, and believe that this will help them test their hypothesis. But unless your hypothesis is about public opinion, this type of question won’t get you very far.

Let’s say you want to learn more about the relationship between educational attainment and income, two straightforward concepts. You wouldn’t ask respondents their opinion about the relationship (“Do you think education is related to income? Circle yes or no”). Although this would be a very easy question to write, it wouldn’t tell you anything about the relationship between education and income. It would tell you whether your informants think there is a relationship, an entirely different (and far less compelling) study.

Instead, you would need to operationalize the concepts of education and income. Put simply, this means that you will need to decide how you will specifically measure these ideas. For this example, we can measure education in years of schooling, and income can be measured in dollars. This would give us two ratio variables, or continuous numbers which will give us access to many different statistical measures for analysis.

But putting operationalized variables in the form of a question can be tricky. Do we ask respondents how many years of education they have? Some people might not be sure (I have to stop and think about this one myself). And what about someone who takes 6 years to earn a degree that others might finish in 3—do they have twice as much education? Or several years of coursework that never led to a degree?

Instead, it might make more sense to ask respondents what is the highest level of education they have completed and provide categories for them to choose from, such as less than high school diploma, high school diploma, some college, college degree, post graduate degree. As ordinal variables, we would have less statistical flexibility for our analysis, but it would likely be easier for respondents to answer and give us the information we are looking for.

We would need to pay particular care to make sure all respondents will have a category that fits their education, and that only one category applies to them. We would also want to think about this conceptually; do we want to include a separate category for people with two-year degrees to separate them out from those with bachelor’s degrees? What about vocational training? GEDs? Do we want to identify people who have attended graduate school but did not finish that degree?

You might think, sure, the more we know the better, but the more categories respondents will have to select from, the longer our survey becomes, and perhaps the more confusing it will be as well. Creators of surveys must have a rationale for both the questions and response choices (for an example, check out the U.S. Census Bureau’s explanation for the questions on the American Community Survey).

What about asking respondents for a simple number, like their annual income? Again, a ratio-level variable would give us lots statistical measures to choose from, but this response option might not be so easy for respondents either. Some people might not know the exact dollar amount of their income. If they do, they might confuse their gross income with their net income, or if they are salaried, they might have that number in mind.

Even if someone does remember their income, they might feel uncomfortable about sharing this information. For these reasons, income categories make more sense in most instances. While creating these categories, we need to keep in mind how they might be easiest for our respondents to choose the appropriate option as well as which is most logical for our sample. If we have a lot of people at the lower end of the income spectrum, have more options at the lower end is a good idea.

As you can see from this seemingly simple example, writing survey questions takes a great deal of thought about what we want to learn, about our respondents, and about what sort of analysis we hope to do with the data.

Comments

Good post, thanks for the information

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Become a Fan

The Society Pages Community Blogs

Interested in Submitting a Guest Post?

If you're a sociology instructor or student and would like us to consider your guest post for everydaysociologyblog.com please .

Norton Sociology Books

You May Ask Yourself

Learn More

Essentials of Sociology

Learn More

The Family

Learn More

The Real World

Learn More

Introduction to Sociology

Learn More

The Everyday Sociology Reader

Learn More

« Community Building and Women’s Activism | Main | Intersectionality for Beginners »