The Art and Science of Survey Writing
While a census by definition is distinct from a survey, which seeks out a representative sample of a population, both types of research tools rely on good question construction to get the most accurate results. Not only should the questions be written clearly, but ideally they should be written in such a way that brings you closer to learning more about the population.
Writing survey questions is something that students in the social sciences often struggle with. The most common mistake I observe is that new students simply pose their hypothesis in the form of a question, and believe that this will help them test their hypothesis. But unless your hypothesis is about public opinion, this type of question won’t get you very far.
Let’s say you want to learn more about the relationship between educational attainment and income, two straightforward concepts. You wouldn’t ask respondents their opinion about the relationship (“Do you think education is related to income? Circle yes or no”). Although this would be a very easy question to write, it wouldn’t tell you anything about the relationship between education and income. It would tell you whether your informants think there is a relationship, an entirely different (and far less compelling) study.
Instead, you would need to operationalize the concepts of education and income. Put simply, this means that you will need to decide how you will specifically measure these ideas. For this example, we can measure education in years of schooling, and income can be measured in dollars. This would give us two ratio variables, or continuous numbers which will give us access to many different statistical measures for analysis.
But putting operationalized variables in the form of a question can be tricky. Do we ask respondents how many years of education they have? Some people might not be sure (I have to stop and think about this one myself). And what about someone who takes 6 years to earn a degree that others might finish in 3—do they have twice as much education? Or several years of coursework that never led to a degree?
Instead, it might make more sense to ask respondents what is the highest level of education they have completed and provide categories for them to choose from, such as less than high school diploma, high school diploma, some college, college degree, post graduate degree. As ordinal variables, we would have less statistical flexibility for our analysis, but it would likely be easier for respondents to answer and give us the information we are looking for.
We would need to pay particular care to make sure all respondents will have a category that fits their education, and that only one category applies to them. We would also want to think about this conceptually; do we want to include a separate category for people with two-year degrees to separate them out from those with bachelor’s degrees? What about vocational training? GEDs? Do we want to identify people who have attended graduate school but did not finish that degree?
You might think, sure, the more we know the better, but the more categories respondents will have to select from, the longer our survey becomes, and perhaps the more confusing it will be as well. Creators of surveys must have a rationale for both the questions and response choices (for an example, check out the U.S. Census Bureau’s explanation for the questions on the American Community Survey).
What about asking respondents for a simple number, like their annual income? Again, a ratio-level variable would give us lots statistical measures to choose from, but this response option might not be so easy for respondents either. Some people might not know the exact dollar amount of their income. If they do, they might confuse their gross income with their net income, or if they are salaried, they might have that number in mind.
Even if someone does remember their income, they might feel uncomfortable about sharing this information. For these reasons, income categories make more sense in most instances. While creating these categories, we need to keep in mind how they might be easiest for our respondents to choose the appropriate option as well as which is most logical for our sample. If we have a lot of people at the lower end of the income spectrum, have more options at the lower end is a good idea.
As you can see from this seemingly simple example, writing survey questions takes a great deal of thought about what we want to learn, about our respondents, and about what sort of analysis we hope to do with the data.