Sampling Bias and Twitter
A recent study using data from Twitter reports on human mood swings throughout the day. The sociologists gathered and analyzed English language tweets from 2.4 million people in 84 countries for over a year. They used software that analyzed the meaning of words in the tweets and assessed their connections to moods and emotions among other things.
When looking across days of the week and across continents and countries, there were not many extreme differences. When parsing the data by type of people – those likely to be “morning birds,” those most active in the afternoon or evening, and “night owls” – most have similar patterns except for those night owls. People who are more active during the night only have one peak of positive feelings in the morning, later than everyone else, and they have two peaks of negative feelings, one in the morning and another late at night when they are most active.
The researchers considered the context of the words that the analysis software identified. For example, words such as “good” could mean the expression of positive feelings yet the word could also be used sarcastically (“good one”), neutrally (“good night”), or negatively (“good grief”). They did look into these possibilities and reported that such occurrences were rare enough so as not to be a problem.
Before we get too excited about these findings, we have to consider the possibility of something called sampling bias.
While Twitter users may be a lot like the rest of us in many respects, we really don’t know that without more evidence. Twitter users most likely fit a particular social profile compared to non-Twitter users. One would guess that they would be wealthier and younger at the very least, and most probably live in more urban and suburban than rural areas. Thinking about how race and ethnicity relate to social class here in the United States, there are probably some differences there too (however this is a worldwide sample). But economic inequality probably impacts who uses Twitter globally.
Most of the news reports I came across mentioned the methods of the study and pointed out that one should have some caution in applying these findings to everyone. However, the enthusiasm about having such a large dataset so available to us for research purposes swept away most of these concerns.
A story in Discover Magazine does mention that studies like this typically use college students as their samples, thus this new dataset improves the representative nature of the research. This is a much broader swath of society than college students, but they are still not truly representative so generalizing outside of Twitter users to larger groups should be done carefully (or not at all).
The Twitter research is certainly not the only major study with potential sampling bias. Masters and Johnson had a pretty biased sample in their lab studies when
they started doing their research measuring the process of sexual arousal. Subsequent studies supported their basic research findings on the sexual arousal cycle as representative even though their research first used primarily college-educated urban white people. In essence, the phenomenon they were measuring was less affected by social differences.
While this Twitter study may now be my favorite tale of sampling bias, another is Literary Digest magazine’s 1936 prediction that Republican Alf Landon would beat Democrat Franklin Roosevelt in the Presidential election using a sample of over two million. Apparently they used a telephone survey or a mailed survey, yet either one could have presented some bias issues in 1936. In the middle of the Great Depression, many lacked phones or a settled address to receive mail. Their results were way off, even with such a big sample, because sampling bias brought them more Republicans and fewer Democrats who agreed to answer their questions.
Since so much social and psychological research uses college students as their sample of choice (and convenience), we should also be cautious in generalizing those findings. However, we do need to start somewhere and using the samples that are available to assess what we can is an important step towards greater understanding. The hard part is interpreting our findings with both caution and enthusiasm, whatever the patterns are that we illuminate.