Sampling Bias and Twitter

By Sally Raskoff

A recent study using data from Twitter reports on human mood swings throughout the day. The sociologists gathered and analyzed English language tweets from 2.4 million people in 84 countries for over a year. They used software that analyzed the meaning of words in the tweets and assessed their connections to moods and emotions among other things.

Throughout the day, there are more positive feelings expressed mid-morning and in the late evening. More negative feelings are expressed late at night.

When looking across days of the week and across continents and countries, there were not many extreme differences. When parsing the data by type of people – those likely to be “morning birds,” those most active in the afternoon or evening, and “night owls” – most have similar patterns except for those night owls. People who are more active during the night only have one peak of positive feelings in the morning, later than everyone else, and they have two peaks of negative feelings, one in the morning and another late at night when they are most active.

The researchers considered the context of the words that the analysis software identified. For example, words such as “good” could mean the expression of positive feelings yet the word could also be used sarcastically (“good one”), neutrally (“good night”), or negatively (“good grief”). They did look into these possibilities and reported that such occurrences were rare enough so as not to be a problem.

Before we get too excited about these findings, we have to consider the possibility of something called sampling bias.

While Twitter users may be a lot like the rest of us in many respects, we really don’t know that without more evidence. Twitter users most likely fit a particular social profile compared to non-Twitter users. One would guess that they would be wealthier and younger at the very least, and most probably live in more urban and suburban than rural areas. Thinking about how race and ethnicity relate to social class here in the United States, there are probably some differences there too (however this is a worldwide sample). But economic inequality probably impacts who uses Twitter globally.

Most of the news reports I came across mentioned the methods of the study and pointed out that one should have some caution in applying these findings to everyone. However, the enthusiasm about having such a large dataset so available to us for research purposes swept away most of these concerns.

A story in Discover Magazine does mention that studies like this typically use college students as their samples, thus this new dataset improves the representative nature of the research. This is a much broader swath of society than college students, but they are still not truly representative so generalizing outside of Twitter users to larger groups should be done carefully (or not at all).

The Twitter research is certainly not the only major study with potential sampling bias. Masters and Johnson had a pretty biased sample in their lab studies when

they started doing their research measuring the process of sexual arousal. Subsequent studies supported their basic research findings on the sexual arousal cycle as representative even though their research first used primarily college-educated urban white people. In essence, the phenomenon they were measuring was less affected by social differences.

While this Twitter study may now be my favorite tale of sampling bias, another is Literary Digest magazine’s 1936 prediction that Republican Alf Landon would beat Democrat Franklin Roosevelt in the Presidential election using a sample of over two million. Apparently they used a telephone survey or a mailed survey, yet either one could have presented some bias issues in 1936. In the middle of the Great Depression, many lacked phones or a settled address to receive mail. Their results were way off, even with such a big sample, because sampling bias brought them more Republicans and fewer Democrats who agreed to answer their questions.

Since so much social and psychological research uses college students as their sample of choice (and convenience), we should also be cautious in generalizing those findings. However, we do need to start somewhere and using the samples that are available to assess what we can is an important step towards greater understanding. The hard part is interpreting our findings with both caution and enthusiasm, whatever the patterns are that we illuminate.

Comments

[PFCdgayo]

Very insightful post, I'm glad to read that other people are worried about the many biases in Twitter (specially self-selection bias).

I've also liked the mention of the (in)famous Literary Digest poll of 1936 :)

BTW, if you are interested in a post-mortem dissecting a failure predicting from Twitter analyzing the possible sources of bias regarding political discourse and elections you should read a paper of mine:

"Don't Turn Social Media Into Another 'Literary Digest' Poll"

http://cacm.acm.org/magazines/2011/10/131406-dont-turn-social-media-into-another-literary-digest-poll/fulltext

And the author's version:

http://www.di.uniovi.es/~dani/downloads/Social-Media-Literary-Digest-authors.pdf

Best, Dani

[erin]

I found this study very interesting. I my self have a twitter and have noticed that people love to get on twitter to complain about life or to boast. I think that a lot of people on twitter lie or embellish their lives to make them more interesting. i personally think this study should be taken with a grain of salt. Also twitter users are not a random or accurate sample of humans as the study claims because not everyone has a twitter just a select group of people.

[MikeB]

Thanks for the thought provoking post! I"m inclined to describe this type of methodological flaw as a generalization bias, rather than a sampling bias. If the author had simply explained the results in terms of Twitter users, rather than generalizing the results to a larger population, it would have been must more convincing.

[Mackenzie]

Very interesting post! I agree that there are other factors that have to be looked at before saying one knows when one is sad or happy based on when tweets are tweeted. All other factors must be taken into account, such as what you said, people using twitter tend to be both younger and wealthier. Thanks for posting!