Behind the Research: Understanding Panel Studies

By Karen Sternheimer

The Pew Research Center recently released a study about online dating. They found that about thirty percent of U.S. adults have used an online dating site or app, and that just over one in ten have used one in the past year. About 12 percent of respondents reported being in a serious relationship or marrying someone that they met online.

When most of us read about or hear about studies like these we don’t think much about how the findings are generated. Who are included in the study, and how do researchers find them?

Finding survey respondents used to be much easier than it is today, ironically because of the widespread use of the technology which once made phone surveys much easier to conduct. You know those ubiquitous spam phone calls you get from numbers you don’t recognize? They use technology similar to random digit dialing, a technique which decades ago made it easy for researchers to conduct household surveys and to include people with unlisted phone numbers. Twenty-five years ago, most households had a landline, not all had caller id, and very few calls came from unknown people. That meant that people almost always answered their phones if they were home.

Today telephone sampling is more difficult, thanks in large part to the billions of spam calls Americans get each year. Most of us won’t answer a call from a number we don’t recognize, and fewer and fewer people have landlines. Pew reports that the response rate for phone surveys is was about 6 percent in 2018, compared with 36 percent in 1997. So how do researchers get random samples of Americans to participate in their research?

One method is a panel study, the method that Pew Research Center used as a large part of their online dating study. In a panel study, researchers recruit a large group of people who comprise a random sample of the population, meaning that researchers can draw conclusions about the population as a whole from this group. This also means that researchers have a constant pool of participants to draw from for their research.

These people agree to participate in a number of surveys over a period of time online. Pew Research Center has created their American Trends Panel (ATP), which as of 2018 had over 13,000 participants who receive small cash incentives to participate.

Pew reports that because they collect demographic information from initial surveys, when a participant completes future surveys they don’t have to ask basic questions that researchers normally would (about things like, age, ethnicity, income, gender, and so forth). Because they are surveying many of the same people over time, researchers can see how the same people’s views change over time and ask them questions about any changes. By contrast, a cross-sectional study might look at responses different time periods but using different participants.

Unlike most online surveys, the ATP randomly selects participants offline first. As Nick Bertoni, the manager of the panel explained:

For the 2018 recruitment, we shifted gears and recruited panelists using mailed invitations for the first time. We used what’s known as address-based sampling, which just means drawing a random, representative sample of residential addresses included in a U.S. Postal Service database that covers about 97% of the U.S. population.

Certainly some of the population gets left out of this recruitment method: people who don’t have mailing addresses, for one, won’t be invited. People who don’t have internet access or who are uncomfortable completing an online survey might be underrepresented, but Pew tried to remedy this in 2016 by providing tablets and technical support. But this remains a problem for the ATP:

The underrepresentation of non-internet households remains a challenge for the ATP. Though they are a relatively small share of the adult population, these households are demographically quite different from those who do have home internet access. Nearly half of those in the panel without internet access are ages 65 and older, about six-in-ten have only a high school education or less and nearly half are nonwhite.

One way that survey researchers address undercounts of populations is to use weighting techniques when calculating the results, or giving more weight to responses from underrepresented groups to try and better approximate their number in the population.

All research methods have drawbacks, and panel studies are no different. But they allow us to learn a great deal of information over a long period of time. Perhaps the most well known is the Panel Study of Income Dynamics (PSID), which has been gathering data on families, income, education, and many other issues since 1968. A great deal of social science knowledge comes courtesy of this project conducted at the University of Michigan.

What other benefits can panel studies provide? Other limitations? Would you ever consider being part of a panel study?