What we learned about online nonprobability polls

For decades the gold standard for public opinion surveys has been the probability poll, where a sample of randomly selected adults is polled and results are used to measure public opinion across an entire population. But the cost of these traditional polls is growing each year, leading many pollsters to turn to online nonprobability surveys, which do not rely on random sampling and instead recruit through ads, pop-up solicitations and other approaches. The advantages of these online surveys are obvious – they are fast and relatively inexpensive, and the technology for them is pervasive. But are they accurate?

Pew Research Center undertook a study to answer this question. We found that not all online surveys perform equally well. A key reason seems to be the different methods employed by vendors. One of the nine nonprobability samples clearly performed better than the others, and it seems to be related to the fact that they use a more sophisticated set of statistical adjustments, both in selecting their sample and weighting their results. Our conclusions about why that sample performed the best are preliminary, though, because we have just one survey from that vendor and the relevant design features were not experimentally tested within that survey. One of the other major findings was that, in general, subgroup estimates from these samples for blacks and Hispanics were very inaccurate. Almost all the samples were off by double digits – more than 10 percentage points – on average in their estimates for blacks and Hispanics. That is quite concerning for researchers like us who study the experiences of different groups within the US population.