Hi there, argmin readers! Today’s post is a live blog of Class 14 of my graduate seminar “Forecasting: A Critical Retrospective,” turning to the less controversial subject of public opinion polling. The syllabus and list of past posts are here.
Our next two units on forecasting focus on forecasting societal behavior, and to do that we first need to talk about how we measure properties of society. One of those main measurements is surveys. Today, I want to describe some techniques survey designers use to avoid ever having to interact with actual people.
Let’s start with the basics.1 Let’s say you want to measure some property of a population, like what percentage of dentists recommend chewing sugarless gum. You can’t ask all of the dentists, so you gather a random sample and then extrapolate the fraction in the sample to represent the fraction in the broader population of dentists.
The main idea of why this is reasonable is simple. If you gathered a true random subset of a population of people and asked them a yes-or-no question, then the average of the answers in the sample would most of the time be very close to the average in the population. You can even quantify how close. The expected gap between the sample average and the population average is nearly independent of the population size and mostly a function of the sample size. This gap is inversely proportional to the square root of the sample size, so if you want to estimate the fraction of a population that answers yes to some question, you can get a “margin of error” of a few percentage points with a few hundred people.
This is a colloquial description of the weak law of large numbers. It often feels miraculous. You can get a good estimate of an average with 600 samples no matter how large the population is. But that margin of error is only true if you have a truly random sample. Unfortunately, when you’re harassing people to answer your annoying surveys, you never get a truly random sample! Not everyone will talk to a pollster, and not everyone is reachable in the first place.
So the data that surveyors gather is a mix of randomness, convenience, and population predilections. You can’t just average the responses and hope for the best. Without a truly random sample, that average will be biased. Fortunately or unfortunately, statisticians have spent decades finding clever ways to use math to patch up social scientific deficiencies. They don’t let people’s actual behavior get in the way of a good time!
One idea is to use post-stratification to recalibrate your survey. The idea behind post-stratification is that once you have a fine enough demographic coarse-graining of people, there are no systematic biases in reporting. There are many justifications for this assumption, but more or less it assumes that within each demographic unit, respondents vote like nonrespondents.2 If this is true, you can weight the respondents in your sample by their demographic proportion in the population and combine them into a new estimate that is unbiased, assuming your model relating demography and human behavior is correct. My friends Avi, Eli, and Erin wrote a clear introduction to how this is done in practice and what assumptions have to hold for this reweighting to be unbiased.
Now those demographic weights themselves are also hard to come by. First, you need an accurate census to guess the fraction of the population that belongs to each demographic cell. This creates a problem of turtles all the way down, because it requires doing censuses that have similar response and bias issues. Second, you need to be sure that you actually sample someone from every demographic stratum in your survey. That is almost never true.
So what you do instead is fit the weights in your sample using a statistical prediction model. There are many dark-art techniques to do this, each with its own ornate justifications. Fitting the weights gives modelers a lot of degrees of freedom. Pollsters are now using their own internal models to weight their numbers. Different models can cause swings in estimates far outside the “margin of error” reported with the poll.
This is only one way statistical prediction blurs into statistical sampling. Indeed, you can lower the variance of your survey with a good prediction. In classical survey calibration, you combine your model’s guess of the outcome with the estimated deviation from that model in your survey sample. In this paradigm, if you really believe in your prediction model, you can throw the survey correction out altogether.
If you want to boost your confidence, you can even combine these two prediction techniques. Statisticians hilariously call this “doubly robust.” If the model that predicts poll answers from demographics is correct, the term that depends on the sample data will be small and not mess up your prediction. If the model that predicts who will be in the sample based on demographics is correct, this will cancel out the effect of the answer-predicting function. Unfortunately, some guy told me that all models were wrong. In that case, there are no guarantees.
Regardless, my point here is to show how statistical modeling can replace statistical sampling. As more and more people refuse to answer polls, these statistical models end up replacing talking to people. In practice, surveys become more about what experts believe about society than what society actually is. Many of these experts seem to believe that they can forecast human behavior without ever observing humans.
I debated putting all the equations here, but I try to keep this blog as equation-free as possible. I will do it if enough of you bug me in the comments.
Statisticians, don’t @ me for oversimplifying your weird assumptions about how people behave.