Guess and check
Remember from a week ago, statistical intervals are all the outputs of Monte Carlo Algorithms. The goal is to find a set that contains some attribute. We gain access to a sample from a distribution, do some computations, and return a set. We guarantee that with probability p (almost always 95%), the set contains the attribute of interest.
Confidence intervals are the most famous of the statistical intervals. Confidence intervals are measurements, and the interval provides a sense of the precision of this measurement.
Today, let’s move from measurement to prediction. Prediction intervals are also Monte Carlo Algiorithms, but they have a different flavor of guarantee. The attribute we’d like to predict now is a random number itself. A prediction interval promises to contain some future measurement with some specified probability.
Let me make the problem concrete. I’m building a camera to track the motion of a spacecraft. I know that my physics model of the ship is pretty good, but there are some inaccuracies in my models of how the spacecraft flies, its environment, and my radar device’s range measurement. After years of testing, I find these errors look random and effectively independent from measurement to measurement. So now, when a craft is flying, my sample is all of the measurements I’ve seen so far, and the prediction interval should contain where I measure the ship one time step from now.
Here’s a second example that might be more of a stretch. Let’s say I want to be a day trader. I observe the price of a stock for a month. I now make some assumptions about the universe. I assume that the change in the stock on any day is a random number. Long-time readers know that I am uncomfortable calling any natural process, including the daily fluctuations in a stock price random. But let’s go with it. With this probabilistic leap of faith, my sample is now the list of price changes of the stock. I’d like to predict the change in the stock tomorrow.
One thing that immediately differentiates the prediction interval from the confidence interval is verification. I lamented that with confidence intervals, we were always stuck with “a chance” that the measurement was wrong. But with prediction intervals, we can always check if the algorithm is correct because we will observe the actual value of the prediction at some point.
This ability to check and correct can be very powerful. In the aerospace example, I might build a Kalman Filter to iteratively inform my predictions for subsequent measurements. We used such filters to send people to the moon.
In my stock trading example, I’ll see the actual value of the asset tomorrow and then know if I called it correctly. What I do with this information is less clear. Hopefully, I didn’t run out of money and can update my predictions for tomorrow’s trading. But it’s not particularly useful for my brokerage account if all I can do is verify correctness.
I suppose there are other benefits of prediction intervals. If you made lots of prediction intervals, and your predictions were mostly right, you could brag that you are good at forecasting. If you’re doing sports analytics or investment advising or data journalism or AI doomsaying, this will help grow your brand. If they were mostly wrong, you could use the data to calibrate more prediction intervals. Either way, if you keep blogging about it long enough, maybe this will turn into a paying gig.
Surely, prediction intervals have applications beyond punditry. I want to know how to use prediction intervals in decision making. In this case, I want to make a prediction and then act based on that prediction to achieve some goal. If I predict a high rainfall, I’ll pack an umbrella. If I predict a high variability in some stock prices, I’ll adjust my portfolio to avoid losing all of my money.
But what if I want to intervene to change an outcome? If my ship is getting off course, I want to steer it back on target. If a doctor predicts some bad outcome in a patient, they want to intervene to prevent that outcome. How can we build prediction intervals that we can act upon? Let’s keep this question in mind as I’ll turn to some trendy ways people make prediction intervals. Let’s see what it might mean to use these intervals for making decisions.