Every Day You See One More Card

The mathematics of transmuting frequencies of the past into odds of the future.

Hi there, argmin readers! As the fall semester picks up, posting volume will, too. So I’m going to commit to writing short descriptive headers to help you sort through the different threads. Today’s post is a live blog of Lecture 3 of my graduate seminar “Forecasting: A Critical Retrospective.” A table of contents is here.

---

Since the Great Depression, US law has required financial management companies that offer products like mutual funds to add a disclaimer to all of their advertising:

“Past performance is not indicative of future results”

The thing is, not a single person believes this. The fund managers don’t believe it, and neither do the customers. Why would you buy a mutual fund if you didn’t think its past performance told you something about how much you’ll have upon retirement? We tend to believe that some investments are riskier than others and some managers are more reputable than others. These beliefs are based upon past observations, and we use them to inform our investment decisions.

So what do we need to do to transform past observations into forecasts? We believe that the past can’t perfectly predict the future. We also believe that a forecaster is only as good as their track record. In today’s class, we’ll link these two together, showing how the evaluation metric for forecast track records leads us to particular forecasting algorithms.

Let’s start with the two main examples from Edmond Halley. We believe the past strongly predicts the future when talking about the motions of celestial bodies. We believe it far less when pricing individual insurance policies.

For his comet, Halley paired three observations together using insights about orbital shapes from Isaac Newton. Given the roughly 76-year gaps between these observations, he predicted we’d see the same object again 76 years later. In 1835, by the time we had seen Halley’s comet twice more, astronomers were uniformly convinced Halley was right and were certain we’d see the comet again in 1910 and 1986 (they were proven correct).

For his life table, Halley grouped people by age and used these cohorts to make demographic forecasts. The proportion of the population aged 25 was 1.668%, and the proportion aged 26 was 1.647%. Therefore, he concluded the odds a 25-year-old would live to see 26 were approximately 80 to 1 in favor.

The move in the demography example to consider odds and chance is interesting, and was something in the air at the time. Proto-demographer John Graunt had made similar calculations of hazard and risk in his tabulations thirty years earlier. Probability applied to casino games had only begun to be formalized forty years earlier. The transmutation of frequencies into risk was intuitive once you started assembling databases. Now we just take it for granted, having created a formal structure that hides the intuition.

In today’s lecture, we’ll work out some of the formalism of this map from rates to risks, deriving the mathematical formulas that encode our assumptions. If you assert that a forecaster will be evaluated on their track record, and if you believe that events are effectively the same, then you bind yourself to making future predictions a deterministic function of the observed rates. The assumptions here are usually implicit. We’re assuming a strong level of interchangeability between past and future events with a particular signature. And we tend to use metrics that beg the question: the common scoring rules always return probabilistic forecasts.

The evaluation ties your hands to making a particular form of forecast. Given a set of knowledge and a statistical score, you are forced to make a constant prediction for all future events. If you allow your predictions to be real-valued, they are suboptimal if they don’t obey the rules of probability. The score itself leads us into a probabilistic mindset. I’ve been calling this metrical determinism, and I find myself inserting some variant of this lecture in every class I teach.

Both the comet example and the life table example can be thought of as scoring track records on average. When you have highly predictable events, a perfect score is possible, but it takes a few hits to convince a skeptic that you really have nailed it down. When events are less predictable, you just want to make sure you’re not losing money on your annuity sales, and maximizing future profits again leads you into a particular form of forecasting.

What’s important here is we don’t have to assume some sort of generative model of randomness to buy into probabilistic prediction. Halley did not have to assume that god was playing dice with who lived and died. Instead, probabilities and odds were simply convenient tools for the actuary to price their products. Probability was the logical consequence of assuming past performance was indicative of future results.