Forecasting WTF? - A Syllabus

A plan for this semester's critical retrospective on forecasting and its culture.

I more or less said what I was going to do in my tongue-in-cheek, cryptic post on Tuesday, but let me dive into the full details of how I’m planning on structuring this grad seminar. I’m going to log the course content here, which now lists a rough schedule for the class sessions. Taking inspiration from Matt Jones and Chris Wiggins, I’m going to do a weekly split between culture and engineering. Focusing on a particular application domain each week, we’ll spend one session discussing the purpose of forecasts in that domain and the other on the methods.

I arranged things into a thematic arc, starting with the weather—forecasting’s biggest success story—then moving to shakier ground in seismology and epidemiology, and ending with, well, millenarianism. As we move across timescales, our ability to predict nature dissipates: we can make accurate ten-day forecasts, but predicting large-scale climate disruptions is far more qualitative. We can predict immediate earthquake impacts at a distance once an earthquake has happened, but we can’t nail down precisely when a big one will happen. What we do with precise, short-term forecasts is completely different from what we do with imprecise long-term forecasts. Short-term forecasts dictate actions; long-term forecasts of discrete shocks inform risk management, preparedness, and rapid-response policies. I’m hoping that by the end of the semester I can better articulate what long-term forecasts of the end of the world do.

We’ll look at forecasts in governance and how they influence and shape policy. I’m particularly interested in discussing the rise of cost-benefit analysis in US governance. Cost-benefit analysis puts a specific number on something completely unknowable, but is now mandatory for any bill or law to pass. I want to trace how we became so reliant on a particular set of methods for guessing costs and benefits. I’m also interested in how economists became convinced you could forecast “the economy.” This required inventing something called “the economy” that could be forecast in the first place. How our system of government got so tied to a particular style of economic prediction will occupy several weeks of the class.

We’ll also get into why we want to predict “the public.” I want to examine how opinion polls went from a question of legibility to one of prediction. How did we get obsessed with using surveys to predict outcomes like elections? In parallel, we’ll look at the history of attempts to simulate the public. Simulation is nice because you don’t have to talk to people, right? We’ll look at the many misses over the history of human simulation in policy scenarios, and dig into the current obsessions with using LLMs to predict what people might do.

Finally, we’ll get into the weird culture of competitive forecasting. We’ll engage with ideas from superforecasting and prediction markets and ask why people think these are useful information-processing systems. We’ll talk about punditry and how it’s not always interested in minimizing a Brier Score. We have to talk about the relationship between forecasts and gambling. And we’ll try to piece together why putting odds on outcomes makes people feel better about the future.

For the methods, I did my topic matching so you could extract a logically ordered half-semester course on forecasting from those lectures alone. This is not a class on how to be a rational forecaster. I want to problematize those methods more than tell you how to implement them. You can ask your friend Claude if you need an honest, load-bearing implementation.

Our methods survey starts with a refresher on my idiosyncratic views of machine learning as optimization-driven algorithmic pattern recognition. This will lead to a lot of discussion of the optimization problems themselves and why people like them. We’ll cover scoring rules, calibration, maximum likelihood, and utility maximization. We will discuss the role of models and look at probabilistic recurrence models, dynamical system models, differential equations, and other simulation-based tools. We’ll spend time on uncertainty quantification and how people come up with error bars (part of being a good forecaster is plausible deniability). Then we’ll look at offline and online optimization methods that let you fill in predictions based on your cost functions and modeling assumptions. I’m interested in highlighting the metrical determinism. The cost functions and models more or less tie your hands algorithmically, and most of the cleverness goes into how you evaluate.

Hopefully this arc will feel coherent as we go. I’m not into predictions, so don’t get mad if the story changes as I go. I’ll blog through it, and then we can reflect on where we land at the end of the semester.

Enrolled students (and those dedicated to following along at home) have an important first assignment: pick something to forecast. I don’t care what it is. Throughout the course, the goal is to learn the practical techniques by making predictions. Every week I’ll ask you to try to apply the tools to your problem. Or at least find how other people have applied those same tools to your problem. At the end of the semester, we’ll present our full findings and see how accurate we can be.