
Hi there, argmin readers. We continue p(doom) week to celebrate the Anthropic IPO. Today’s post is also a live blog of Class 20 of my graduate seminar “Forecasting: A Critical Retrospective.” We’ll discuss scientific forecasts of doom. The syllabus and list of past posts are here.
We’ve come to the end of the semester, and it’s time to talk forecasts of doom. You all know I’ve lived in the Bay Area for over a decade, so I’ve been surrounded by the millenarian death cult warning of a coming machine god that will kill all of humanity. Of course, millenarian death cults are not a new phenomenon. People overrun by societal anxieties often band together and predict that current troubles are signs of the coming apocalypse. What makes this group uniquely 21st century is their embrace of scientific computation.
Computers give the millenarian a new authoritative tool: illusory objectivity through charts, numbers, and simulations. In our class on simulation, we touched on the seminal computational millenarians who warned of population explosions in the 1970s. Their computer models generated dire forecasts of overpopulation leading to environmental collapse. Paul Ehrlich’s The Population Bomb prophesied that the world would run out of food in the 1970s. With the more detailed computer simulation, World3, these ideas were backed by charts and numbers in The Limits to Growth, bringing “scientific rigor” to Ehrlich’s back-of-the-envelope calculations. Exponentials, you see, quickly blow up over time. And when they do, boy, are we in trouble.
Well, that should sound familiar if you listen to any of these crazy Bay Area doomsayers. We are building intelligent systems. Their intelligence increases exponentially. If this continues unabated, we all die.
The language isn’t too far from something I’m more sympathetic to: climate change activism. Climate activists have been promoting fear of doom over global warming exponentials for decades. They also provide compelling simulations of tipping points and global disasters that make the Limits to Growth people look tame. Their models are vastly complex and need supercomputers to enumerate the many predictive scenarios consistent with centuries of climate evidence.
The activists on both of these issues are sincerely convinced they are right and that anyone who doesn’t listen is an idiot. They ask us to come back to their predictions and see how many things they got right. Never mind that they got some things wrong. The exponential is still increasing. If you don’t heed their very scientific warnings, everyone dies.
These examples are extreme, but they’re a troubling part of how scientific language shows up in policy arguments. Sheila Jasanoff calls these mathematized risk assessments, cost-benefit analyses, and forecasts “technologies of hubris.” They promise control over uncertainty by quantifying it. Quantification makes it not only objective but real. Jasanoff argues this defers too much authority to expertise that shuts down debate with a collection of stories about what matters, what’s inevitable, and what must be done. People aren’t allowed to engage because the scientists make the decisions technical. Even though the policy decisions associated with the forecast affect many people’s lives, only a particular sort of expertise is allowed the right to make decisions. This is illiberal, and negates the importance of not only democratic value judgments, but also of any expertise that isn’t blessed as scientific.
Daniel Sarewitz goes further, arguing that scientific language is uniquely ill-suited for adjudicating hard decisions. Science is a system of knowing uniquely suited to increase uncertainty, not mitigate it. Two scientists motivated to disagree will never agree about scientific predictions. They will only agree to do more experiments and collect more data. This keeps them gainfully employed. This process of exponential positivism is what Michael Strevens calls the iron rule of scientific explanation. When two scientists disagree, they settle their dispute with a new experiment. This experiment is expected to bring new evidence into the corpus of human knowledge, but it is decidedly not expected to bring closure—that would mean no more grants and papers! After new data, scientists have new things to argue about. They’ll always have questions and more experiments.
In a political context, however, this allows people with different values to entrench themselves in a battle of constant uncertainty amplification. No side’s forecasts ever become dispositive. In the end, when science makes political decisions worse, the scientific debate is just set aside and empowered political actors make the decisions.
However, a troubling catch emerges when we move from science to engineering. While science may be, as Harry Collins and Trevor Pinch put it, a powerful golem that wanders unpredictably, engineering is all about control. And what every good engineer knows is that the more authority they have, the more they can bend reality to their will. The only question is how much authority they can access. Marshaling resources for a major product and forcing the world to look the way you want has long been the engineer’s superpower. Exponential resource allocation yields exponential progress, but you need the first to get the second.
And this is perhaps the most troubling thing about the AI doomsayers. We’re supposed to be impressed by how many things they forecast correctly, but we’re not supposed to talk about how their predictions are documented KPIs for companies being allocated trillions of dollars in resources. The doom scenarios in millenarian screeds like AI 2027 are extracted from conversations with engineers and speeches by tech executives and venture capitalists. They are design documents, not forecasts.
Gathered through online posting and in-person meetups, doomer PDFs are snapshots of the zeitgeist. They are what OpenAI and Anthropic want. It’s what XSI and Google and Microsoft and Amazon want. Don’t take my word for it. They have been saying so for decades! They want to build a machine god. They want to see this god rule the Earth, either by killing us all or making us pets. These so-called scientific forecasts distill technologists’ desires. Those desires are what should scare us.

