Hi there, argmin readers! Today’s post is a live blog of Class 13 of my graduate seminar “Forecasting: A Critical Retrospective,” in which we touched the third rail of forecasting discourse: pandemic modeling. The syllabus and list of past posts are here.
No forecast had a greater impact on our lives than “Report 9: Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand.” Written by epidemiologist Neil Ferguson and colleagues, the report presented prefactual scenarios of what would happen in the UK and the US under various pandemic mitigation strategies. Report 9 was released on March 16, 2020, and many nations and states soon heeded its advice, closing schools and businesses and mandating that citizens lock themselves in their houses.
The report estimated the death tolls—without error bars—for three potential policy scenarios. Doing nothing would lead to 510,000 deaths in the UK. Enacting a “mitigation” policy, where symptomatic people isolated at home and people at risk practiced “social distancing,” would lead to 250,000 deaths. Most optimistically, following the lead of China and enacting a policy of “suppression”—an aggressive, authoritarian shutdown of society—would “flatten the curve” and lead to only 20,000 deaths.
These numbers came from a complex computer simulation with over 900 parameters, almost all of which were unknown. Despite this tremendous uncertainty, the report concluded that “epidemic suppression is the only viable strategy at the current time.”
Even at the time, it was clear that Report 9 was more an advocacy document than a scientific report. Report 9 provided a scenario under which a better future, under a completely untried and untested policy, was possible. Unfortunately, that better future didn’t manifest itself, and faith in public health agencies and other decision-making institutions has never recovered. The team leaned on its institutionally approved scientific expertise to sway public opinion. This was a major risk, and, in retrospect, I wonder if the broader public health community thinks it was worth it.
I don’t want to litigate pandemic policies today, but we can’t talk about what forecasting does without talking about Report 9. It has all the hallmarks of a bad forecast. Its authors used a complex model to produce concrete numbers with no qualification of the tremendous uncertainty in their numbers. They didn’t share their baroque model with the public for over a month. Once it was released, others showed the model itself only shakily supported its own findings. Indeed, Edeling et al. showed that under a mild sensitivity analysis, the 20,000-deaths forecast was at the lower boundary of the 68% confidence interval, well below the mean forecast. It’s as if the team chose parameters to make their policy look as good as possible.
Moreover, the team refused to estimate any of the impacts of their policies beyond burden on the health system and deaths. Report 9 wrote, “while experience in China and now South Korea show that suppression is possible in the short term, it remains to be seen whether it is possible long-term, and whether the social and economic costs of the interventions adopted thus far can be reduced.” They had only a single sentence estimating these costs: “The social and economic effects of the measures which are needed to achieve this policy goal will be profound.” They claimed estimating these social impacts, whether on educational outcomes or economic output, was outside their expertise. Many interpreted this as the team asserting these outcomes were not ethically significant enough to even attempt to model.
As the pandemic progressed, it became apparent that the epidemiology community was bad at short-term forecasting. Human pandemics had basic rise-and-fall shapes predicted by elementary compartment models (known as SIR models), but when they would rise and peak was much harder to guess than Report 9 suggested. Indeed, recent work by White and Leon suggests that overconfidence in forecasts comes from trends that are easy to spot: seasonal trends, statistical smoothing, and exponential growth make pandemics easy to predict. Without these features, disease impact is impossible to predict. Scoring rules paper over the fact that we mostly have no idea how to forecast these things. We all learned in real time that epidemiological tools have very little predictive power. They are, at best, descriptive. And yet we were told in March 2020 to trust their long-term forecasting.
The actual death toll in the UK was later estimated to be around 230,000. Whatever the actual number is, the consensus is that it is much, much larger than 20,000. As is always the case when a forecast doesn’t pan out, the forecasters can argue there is no true Scotsman. They might say the policy was poorly implemented and the disease would have gone away if we had acted better. Such arguments carry no water in medicine, and they shouldn’t carry water here. A medical intervention is only effective if a patient population is willing to take it. A real population isn’t what you wish it would be. None of the proposed non-pharmaceutical interventions stopped the eventual wave of disease anywhere on earth, even in China.
Whether the suppression strategy worked is unknowable. We know that more than twenty thousand people died. We don’t know how many would have died under a different policy. We can’t know what the economic and educational outcomes would have been under a different policy either.
What exactly can we conclude then? A while back, I wrote about the Y2K bug and the preparedness paradox and latched onto John Quiggin’s argument that there were multiple true interpretations of the decision making. The COVID pandemic brought a similar pattern of simultaneous truths. It’s true that many people suffered and died, and some were saved by avoiding contact with others. It’s true that we overreacted and harmed children’s education and socialization. It’s true that “the other side” promoted plenty of craziness (Ivermectin? Hydroxychloroquine? Injected bleach?). I could make a long list of truths that would logically contradict each other.
But it’s also true that no one was able to predict the future. It’s obvious in retrospect, but questions of what should be normalized are questions of values, not data science. And in most political decisions, values have to trump science and its messy uncertainty. COVID forced us to make massively consequential decisions in the face of massive uncertainty. We used numerical reports and forecasts to dodge the moral consequences of our actions.
no error bars or some robustness metric seems a big oversight
I recently discussed with a group, that I trust, how six years has passed and nobody talks about the millions, no not 10,000 not 20,000, millions of life, lost. Those people’s lives, especially healthcare workers and Frontline workersx who were killed by the pandemic, is neither mourned nor commemorated for the sacrifices made: not here and not in India where more than 5 million people died. And those 5 million deaths happen in the second wayvewhen people were told not to mask by the modeling of the leadership. The data is very clear and to bring our own politics to this, despite calling the third rail is not good science. Indeed, every forecast and every policy has side effects and that’s worth bringing up. In the grid, if you make a forecast for an extreme situation that doesn’t pan out that’s a lot of money wasted. If it does pan out and you underestimated that also leads to billions of dollars of loss. There is no perfect solution.