Don't be resulting

On the vulgar arrogance of the vulgar positivism of decision theory

I will likely start to lose readers if I write a football analytics post every week, but bear with me to the end—this post isn’t really about football analytics.

Yesterday in the NFL’s too-early Sunday game from Dublin, the Pittsburgh Steelers were beating the Minnesota Vikings 24 to 21. With 1:08 to go, the Steelers had the ball on Minnesota’s 40-yard line. It was 4th and 1. This brought Coach Mike Tomlin to a key decision point. They could either attempt to move the ball a single yard and get a first down or punt the ball back to the Vikings and try to stop them from scoring. If the Steelers got the first down, the game would have been over. If they didn’t convert, Minnesota would get the ball and would need about 20 yards of offense to get into field goal range. If they punted it, the ball would move to Minnesota’s 20-yard line, and Pittsburgh’s defense would have to prevent Minnesota from moving the ball 40 yards.

The Steelers opted to punt, and announcer Greg Olsen was apoplectic. The correct answer in his mind was unambiguous: they should have gone for it. He exclaimed, “They can still win the game. But the outcome doesn’t mean the decisions are correct.”

The Steelers did indeed win the game. They punted, and Minnesota only managed to move the ball about 19 yards before backing itself up due to bad quarterback play and penalties. But how is it possible to know that the decision was wrong even though the outcome was correct? What on earth does it mean for the decision to be wrong?

A prevalent mindset with outsized influence in contemporary football says you can decision-theoretically analyze every move. Olsen has analogized this to blackjack, saying that even if you occasionally make 21 on a daring hit, you can precisely calculate your probability of winning and know that if you did that enough times, you’d go broke.

But football is not blackjack. You can make an argument that baseball is statistical, but football is a weirdly overcomplex game with bizarre, overly legalized rules (what is a catch? yikes). The idea that you can statistically analyze every nook and cranny and come up with an optimal strategy is just wrong. Indeed, Aaron Schatz, a pioneer in football analytics and inventor of the popular statistic DVOA (Defense-adjusted Value Over Average), chimed in on Bluesky, befuddled. He wrote, “Here’s the thing about Tomlin. He makes a ton of game management mistakes according to analytics yet he consistently wins more games than you would expect from the underlying stats.”

What does this say? If Mike Tomlin consistently proves your stats wrong, doesn’t that mean your statistics might be wrong? That your win probability calculators might not be universally applicable? That a coach who has never had a losing season might understand the game is not solely about number crunching?

Frankly, statistics are almost always far too coarse to make individual decisions in football anyway. Let’s go through the simplistic game tree that Greg Olsen must have worked out in his head. Without context, teams convert on 4th-and-1 about 65% of the time. Teams score when starting at their 20-yard line about 20% of the time. Teams score from their 40-yard line more than 50% of the time. Again, these probabilities are close enough for actuarial work. This means the odds of winning when going for it are 82.5% (65% + 35% x 50%). The odds of winning when kicking are 80%. I dunno guys, that doesn’t look like a no-brainer call to me, especially when the opposing team is starting Carson Wentz at quarterback.

The idea that football coaches who defy analytics are victims of outcome bias is symptomatic of the broader decision-theoretic mind virus. Decision theory itself is a deeply flawed framework for decision making. It not only assumes that we can capture statistical frames of reference for every decision we will ever make in our lives, but that we can chain such statistics together without destroying the statistics of the past. Somehow, people insist on blindly following this framework while refusing to engage with the Lucas Critique.

I like to point out the absurdity in football because it’s clear where the mindset goes wrong in the context of an entertaining game. But the same critiques apply tenfold to the technocrats who believe we can data-drive our way to optimal policy and politics. Lily Hu recently wrote a phenomenal review of this bizarre mindset in democratic politics in the Boston Review.

It’s not surprising that cynical democratic consultants riding on Nate Silver’s coattails have adopted the same destructive language as sports analytics nerds. Hu discusses the bizarre, clearly inaccurate assumptions that underlie Lakshya Jain’s “wins above replacement” score of political candidates. The vulgar cynicism required to name a political analysis after a baseball statistic bleeds into treating politics as if it were a sporting event. Just like football analytics seems to always argue for more aggressive play, political analytics always seems to argue for adopting the politics of the analysts. Funny how that works.

Hu also emphasizes an important point: politics is predicated on being able to change the world. If that’s the case, then statistics are never representative of how you should act. Outcome bias is what you want in politics. Decision Theory is useless if you want to change the world. If we want to create a different world, we can’t simply mimic what’s worked in the past.