Individual Experience vs. The Cochrane Review

On my decade-long exploration seeking a scientific language for singular evidence.

I had a bit of a throwaway line in the last post about how maximizing the welfare of populations requires the erasure of individuals. You might ask why? Individuals are units in a broader population. A population is a group of individuals. Improving the welfare at scale must improve the welfare of the units.

Except we all know this isn’t true. Maximizing averages doesn’t say anything about the outcomes of any particular individual. In fact, decisions that maximize averages often harm some of the individuals in the total sum. Individuals in a community always have some shared interests, but they have plenty of disparate interests too. Which interests are maximized is a political decision that necessarily leaves other interests neglected. Our metrics and measures can have broad societal value while still making many unhappy.

And how should those individuals make decisions about their own lives? You may be able to convince yourself that Mathematical Rationality makes sense for a bureaucratic state or company. However, it’s much harder to make the case for being a mathematically rational individual.¹ Quantification, abstraction, hierarchy, and statistics can help organize and steer decision making at scale. But if one of the core goals of quantification is legibility for intersubjectivity, why do you need numbers to make sense of your personal experience? Why is it useful to see yourself like a state?

Nothing motivates my research more than this tension between the population and the individual. It’s been my main focus since 2020.² But getting traction on these topics has been an uphill battle. Try telling someone in the human-facing sciences that you want to study the epistemology of case studies. It’s so easy to fall into the cracks of crankhood.

Now, of course, scientists are incapable of seeing the pure irrationality of science. Blindly applying population results to individuals requires a lot of faith. We have formal scientific language to understand population averages. This language is incoherent when directed back towards individuals. Take our gold standard of causal inference, the randomized controlled experiment. These trials can estimate the fraction of people in an experimental cohort who would benefit from taking a drug. But let’s say you do a trial with 600 people and find that 20% of the control group has a bad outcome, and 10% of the treatment group has a bad outcome.³ What does this say about my outcome? Unfortunately, that result alone says nothing. We’d like to argue that the intervention reduces my risk by a factor of 2. But what even is individual risk?

Despite this bizarre inability to really say anything about individual benefit, when you try to come up with a non-statistical, non-quantized language to say precise things about people, you are relegated to the bucket of pseudoscience or, if you can bench enough, bro-science. Anecdotes are not data. Your miracle cure is always a fluke. Your personal experience is trumped by this impenetrable 500-page systematic review. Anyone who disagrees with the consensus of experts is being an irrational contrarian.

However, a lot of practices people find beneficial are immune to the postmodern lens of the randomized trial. It’s really hard to do RCTs on organ transplantation. You can’t do RCTs for physical therapy. You certainly can’t do them for massage or chiropractic. It’s hilarious that we have convinced ourselves that we can do this for psychology, despite decades of embarrassing “scientific” failures. And when you start looking at “sports science,” you realize how silly it is to try to put a scientific corset on all of human experience. No randomized trial explains Victor Wembanyama.

I could pick on more than just the RCT. Individuals don’t exist in the calculus of rationality. None of the pillars of mathematical rationality I talk about in The Irrational Decision make much sense for individual people. I could give similar spiels about game theory, statistical prediction, or optimization.

Optimization, in particular, is tough to grapple with. I got a lot of questions about personal optimization in the conversations about my book. Many people find it useful to think about their lives as a collection of optimization problems. If you want to strive for the best, that means some number must go up, right?

Don’t get me wrong, I love optimizing too! Do I obsess about my home coffee setup, my exercise program, my writing schedule? You bet I do. Is that bad? Does that mean I’m just a cog in the capitalist machine? These questions form the basis of a conversation worth having.

Now, here’s an annoying paradox. If we want a language to talk about individual experience, it has to have some element of intersubjectivity. This is where the quantification trap comes in. The mimetic power of the quantification trap means that a shared language for discussing individual experience is always at risk of being contaminated by scientific quantification. But it doesn’t have to. Most people share their experiences without numbers and charts. We obviously share experience through art, music, and literature. These are all shared languages, too.

For the next bit on this blog, I want to find language to talk to each other about individual experiences. I want to write about this weird tension between the quantification trap and the individual. People figure out how to do amazing things without consulting the scientific literature all the time. How can we talk about commonalities without reducing them to numbers or statistics? I initially thought this would be the topic of the final chapter of The Irrational Decision, but I realized it was far too sprawling and unwieldy to fit. It will have to be its own book. Some day.

In the meantime, I’m going to try to type it out on here.

1. Unless you drink a lot of slate-star-less-wrong Kool-Aid.

2. What happened in 2020? I don’t remember.

3. (p<0.001)