Can anyone fix science?
Science has always been in crisis. This is fine.
This is Part 1 of 7 of a blogged essay “Steampunk Data Science.” A table of contents is here.
In a back-and-forth with Ryan Briggs about the “crisis” of hiding null results in political science, Stephen Wild asked a loaded question.
“How different is the current situation from the past, whether it be 50 years ago or 150 years ago? Phrased in another, slightly more provocative way, what if it’s always been this way?”
In response, Ryan suggested that quantitative social science is very young and is getting better, and can be fixed. I’m not sure. Quantitative social science is only young by relative standards. You could argue the modern discipline starts after the French Revolution with academic bureaucrats like Adolphe Quetelet in the 1800s. Udny Yule was using linear regression to suss out the causation of poverty in 1900. John Maynard Keynes wrote about probabilistic methods for understanding society in the 1920s. The first application of machine learning techniques to the prediction of prison recidivism was in 1928.
While quantitative social science isn’t new, you could argue that we’ve reached a new crisis. Perhaps because of publish-or-perish incentives, science has reached new lows, where nothing is reproducible, and most published results are wrong. Researchers, journalists, and politicians alike complain that contemporary science faces a terrible reproduction crisis. People love writing meta-analyses like the one I wrote about last week. Psychology has been in crisis for a decade. Cancer research is in crisis. Machine learning is in crisis.
Conventional wisdom suggests that the reproduction crisis threatens the fundamental legitimacy of the sciences. Operatives in the United States federal government have been using this “crisis” as an excuse to slash science funding and attack universities. All of this panic begs Stephen’s question of when science used to be better. It implicitly suggests that there was a golden age when all published results were true, and scientists diligently reproduced each other’s work.
Is this true? Was reproduction a core element of prior scientific or engineering revolutions?
It’s not hard to find people decrying the demise of science throughout history. I wrote about when Science Magazine dedicated an entire editorial section to worrying about the public’s turn against science in 1957. The fifties were supposedly the wonder years. You want to go back even further? Read Charles Babbage complaining about the state of affairs in 1830.
Now, you might argue, like Ryan does, that better methods ameliorated these crises. That somehow doing science better fixed the fields. The null hypothesis significance test, the metascientist’s best friend, is barely one hundred years old. Is it true that the science of the 1800s was doomed and hopeless because they couldn’t run randomized controlled trials and compute p-values?
In the midst of the covid lockdowns, I also found myself intrigued by this question and started digging into how science was done before the formalization of statistics. How did we figure out which interventions worked before the null hypothesis test? I ended up dumping this research into a long essay about one of the most important discoveries in human-facings sciences of the early 20th century: vitamins.
Some discoveries have simple, pithy origins, like finding mold in a petri dish. But discovering the body’s need for vitamins took decades, as physicians, biologists, and chemists assembled evidence from an array of confusing empirical findings from many distant parts of the world. There were dozens of erroneous hypotheses put forward as truths in the literature. There were sloppy data practices and poorly documented experimental protocols. But through this confusion, clarity eventually coalesced into a completely new understanding of diet and disease.
I wrote this up about four years ago, but I’ve never figured out how to publish it. Initially, I thought it would be a core part of my book on automated decision making, The Irrational Decision (preorder now! It helps the substack), but the vitamin story ended up being too much of a prequel. The discovery of vitamins happened two world wars before the development of the computer. It happened a decade before the formalization of statistics. That it’s one of the last major discoveries of the pre-data scientific age is why I find the story so fascinating.
Since The Irrational Decision comes out in two weeks, it feels appropriate to put out the prequel now. My plan is to interleave two threads: I’ll continue my liveblogging of class, and I’ll tell the story of the history of vitamins as a quintessential case study of the human-facing sciences before the mathematical formalization of statistical inference.
The discovery of vitamins and deficiency diseases shows how a chaotic lack of reproducibility might be core to science itself, and that we were quite capable of making sense of the natural world before we had access to spreadsheets and statistics software. I’ll try to convince you that none of our fancy modern machinery would have helped at all. Both then and now, there are no formal paths to discovery.
"The discovery of vitamins and deficiency diseases shows how a chaotic lack of reproducibility might be core to science itself"
Is this a descriptive or normative claim? If descriptive, then sure, nobody disagrees with your title "science has always been in crisis". But if it's normative, then no argument has been given, and hence, the second part of the title "This is fine" has been entirely ignored in substance. You've offered a single example of a successful discovery made pre-formal methods--what of the thousands of failures? To be sure, this post seems merely suggestive so maybe I am being too harsh. Looking forward to your substantive arguments in future posts then.
I recently read Carlos Chavez’s line that “the credibility revolution succeeded by developing research designs where the identifying assumptions were embedded in the structure of the data rather than imposed by the researcher.” (https://carloschavezp29.substack.com/p/why-macro-never-had-a-credibility).
To me, that framing captures the wrong turn that keeps the social sciences in permanent crisis. The identifying assumptions never vanish, they just move upstream into measurement, classification, sampling, and the choice of what even counts as "treatment," "unit," and "outcome." But many applied social scientists still convince themselves that data are somehow self-interpreting facts rather than acknowledge that they become evidence only through theory, and are already theory-laden categories. If the theory is thin or wrong, the categories can be wrong, and the statistics will deliver precise answers to a misconceived question. The fix is not to pretend we can design away judgment, it is to make judgment explicit, contestable, and honestly limited in scope.