This Is Fine
There is no reproducibility crisis in science. So what is JD Vance on about?
JD Vance, the humorless dorky academic wannabe who serves as Vice President, took to Elon Musk’s right-wing microblogging site this weekend with a masterclass in sophistry. Justifying the administration’s stance on higher education, he tweeted,1
“There is an extraordinary ‘reproducibility crisis’ in the sciences, particularly in biology, where most published papers fail to replicate.”
Far away on Bluesky, the internet’s Yavin 4, massive consternation erupted. Vance was using the language of science reform to attack science itself. What had felt like a good-faith effort to rigorize scientific practice was now being used as part of the reason for defunding all of science. An effort to make science more open would lead to science closing down.
You can’t take Vance and his tweeting as serious argument. History is governments with regimes finding one reason or another to attack universities. You don’t even have to look to scary regimes like Nazi Germany or modern right-wing authoritarian states. Even in the height of the Cold War university expansion, tensions between American universities and governments flared over communist witch hunts. America has a proud history of politicians casting universities as hotbeds of radical dissident thought that leech off the government.
Still, the newfound obsession with reproducibility is worth unpacking. Vance didn’t come to his assessment of the state of scientific research on his own. A recent executive order echoes the same language. “A majority of researchers in science, technology, engineering, and mathematics believe science is facing a reproducibility crisis.” The order continues: “Over the last 5 years, confidence that scientists act in the best interests of the public has fallen significantly.”
I wonder what happened over those 5 years? Is the Trump administration upset about Francesca Gino’s fraudulent research on dishonesty? Let us read on.
“For example, under the prior Administration, the Centers for Disease Control and Prevention issued COVID-19 guidance on reopening schools that incorporated edits by the American Federation of Teachers and was understood to discourage in-person learning. This guidance’s restrictive and burdensome reopening conditions led many schools to remain at least partially closed, resulting in substantial negative effects on educational outcomes — even though the best available scientific evidence showed that children were unlikely to transmit or suffer serious illness or death from the virus, and that opening schools with reasonable mitigation measures would have only minor effects on transmission.”
Welp. They are still pissed about COVID.2 And we can’t understand this current attack on the American academy without talking about its hyperpolarization during the pandemic years.
In this week's New Yorker, Daniel Immerwahr has a must-read piece on RFK and the complex relationship between science, policy, and conspiracy. Citing the work of sociologist Gil Eyal, Immerwahr describes how political decisions can render scientific opinions as truth, leading communities to ostracize scientific skeptics.
“It’s when uncertainty collides with urgency that the authorities enter the fray, convene commissions, and issue findings. Those who accept the sanctioned conclusions gain official backing. Those who don’t are ruled out of bounds. No longer recognized as colleagues with legitimate hypotheses, they risk being treated as crackpots, deniers, and conspiracy theorists.”
Immerwahr notes that decisions have to be made, so some degree of boundary setting is necessary. However, there is a downside to turning dissenters into apostates.
“Keep things too open and you’re endlessly debating whether Bush did 9/11. Close them too quickly, though, and you turn hasty, uncertain conclusions into orthodoxies. You also marginalize too many intelligent people, who will be strongly encouraged to challenge your legitimacy by seizing on your missteps, broadcasting your hypocrisies, and waving counter-evidence in your face.”
The biggest downside is that in our fractured, polarized American politics, every four years, the crackpots get to take over the house.3
One marginalized “COVID crackpot,” Jay Bhattacharya, is now the head of the NIH. I know Jay well, and he’s not the evil demon imagined by some parts of online left-wing discourse. I know Jay agrees with the administration’s reproducibility assessment. I also know many of the folks on the HHS team (notably Marty Makary and Vinay Prasad at the FDA) are also on board.
The argument of these medical scientists goes that since most science doesn’t replicate, it’s ok to cut it back tenfold and only fund the best. This opinion is more widespread than you might think. Friend-of-the-blog John Mandrola concurs:
“I know many people worry that the new administration has cut science funding. I don’t. Because every week, when I review medical evidence, I see tremendous amounts of waste.”
Because I appreciate John’s critical eye and enjoy reading his blog, it’s here that I want to push back. As Immerwahr points out, scientific waste is a subjective concept. Is the fraction of wasteful science larger than ever? Has anyone attempted a metascientific account of the reproducibility of science in 1925? What if, bear with me, science has always been a mess? What if most scientific papers have always been forgotten? What if, as Dan De Kadt tweeted, “Spicy take: There is no replication (or "reproducibility") crisis in science?”
These rhetorical questions merely echo the basic arguments of Thomas Kuhn. Scientists love Kuhn because they all think they will be the paradigm shifter. By being more rigorous and by adhering to “Gold Standard Science,” they will be the ones who pipette themselves to glory. But there is no paradigm shift without the vast worker bees pushing the current paradigm along. Even in my field, where everything is highly “reproducible” as code, most of the work is pretty bad and immediately forgotten.
No one likes to admit it, but we need bad science to do good science. Learning from the mess is how the “progress” occurs. It’s quite possible that gutting science funding will have an opposite effect. If we need the bad science to get the good science, a tenfold cut could lead to a hundredfold contraction of innovation. If I’m right about this, then the administration’s actions will be proven disastrous. I look forward to the million regression discontinuity studies a decade from now that argue both sides of the case.
I don’t care where you microblog, I will forever memorialize the act of microblogging as Tweeting.
I’m still pissed about COVID too! I end up talking about it in subtext on the blog, though sometimes that subtext is far from subtle. We are nowhere near closure about those years.
Like Benicio Del Toro in The Last Jedi, I’m not taking sides here. A pox on both houses!
Wonderful. They've decided to replace a reproducibility crisis with a producibility crisis.
Reproducibility issues are not uniformly spread. Social sciences seem to have a particular problem, as the subjects have agency and know about prior experiments. The famous Milgram experiment is not reproducible in its original form because we all know about it and what it says about the test subjects.
Medical science is another, partly because it is funded by vested interests that want favorable outcomes for their interests. Data mining was also a problem, which noted journals like JAMA have countered by demanding that the experiment's aim is published first to avoid being rewritten around a p-value discovered "target".
China has been vexed with extreme publish or perish incentives for researchers. And let us not forget that the US has a similar problem that incentivizes researchers to cheat.
But reducing funding is not the answer. Lack of reproducibility just means that science is doing its correction work. It helps cancel the noise of false results.
How can we "pick gold standard winners"? We cannot, any more than governments can generate a track record of always picking winners for industrial policy.
In reality, federal funding of science has declined a fraction of GDP. The slack has been taken up by commercial R&D, with the result that blue-sky experimentation has declined. This lack of blue-sky funding has also affected the NSF, where funding has become more cautious, aimed at experiments where the outcome is fairly assured. So science increasingly fills in the details.
As a warning, look what happened in Nazi Germany when the "Jewish science" was destroyed and the scientists forced out. The result was that those Jewish scientists helped ensure that the US was able to build the atomic bomb first. Germany was very lucky to have surrendered before it was quite ready, otherwise, Berlin might have been the first target, rather than Hiroshima. We should be very, very wary of going down a similar path. As it is, we may have already disrupted the US lead in science, almost guaranteeing that we lose our future global standing.