The Good, The Bad, and The Science

Who should define what is "good" and "bad" science?

Mark Mueller asked a really important question in the comments of Monday’s post: “Who defines what is 'good' and 'bad' science?” Much of the current war the administration is waging against universities is rooted in this question of who gets to claim the mantle of scientific adjudication.

Should it be the scientists themselves? Mark argues that “Researchers serve on review panels for grants; they argue against grants being awarded that would not (in their professional opinion) yield good science.” I disagree with Mark’s assessment of peer review. Our prestige systems fail to determine what is good and bad all the time. Peer review completely fails to improve science and can’t even reliably detect fraud. At this point, it’s clear most scientists don't work hard at paper reviews, and LLM reviews are rapidly on the rise. Grant panels are frequently plagued by old guys yelling at the clouds that forgot to cite their work. Scientists are prone to herding around hype. That science trudges forward despite all of the apparent "irrationality" is what makes it such a remarkable human endeavor.

Should it be the funding agencies? Manfred Morari sent me a video of Michael Kratsios, the current head of OSTP and former chief of staff to Peter Thiel, addressing the National Academy of Sciences about Gold Standard Science and the administration’s science funding priorities. Given his background with Thiel, it was unsurprising to hear Kratsios arrogantly claim that they could cut funding and improve science at the same time. "Spending more money on the wrong things is far worse than spending less money on the right things." He blathered on about “Prizes, challenges, public-private partnerships, and other novel funding mechanisms, can multiply the impact of targeted federal dollars,” as if DARPA challenges and SBIR grants didn’t exist for decades. Like everyone else in this administration, he didn’t articulate a clear plan of how OSTP was going to ensure this gold standard and leaned on the tired grievances of COVID policies and DEI initiatives. We know the Thielites have been set on destroying universities, and we shouldn’t be surprised they’re trying to do it now, given the opportunistic time when they staff parts of the administration. But it’s not at all clear that they have anything but their tired bromides about disruption.

Should it be the federal bureaucracy? I mean, this one is easy. No! Part of the mess we’re in now is because the pandemic response cast science, which is always a mess, as THE SCIENCE, which enforced policy. The federal science agencies in the US were not equipped for this task. I was pissed during COVID at the absurdly shoddy, closed work being shovled by the CDC as evidence. The CDC’s house journal, the lugubriously titled Morbidity and Mortality Weekly Report, put out horrible, easily debunkable studies with cheesy infographics. MMWR was the opposite of open. Its editorial policies were at best murky. None of the papers adhered to standards of transparent data or reporting. The MMWR editorial policy was to publish only papers that aligned with CDC political policy.

This would be fine if no one read these things, but if the CDC published something in MMWR, it became effectively true as legal policy. Court documents would cite MMWR as evidence for more aggressive school closings, vaccine mandates, and other restrictions. It should have been clear that this arrogant expertise, holding up clearly questionable research as unassailable foundations of policy, would backfire. But you can still go over to Bluesky and find lefty people singing about adhering to the unassailable advice of experts.

But I’ll punch right too! A pox on all houses. The Gold Standard Science crew under RFK certainly isn’t living up to their word. A new report from NOTUS found that the Make America Healthy Report is riddled with made-up citations. [Update: The New York Times found even more made-up citations later the same day.] Who could have guessed? The new federal bureaucracy is even worse than the old one! He won’t say anything because he stopped trolling on social media after being appointed to the FDA, but I’d be curious to hear Vinay Prasad’s reaction. He spent the last decade railing against shoddy work like this. Now the bad metaanalyses are coming from inside his house.

So who should decide what’s good and bad science? Look, I’m as pissed as the rest of you. I think this brazen attack on universities is maddening and hallmark of reactionary populist regimes. It’s compounded by the arrogance of Silicon Valley who, despite getting their degrees and heavily recruiting from Berkeley and Stanford, wants universities closed. On the other hand, I think science is a permanent mess and have blogged about the issues with numbers, causality, statistics, and observational studies. So what’s the right middle ground? What is the argument for the Biden level of US government funding of science? How do we make a case for why this funding is essential despite the slow and confusing emergence of progress from a lot of “bad” work?

Dammit, I’m going to ask my friends in the much maligned humanities. I was recently on a panel with Zeus Leonardo, a professor of Education here at Berkeley. In making sense of the administration’s attacks on higher education, Zeus described the dialectic between acquiring expertise and claiming expertise. One can be a scholar who devotes their life to acquiring expertise in a subject without putting themselves forward as an expert. Research is curiosity-driven, but when you declare yourself an expert, you abandon that curiosity. You abandon the embrace of uncertainty. You declare the knowledge you have sufficient to make decisions. Rather than explaining all you don’t know, you declare yourself an arbiter of truth.

Making decisions in the face of uncertainty is a core task democratically delegated to government. But it’s never the science that makes the decision. The Gold Standard might just be a firm stance that science can’t answer political questions, and any science that claims to do so is probably more akin to a cryptocurrency shitcoin.