I don’t think that’s the right way to look at it. The administration has a bunch of different actors with very strongly held beliefs that are simultaneously making decisions. These people don’t all agree with each other, but Trump is letting everyone who is nice to him do whatever they want at the same time. So the collective behavior looks incoherent, but the individual behavior is quite calculated.
Reproducibility issues are not uniformly spread. Social sciences seem to have a particular problem, as the subjects have agency and know about prior experiments. The famous Milgram experiment is not reproducible in its original form because we all know about it and what it says about the test subjects.
Medical science is another, partly because it is funded by vested interests that want favorable outcomes for their interests. Data mining was also a problem, which noted journals like JAMA have countered by demanding that the experiment's aim is published first to avoid being rewritten around a p-value discovered "target".
China has been vexed with extreme publish or perish incentives for researchers. And let us not forget that the US has a similar problem that incentivizes researchers to cheat.
But reducing funding is not the answer. Lack of reproducibility just means that science is doing its correction work. It helps cancel the noise of false results.
How can we "pick gold standard winners"? We cannot, any more than governments can generate a track record of always picking winners for industrial policy.
In reality, federal funding of science has declined a fraction of GDP. The slack has been taken up by commercial R&D, with the result that blue-sky experimentation has declined. This lack of blue-sky funding has also affected the NSF, where funding has become more cautious, aimed at experiments where the outcome is fairly assured. So science increasingly fills in the details.
As a warning, look what happened in Nazi Germany when the "Jewish science" was destroyed and the scientists forced out. The result was that those Jewish scientists helped ensure that the US was able to build the atomic bomb first. Germany was very lucky to have surrendered before it was quite ready, otherwise, Berlin might have been the first target, rather than Hiroshima. We should be very, very wary of going down a similar path. As it is, we may have already disrupted the US lead in science, almost guaranteeing that we lose our future global standing.
I live in Sweden where the schools mostly stayed open during covid and from here the school closings in the US looked as crazy ad they turned out to be in hindsight. I think covid responses as the proximate cause of the attacks on American universities is only capturing part of the picture though. I think that many people also remember how the so called experts also caused the global financial crisis and this plays in as well.
Ultimately though the real cause is I believe an epistemic one: scientists are elites even if we won't admit it and social inequality leads elites to inhabit a different world than most people. Worse, being trained to be an elite is largely a training in how to follow the rules, so this is not conducive to paradigm shifts.
Finally, historically most innovation has been the product of research within firms, so perhaps it is advisable to move away from a university lead research model anyway.
I was with you up to that last sentence! There's an ebb and flow between academic and industrial research, no? Moreover, plenty of science at the university is targeted towards building a system of understanding rather than innovation.
This is the trickier part of the debate surrounding reproducibility. A lot of what is called "science" in these debates (e.g., medicine, development, etc.) is probably better described as engineering.
I am not trying to claim that there is no role for the state in organizing research. But consider that thermodynamics and information theory were both developed largely or entirely by people in firms trying to solve very practical problems. Perhaps another way to put my claim (I would not claim it too strongly as a fact) is that if companies were forced to do their own research as opposed to relying on the state to do it for them, said companies might have a better insight into what research they need to do. Perhaps the division of labor is too advanced for this model to work any more. But the fundamental point I want to make is about decentralization of decision making, not really that companies necessarily do research better than unis.
Even in the context of engineering it's an ebb-and-flow. For example, the first publicly known practical use of the Kalman filter was by the NASA Ames Research Center for the Apollo mission (https://ntrs.nasa.gov/api/citations/19860003843/downloads/19860003843.pdf). It became an industry mainstay only after that.
I'm curious what part of it was crazy? Sweden so I understand did push for schools to go remote and did implement limits on gatherings as well as masking and distancing rules. Sweden also implemented functional contact tracing according to public reports which the US did not.
Is it that our schools stayed closed for so long or was the closing at all odd?
In Sweden only high schools went remote and only for a short period. The limits om public gatherings we only on very large events. Bars were also eventually made to close a few hours early. There were no masking or distancing rules. I never heard about contact tracing (or maybe don't remember). The main actions taken were having robust sick leave policies already in place.
Interesting, what I read online suggested that Sweden did have masking rules and contact tracing but also that many Swedes also generally complied with the distancing without being forced.
What is the general level of health and healthcare in Sweden?
I ask because in my experience one of the things that stood out in the US is that some states (e.g. Colorado) performed better than others (e.g. Florida) in part because they just had better healthcare an overall healthier population, and better compliance, despite the same level of public restrictions.
There were no mask requirements. There was for a time a recommendation to wear masks on public transit but compliance wasn't high, may 15% at peak. Everyone gets healthcare. You have unlimited sick days but don't get paid on the first sick day and need a doctor note after 14 days. You also get sick days if your kid is sick. I often travel for work and when I went to Italy or Germany where masks were required you would see people on the train with masks but everyone would sit in the office with a mask hanging from one ear in case someone who actually cared about masks came by. That is clearly a worse stragegy than just having people who can work from home.
as Stafford Beer said "it is only when you have evidence that some part of a research budget has definitively been wasted that you can have any confidence whether any of it has been well spent at all".
It's difficult to distinguish between good and bad science, or at least to predict what will be quickly forgotten and what will stand the test of time. The field of AI offers a compelling example of this conundrum.
One thread I'd like to pull on more is that engineering research is a very particular branch that plays by different rules than those of what most people call "science." And also that medical research is much closer to engineering than science. To be continued...
Frequently whole scientific communities get stuck in dead ends but do not realize or cannot acknowledge that they are stuck. Is there some way to incentivize change?
There is a lot of truth in "one funeral at a time". And it is also true that progress demands sacrifice. Still, personally I am opposed to killing off senior researchers on humanitarian grounds.
Can you unpack how Bhattacharya isn't a demon? He ran one bad study of COVID infection rates, got the result very very wrong, and then used it to make policy recommendations that, if followed, would have killed many many people. Rather than take ownership of any part of that, or just learn from experience, he is has called for accountability for the people who disagreed with him and is now apparently pushing to deny vaccines to pregnant women and children.
While he may be personally good and compelling nothing in his public actions suggests that he should be trusted. I'm honestly open to counter evidence but I have not seen any.
I wouldn't call anyone a "demon", including Bhattacharya. But I agree, his public protestations, intemperance, and advocacy for bad policy have rendered him, if nothing else, an exceptionally unreliable narrator.
Hi Ben, first time your blog has moved me to comment. Who defines what is "good" and "bad" science? Is it the administration? Is it you? It strikes me that this is a bad faith argument, and trying to engage in it can yield no benefits.
We clearly do not "need" bad science (by whatever definition). In fact, we work very hard to avoid it. Researchers serve on review panels for grants; they argue against grants being awarded that would not (in their professional opinion) yield good science.
Vance is arguing in bad faith, but I don't think that science-reform metascientists are. Those who argue for "better science" have their heart in the right place, even though I don't think their desire to impose rigid rules on how to conduct science will achieve much of anything. I have written about this before, but I'm in favor of open science, open code. I'm not in favor of forcing everyone to preregister their data analysis plans and teaching everyone about robust confidence intervals. Scientists should be free to explore new methods and ways of seeing and interpretation.
Now, with regards to your lat paragraph, I disagree with your assessment of scientists working hard to avoid promoting bad science by others. I have a different opinion of peer review, be it about papers or grants. Our prestige systems fail to determine what is good and bad all the time. People don't work hard at peer review. Grant panels are usually plagued by old guys yelling at clouds. Scientists are prone to herding around hype. That science trudges forward despite all of the apparent "irrationality" is what makes it such a remarkable human endeavor.
Wondering if "give us ever increasing amounts of money and don't ask too many questions about what we do with it, it's always been a mess and the mess is good and necessary actually" might be construed as a bit too self-serving by the general public.
"don't ask too many questions about what we do with it"
Every cent spent on a grant needs to be accounted for when reporting, and heaven forbid if it was spent on something outside of the scope of the grant (e.g., a printer)! Every study funded by a grant has been searchable and is folded into every year's update. Hell go to NIH Reporter if you want to verify for yourself.
Transparency has been baked into the entire process, for at least my brief time in the sciences so far, and I dont see much evidence to suggest this has been a recent change.
So let's all drop the strawman bullshit about being self serving.
Yes to this. There is so much reporting, explaining, and oversight, not to mention consequences for mistakes that the idea that there are no questions is disconnected. Indeed I would argue that the primary result from reducing overhead will just mean more reporting and more paperwork, technically copy charges, phone fees, etc, can be put back into the budget lines and they will, so more paperwork with no savings.
Wonderful. They've decided to replace a reproducibility crisis with a producibility crisis.
The admin does not have a plan for literally any executive decision they take.
I don’t think that’s the right way to look at it. The administration has a bunch of different actors with very strongly held beliefs that are simultaneously making decisions. These people don’t all agree with each other, but Trump is letting everyone who is nice to him do whatever they want at the same time. So the collective behavior looks incoherent, but the individual behavior is quite calculated.
Reproducibility issues are not uniformly spread. Social sciences seem to have a particular problem, as the subjects have agency and know about prior experiments. The famous Milgram experiment is not reproducible in its original form because we all know about it and what it says about the test subjects.
Medical science is another, partly because it is funded by vested interests that want favorable outcomes for their interests. Data mining was also a problem, which noted journals like JAMA have countered by demanding that the experiment's aim is published first to avoid being rewritten around a p-value discovered "target".
China has been vexed with extreme publish or perish incentives for researchers. And let us not forget that the US has a similar problem that incentivizes researchers to cheat.
But reducing funding is not the answer. Lack of reproducibility just means that science is doing its correction work. It helps cancel the noise of false results.
How can we "pick gold standard winners"? We cannot, any more than governments can generate a track record of always picking winners for industrial policy.
In reality, federal funding of science has declined a fraction of GDP. The slack has been taken up by commercial R&D, with the result that blue-sky experimentation has declined. This lack of blue-sky funding has also affected the NSF, where funding has become more cautious, aimed at experiments where the outcome is fairly assured. So science increasingly fills in the details.
As a warning, look what happened in Nazi Germany when the "Jewish science" was destroyed and the scientists forced out. The result was that those Jewish scientists helped ensure that the US was able to build the atomic bomb first. Germany was very lucky to have surrendered before it was quite ready, otherwise, Berlin might have been the first target, rather than Hiroshima. We should be very, very wary of going down a similar path. As it is, we may have already disrupted the US lead in science, almost guaranteeing that we lose our future global standing.
I live in Sweden where the schools mostly stayed open during covid and from here the school closings in the US looked as crazy ad they turned out to be in hindsight. I think covid responses as the proximate cause of the attacks on American universities is only capturing part of the picture though. I think that many people also remember how the so called experts also caused the global financial crisis and this plays in as well.
Ultimately though the real cause is I believe an epistemic one: scientists are elites even if we won't admit it and social inequality leads elites to inhabit a different world than most people. Worse, being trained to be an elite is largely a training in how to follow the rules, so this is not conducive to paradigm shifts.
Finally, historically most innovation has been the product of research within firms, so perhaps it is advisable to move away from a university lead research model anyway.
I was with you up to that last sentence! There's an ebb and flow between academic and industrial research, no? Moreover, plenty of science at the university is targeted towards building a system of understanding rather than innovation.
This is the trickier part of the debate surrounding reproducibility. A lot of what is called "science" in these debates (e.g., medicine, development, etc.) is probably better described as engineering.
I am not trying to claim that there is no role for the state in organizing research. But consider that thermodynamics and information theory were both developed largely or entirely by people in firms trying to solve very practical problems. Perhaps another way to put my claim (I would not claim it too strongly as a fact) is that if companies were forced to do their own research as opposed to relying on the state to do it for them, said companies might have a better insight into what research they need to do. Perhaps the division of labor is too advanced for this model to work any more. But the fundamental point I want to make is about decentralization of decision making, not really that companies necessarily do research better than unis.
Even in the context of engineering it's an ebb-and-flow. For example, the first publicly known practical use of the Kalman filter was by the NASA Ames Research Center for the Apollo mission (https://ntrs.nasa.gov/api/citations/19860003843/downloads/19860003843.pdf). It became an industry mainstay only after that.
I'm curious what part of it was crazy? Sweden so I understand did push for schools to go remote and did implement limits on gatherings as well as masking and distancing rules. Sweden also implemented functional contact tracing according to public reports which the US did not.
Is it that our schools stayed closed for so long or was the closing at all odd?
In Sweden only high schools went remote and only for a short period. The limits om public gatherings we only on very large events. Bars were also eventually made to close a few hours early. There were no masking or distancing rules. I never heard about contact tracing (or maybe don't remember). The main actions taken were having robust sick leave policies already in place.
Interesting, what I read online suggested that Sweden did have masking rules and contact tracing but also that many Swedes also generally complied with the distancing without being forced.
What is the general level of health and healthcare in Sweden?
I ask because in my experience one of the things that stood out in the US is that some states (e.g. Colorado) performed better than others (e.g. Florida) in part because they just had better healthcare an overall healthier population, and better compliance, despite the same level of public restrictions.
There were no mask requirements. There was for a time a recommendation to wear masks on public transit but compliance wasn't high, may 15% at peak. Everyone gets healthcare. You have unlimited sick days but don't get paid on the first sick day and need a doctor note after 14 days. You also get sick days if your kid is sick. I often travel for work and when I went to Italy or Germany where masks were required you would see people on the train with masks but everyone would sit in the office with a mask hanging from one ear in case someone who actually cared about masks came by. That is clearly a worse stragegy than just having people who can work from home.
as Stafford Beer said "it is only when you have evidence that some part of a research budget has definitively been wasted that you can have any confidence whether any of it has been well spent at all".
Lmao Ben never misses an opportunity to dunk on RDDs.
Amen, brother.
It's difficult to distinguish between good and bad science, or at least to predict what will be quickly forgotten and what will stand the test of time. The field of AI offers a compelling example of this conundrum.
One thread I'd like to pull on more is that engineering research is a very particular branch that plays by different rules than those of what most people call "science." And also that medical research is much closer to engineering than science. To be continued...
Frequently whole scientific communities get stuck in dead ends but do not realize or cannot acknowledge that they are stuck. Is there some way to incentivize change?
"Science progresses one funeral at a time."
There's much truth in that statement.
But also many paths can appear to be dead ends before ultimately leading to breakthroughs...
There is a lot of truth in "one funeral at a time". And it is also true that progress demands sacrifice. Still, personally I am opposed to killing off senior researchers on humanitarian grounds.
Hah, I agree that as senior researchers we have to argue on behalf of our own self-interest!
Shh.. Don't tell anyone!
"If we need the bad science to get the good science, a tenfold cut could lead to a hundredfold contraction of innovation."
What scaling law do you assume here for the good/bad science ratio?
LOL. Clearly a power law...
Can you unpack how Bhattacharya isn't a demon? He ran one bad study of COVID infection rates, got the result very very wrong, and then used it to make policy recommendations that, if followed, would have killed many many people. Rather than take ownership of any part of that, or just learn from experience, he is has called for accountability for the people who disagreed with him and is now apparently pushing to deny vaccines to pregnant women and children.
While he may be personally good and compelling nothing in his public actions suggests that he should be trusted. I'm honestly open to counter evidence but I have not seen any.
I wouldn't call anyone a "demon", including Bhattacharya. But I agree, his public protestations, intemperance, and advocacy for bad policy have rendered him, if nothing else, an exceptionally unreliable narrator.
Hi Ben, first time your blog has moved me to comment. Who defines what is "good" and "bad" science? Is it the administration? Is it you? It strikes me that this is a bad faith argument, and trying to engage in it can yield no benefits.
We clearly do not "need" bad science (by whatever definition). In fact, we work very hard to avoid it. Researchers serve on review panels for grants; they argue against grants being awarded that would not (in their professional opinion) yield good science.
Vance is arguing in bad faith, but I don't think that science-reform metascientists are. Those who argue for "better science" have their heart in the right place, even though I don't think their desire to impose rigid rules on how to conduct science will achieve much of anything. I have written about this before, but I'm in favor of open science, open code. I'm not in favor of forcing everyone to preregister their data analysis plans and teaching everyone about robust confidence intervals. Scientists should be free to explore new methods and ways of seeing and interpretation.
Now, with regards to your lat paragraph, I disagree with your assessment of scientists working hard to avoid promoting bad science by others. I have a different opinion of peer review, be it about papers or grants. Our prestige systems fail to determine what is good and bad all the time. People don't work hard at peer review. Grant panels are usually plagued by old guys yelling at clouds. Scientists are prone to herding around hype. That science trudges forward despite all of the apparent "irrationality" is what makes it such a remarkable human endeavor.
Wondering if "give us ever increasing amounts of money and don't ask too many questions about what we do with it, it's always been a mess and the mess is good and necessary actually" might be construed as a bit too self-serving by the general public.
"don't ask too many questions about what we do with it"
Every cent spent on a grant needs to be accounted for when reporting, and heaven forbid if it was spent on something outside of the scope of the grant (e.g., a printer)! Every study funded by a grant has been searchable and is folded into every year's update. Hell go to NIH Reporter if you want to verify for yourself.
Transparency has been baked into the entire process, for at least my brief time in the sciences so far, and I dont see much evidence to suggest this has been a recent change.
So let's all drop the strawman bullshit about being self serving.
Yes to this. There is so much reporting, explaining, and oversight, not to mention consequences for mistakes that the idea that there are no questions is disconnected. Indeed I would argue that the primary result from reducing overhead will just mean more reporting and more paperwork, technically copy charges, phone fees, etc, can be put back into the budget lines and they will, so more paperwork with no savings.
Oh, I have no doubt that they would have the receipts for that $11k they paid to Springer Nature for a useless, junk paper.
Edit: turns out it's actually $12690.
Masterful goalpost maneuver! You should take pride in your useless cynicism!