So far, nothing about the Open Science movement has convinced me that Philip Mirowski was wrong when he wrote this:
“Almost everyone is enthusiastic that ‘open science’ is the wave of the future. Yet when one looks seriously at the flaws in modern science that the movement proposes to remedy, the prospect for improvement in at least four areas are unimpressive. This suggests that the agenda is effectively to re-engineer science along the lines of platform capitalism, under the misleading banner of opening up science to the masses.”
I understand your skepticism, but people do it all the time. Any time someone tells you they are going to apply causal inference to a problem, they are almost always saying that they believe there is an ATE they can measure. In psychology it might be the size of a priming effect; in political science the amount of turnout induced by a campaign choice; in economics the amount of well being induced by a cash infusion.
"A 5-sigma intervention is one with an estimated average treatment effect that passes a well-stated statistical test with a p-value of less than 1 in a million."
I would add that it has to pass it under the conditions which do not deviate drastically from the assumptions under which the test is derived.
I'm a bit puzzled by this last series of posts. Yes, of course RCTs and p-values are just a convention, but conventions can be useful! We could reason everything from scratch, but then we'd likely make even more errors than we currently do, given that most researchers aren't statisticians.
Linguistics is famously anti-statistical, but I bet I can find lots of 5-sigma effects involving language. For example, give English-speaking subjects newspaper articles that are either from English or Japanese newspapers, and ask an English speaker to decide if the article is in English or not. Maybe I've missed your point?
Is your point that I don't need statistics to know that a person who can read English can recognise if an article is written in English? (Being able to recognise that a text is in English is part of being able to read English, so it's true by definition).
Maybe psycho-physics is what you're looking for? There are very robust effects involving visual and auditory perception which are basically categorical away from the perceptual boundary.
Psychophysics is exactly what I'm looking for, and I wrote about it today. Thanks for the pointer.
FWIW: This whole series of posts arose from arguments with statisticians over my Bureaucratic Statistics paper (https://arxiv.org/abs/2501.03457). There, I argue that RCTs are useful as an approval mechanism. But most of the "inferential" uses are deeply tenuous. I keep looking for counter evidence to my claims there, and I keep coming up empty.
Is "eating fewer calories causes weight loss" medicine or not medicine? Ooh, what about "progressive overload causes strength gains?"
Zooming out, I'm also tempted to half-assedly argue that in recent years, medicine has fiipped to having surprisingly few five sigma interventions? What are the big ones this century? GLP-1's?
In the last 5 years off the top of my head: covid mrna vaccines, glp-1 agonists, crispr gene therapy. If you look earlier, you'll see a lot of great cancer therapies (like gleevec and car-t cell treatment) and vaccines (like gardasil). At some point, I will compile a list.
I certainly think "measure theoretically" or whatever that most interventions don't work. I'm just saying that we find "one in a million" type therapies a lot more than we might think. And it's incredibly hard to find these in fields like economics, political science, psychology, etc.
Propaganda got to be at least 5 sigma on collective human behavior, but tricky to quantify. Also 5 sigma effects on antibody levels does not imply 5 sigma effects on overall mortality, it might still be insignificant on a population level, or not. Transient medical/chemical effects in an individuaI can be 5 sigma initially on some medical marker, but 0-sigma over longer time in the same individual on outcomes that matter. I still do not understand how to define human-facing, or medicine. But it is all interesting. Thank you for writing this.
So far, nothing about the Open Science movement has convinced me that Philip Mirowski was wrong when he wrote this:
“Almost everyone is enthusiastic that ‘open science’ is the wave of the future. Yet when one looks seriously at the flaws in modern science that the movement proposes to remedy, the prospect for improvement in at least four areas are unimpressive. This suggests that the agenda is effectively to re-engineer science along the lines of platform capitalism, under the misleading banner of opening up science to the masses.”
https://journals.sagepub.com/doi/10.1177/0306312718772086
Mirowski never misses.
"divided by the standard deviation times the square root of one minus the sample size"
I think that should be "the square root of the sample size minus one".
Thanks! Fixed it.
How is human-facing science defined?
Yes, I deliberately left that open to interpretation. How would you define it?
Simplest definition seems to be the setting in which the intervention is applied to the human (this definition seems to include economics)
I have in mind a big tent that includes medicine, psychology, political science, economics, and human-computer interaction.
How does one quantify the ATE for interventions applied in the last three settings?
I understand your skepticism, but people do it all the time. Any time someone tells you they are going to apply causal inference to a problem, they are almost always saying that they believe there is an ATE they can measure. In psychology it might be the size of a priming effect; in political science the amount of turnout induced by a campaign choice; in economics the amount of well being induced by a cash infusion.
"A 5-sigma intervention is one with an estimated average treatment effect that passes a well-stated statistical test with a p-value of less than 1 in a million."
I would add that it has to pass it under the conditions which do not deviate drastically from the assumptions under which the test is derived.
Absolutely, but in this post I even cut them slack on this very important clause!
I'm a bit puzzled by this last series of posts. Yes, of course RCTs and p-values are just a convention, but conventions can be useful! We could reason everything from scratch, but then we'd likely make even more errors than we currently do, given that most researchers aren't statisticians.
Linguistics is famously anti-statistical, but I bet I can find lots of 5-sigma effects involving language. For example, give English-speaking subjects newspaper articles that are either from English or Japanese newspapers, and ask an English speaker to decide if the article is in English or not. Maybe I've missed your point?
Is your point that I don't need statistics to know that a person who can read English can recognise if an article is written in English? (Being able to recognise that a text is in English is part of being able to read English, so it's true by definition).
Maybe psycho-physics is what you're looking for? There are very robust effects involving visual and auditory perception which are basically categorical away from the perceptual boundary.
Psychophysics is exactly what I'm looking for, and I wrote about it today. Thanks for the pointer.
FWIW: This whole series of posts arose from arguments with statisticians over my Bureaucratic Statistics paper (https://arxiv.org/abs/2501.03457). There, I argue that RCTs are useful as an approval mechanism. But most of the "inferential" uses are deeply tenuous. I keep looking for counter evidence to my claims there, and I keep coming up empty.
Is "eating fewer calories causes weight loss" medicine or not medicine? Ooh, what about "progressive overload causes strength gains?"
Zooming out, I'm also tempted to half-assedly argue that in recent years, medicine has fiipped to having surprisingly few five sigma interventions? What are the big ones this century? GLP-1's?
In the last 5 years off the top of my head: covid mrna vaccines, glp-1 agonists, crispr gene therapy. If you look earlier, you'll see a lot of great cancer therapies (like gleevec and car-t cell treatment) and vaccines (like gardasil). At some point, I will compile a list.
I certainly think "measure theoretically" or whatever that most interventions don't work. I'm just saying that we find "one in a million" type therapies a lot more than we might think. And it's incredibly hard to find these in fields like economics, political science, psychology, etc.
Propaganda got to be at least 5 sigma on collective human behavior, but tricky to quantify. Also 5 sigma effects on antibody levels does not imply 5 sigma effects on overall mortality, it might still be insignificant on a population level, or not. Transient medical/chemical effects in an individuaI can be 5 sigma initially on some medical marker, but 0-sigma over longer time in the same individual on outcomes that matter. I still do not understand how to define human-facing, or medicine. But it is all interesting. Thank you for writing this.
That's interesting because I think the phonetic-confusion effect made a lot of intuitive sense. I'm curious which part they found confusing.
I also totally agree with your point about the "this matters practically" gap.