To your point about potential flaws in randomized studies, the ultimate takeaway for hormone replacement therapy is not so easily inferred from the results of WHI's randomized control trial. A recent NYT piece argues against the now common wisdom that the costs of treatment outweigh the benefits -- both because the scientifically measured "benefits" are narrow in scope compared with women's experiences, and because the focus on /post/ menopausal women misses a key demographic:
> The study itself was designed with what would come to be seen as a major flaw. W.H.I. researchers wanted to be able to measure health outcomes — how many women ended up having strokes, heart attacks or cancer — but those ailments may not show up until women are in their 70s or 80s. The study was scheduled to run for only 8½ years. So they weighted the participants toward women who were already 60 or older. That choice meant that women in their 50s, who tended to be healthier and have more menopausal symptoms, were underrepresented in the study.
Right, don't disagree. But the observational trials suggested uniform benefit. That was definitely wrong.
The benefits of HRT are incredibly heterogenous, and their impacts are complex. this is why you have to turn to case studies, n-of-1 trials, personalized medicine, and *actually listening to patients.*
"Using individual fixed effects and cross-lagged panel models, the results provide strong evidence of trust in state institutions exercising a causal impact on social trust, whereas the evidence for a reverse relationship is limited."
If social trust clearly fluctuates temporally, with a lag, as a function of a trust in state institutions, and a change in one is followed by a change in the other, is that a case for causality? Maybe.
But even if one follows the other, in the sense that one "causes" the other, why do we even care? We can't intervene on trust in state institutions. Some, I guess, will say we can, we just have to intervene on something else that then increases trust in state institutions (e.g., built better institutions).
It's a weird world in which we want to increase something else (quality of real institutions) that then increases trust in state institutions BECAUSE (our end) is that we want to increase social trust. That's just implausible, a theoretical house of cards. Nobody believes that. But otherwise, why do we care about the “evidence of trust in state institutions exercising a causal impact on social trust”?
Alternatively, perhaps the researchers just want to “build knowledge for knowledge’s sake”. Perhaps, more plausibly, they just have to publish, that’s their job. Yes, social science is a mess…
People often accuse critics of observational statistics as not providing a path forward. My suggestion in this blog is a turn to an engineering mindset, which I do believe. But as you point out, sometimes this leads to its own mess of cards. So let me just add that I believe that these criticisms of statistics, by diminishing the authority of bad social science, themselves are providing a path forward.
I would be curious to hear the “engineer” mindset unpacked a bit. I ask because from where I stand I consider engineers to be the end member observational scientists - maybe that is an unfair view.
I'm not sure a firm boundary can be drawn between science and engineering, but I do think that there's something uniquely important about engineering in knowledge discovery. I promise to address this in a future blog.
This reminds me of "design research," where a researcher designs some object, tests its use in the world, and uses that to improve both relevant scientific theories and future designs. My understanding is that this comes from engineering-associated fields, but I'm actually most familiar with it in the context of education—where the researcher designs e.g. a curriculum, works with teachers to deploy it in the classroom and study how this goes with a variety of methods (including ethnography), and uses insights from this process to refine scientific understandings of how students learn and improve the curriculum. I agree that this is a really powerful combination!
This was a fun read until the second-to-last paragraph. It could've been replaced with: "In public health, it turns out that civil engineering for clean water and sanitation was always the best intervention. And it is observational.".
Interesting take! I am sympathetic to the argument that "engineering" is more important than we give it credit for. However maybe some of this is more of an "effort" issue, or misdirected effort into e.g. robustness checks (as you say) and fancy statistical or machine learning models rather than sound study design.
Actually, the paper you link from Freedman contains what I think is a great example of an observational causal study: John Snow's work on cholera -- elaborated on in more detail (and explicitly linked to appropriate "effort") in Freedman's "Statistical Models and Shoe Leather".
Perhaps it is more accurate to say that it's very easy to fool yourself into thinking your observational study has causal conclusions, and perhaps the ease of this self-deception is proportional to the amount of fancy statistics used, rather than "all observational [causal] studies are wrong".
> I have a particular soft spot for David Freedman, a Berkeley statistician who passed away a few years before I arrived on campus. Freedman was making the same arguments I’m making in the 1990s, and his criticism from then has never been addressed. Instead, it has been ignored. We keep deriving more methods and publishing more observational studies
So true. I'm a long time fan of Freedman. I realized in 2016 that most of human neuroscience needed to synthesize his advice. Really hard to do it though. Now that we have bigger/fancier nonparametric methods, the problem has just gotten a bit more obscure.
Preach!
To your point about potential flaws in randomized studies, the ultimate takeaway for hormone replacement therapy is not so easily inferred from the results of WHI's randomized control trial. A recent NYT piece argues against the now common wisdom that the costs of treatment outweigh the benefits -- both because the scientifically measured "benefits" are narrow in scope compared with women's experiences, and because the focus on /post/ menopausal women misses a key demographic:
> The study itself was designed with what would come to be seen as a major flaw. W.H.I. researchers wanted to be able to measure health outcomes — how many women ended up having strokes, heart attacks or cancer — but those ailments may not show up until women are in their 70s or 80s. The study was scheduled to run for only 8½ years. So they weighted the participants toward women who were already 60 or older. That choice meant that women in their 50s, who tended to be healthier and have more menopausal symptoms, were underrepresented in the study.
https://www.nytimes.com/2023/02/01/magazine/menopause-hot-flashes-hormone-therapy.html
Right, don't disagree. But the observational trials suggested uniform benefit. That was definitely wrong.
The benefits of HRT are incredibly heterogenous, and their impacts are complex. this is why you have to turn to case studies, n-of-1 trials, personalized medicine, and *actually listening to patients.*
Great text. I mostly agree. When I hesitate to fully agree, it's that I still wonder if there might be value in something like: https://link.springer.com/article/10.1007/s11109-015-9322-8
"Using individual fixed effects and cross-lagged panel models, the results provide strong evidence of trust in state institutions exercising a causal impact on social trust, whereas the evidence for a reverse relationship is limited."
If social trust clearly fluctuates temporally, with a lag, as a function of a trust in state institutions, and a change in one is followed by a change in the other, is that a case for causality? Maybe.
But even if one follows the other, in the sense that one "causes" the other, why do we even care? We can't intervene on trust in state institutions. Some, I guess, will say we can, we just have to intervene on something else that then increases trust in state institutions (e.g., built better institutions).
It's a weird world in which we want to increase something else (quality of real institutions) that then increases trust in state institutions BECAUSE (our end) is that we want to increase social trust. That's just implausible, a theoretical house of cards. Nobody believes that. But otherwise, why do we care about the “evidence of trust in state institutions exercising a causal impact on social trust”?
Alternatively, perhaps the researchers just want to “build knowledge for knowledge’s sake”. Perhaps, more plausibly, they just have to publish, that’s their job. Yes, social science is a mess…
People often accuse critics of observational statistics as not providing a path forward. My suggestion in this blog is a turn to an engineering mindset, which I do believe. But as you point out, sometimes this leads to its own mess of cards. So let me just add that I believe that these criticisms of statistics, by diminishing the authority of bad social science, themselves are providing a path forward.
I would be curious to hear the “engineer” mindset unpacked a bit. I ask because from where I stand I consider engineers to be the end member observational scientists - maybe that is an unfair view.
I'm not sure a firm boundary can be drawn between science and engineering, but I do think that there's something uniquely important about engineering in knowledge discovery. I promise to address this in a future blog.
This reminds me of "design research," where a researcher designs some object, tests its use in the world, and uses that to improve both relevant scientific theories and future designs. My understanding is that this comes from engineering-associated fields, but I'm actually most familiar with it in the context of education—where the researcher designs e.g. a curriculum, works with teachers to deploy it in the classroom and study how this goes with a variety of methods (including ethnography), and uses insights from this process to refine scientific understandings of how students learn and improve the curriculum. I agree that this is a really powerful combination!
I recommend looking into Latour, who brings an interesting critique of fact making in scientific literature in Science in Action.
Yes, old school Latour is great. I really should write up a syllabus for a modern course on science studies and statistics. Perhaps next semester...
This was a fun read until the second-to-last paragraph. It could've been replaced with: "In public health, it turns out that civil engineering for clean water and sanitation was always the best intervention. And it is observational.".
Interesting take! I am sympathetic to the argument that "engineering" is more important than we give it credit for. However maybe some of this is more of an "effort" issue, or misdirected effort into e.g. robustness checks (as you say) and fancy statistical or machine learning models rather than sound study design.
Actually, the paper you link from Freedman contains what I think is a great example of an observational causal study: John Snow's work on cholera -- elaborated on in more detail (and explicitly linked to appropriate "effort") in Freedman's "Statistical Models and Shoe Leather".
Perhaps it is more accurate to say that it's very easy to fool yourself into thinking your observational study has causal conclusions, and perhaps the ease of this self-deception is proportional to the amount of fancy statistics used, rather than "all observational [causal] studies are wrong".
> I have a particular soft spot for David Freedman, a Berkeley statistician who passed away a few years before I arrived on campus. Freedman was making the same arguments I’m making in the 1990s, and his criticism from then has never been addressed. Instead, it has been ignored. We keep deriving more methods and publishing more observational studies
So true. I'm a long time fan of Freedman. I realized in 2016 that most of human neuroscience needed to synthesize his advice. Really hard to do it though. Now that we have bigger/fancier nonparametric methods, the problem has just gotten a bit more obscure.