There is a further issue with this approach, especially in the age of generative AI. Medicine is the ultimate n-of-one science...As I tell every patient the first time I see them: "You are your own science experiment. I have seen thousands of patients before and have access to data on likely millions of other patients. But none of them are you. Together we will learn about you, informed but not controlled by what is known about all those other patients".
Population data, which is the basis of EBM, is nice to have but it is 100% clinical judgement as to whether or not it applies to the patient sitting in front of you now. As von Eye pointed out years ago (paraphrased): "If you know everything about every individual, you know everything about the population. But if you know everything about the population, you know not one single fact about any individual."
As an input to the decision-making process about a patient, EBM is important, of course. As an algorithm to be applied -- not so much.
I would still love for you to set out an argument about how we know acupuncture and chiropractic medicine work. I get that you dislike the epistemic framework of the people who claim it doesn't, but you need a positive argument for how the people who believe in it are justified.
Another reason to take Evidence Based Medicine with a grain of salt : James Phelps MD (https://psycheducation.org/author/drphelps/ ) has made the case that, for treatment of mild episodes of bipolar disorder, while there is MORE evidence supporting the "atypical antipsychotics" compared with evidence for old drugs like Lamictal (lamotrigine) or low-dose lithium, this abundance of data comes from short-term studies that track efficacy and largely ignore the long-term risks of antipsychotics. And bipolar disorder treatment is often long term. Moreover, the simple fact that there is MORE EVIDENCE is because drug companies only fund studies for their profitable new drugs. Old drugs like Lamictal and lithium can't keep up with racking up evidence.
EBM has been a great movement that discredits harmful treatments, but we have to consider limitations like the ones Phelps brings up.
The thing is, in my whole life as a guy going to the doctor, I have typically found that while doctors do read papers and care to some extent about the results of RCTs, they care more about their own experience and what they see as *my* approach and priorities towards my own health. In some sense, their attitude reminds me of mine as a teacher; research in pedagogy exists, and I've read and been influenced by some of it, but in the end I treat my teaching like something I do in my own way conditioned by my own experiences and by the indivduals in front of me, not like a set of actions rigidly determined by empirical studies. And I would be pretty resistant if somebody told me I had to do it in the latter way.
great to see a statistician’s take on EBM! I wrote a similar series on tensions between ebm and what I deem “mechanism based medicine”, you might find it interesting
The core hazard is as always that the only thing in the ballpark of crappiness with having no statistics is having a naive confusion about what your abundance of stats actually means. My major frustrations in recent encounters with docs hasn't been that they weren't minding statistical evidence; it was that they weren't actually reasoning through the inherent physical implications of the empirical truths in their hands in that A to B, Sherlockian way that we're all taught has something to do with doctoring and instead were Goodhearting themselves in real time. If *most* patients on this medication don't get condition X, but some do, but that's the reason you don't do the actual test to see if they have condition X, then lo and behold that stat is only going to get stronger even as it gets wronger. Or even more fun- doing test A suggests a lower but non-zero probability of successful detection with test B, so we will never do test B!
I really don't understand this position. You lurch back and forth between "studies are good" to "lol EBM is terrible," without actually saying why EBM is bad beyond some vague handwaving about every patient being an individual, and not being able to do a hyperparameter sweep of reality with RCT's.
And obviously we're not able to do hyperparameter sweeps? But we've carved a known treatment-and-effects space out of reality over the years, and it's good to pay attention to both the effect sizes and quality of evidence in that space. At a high level, that's EBM.
Care sucks in the US because doctors are all managed by HMO's or corporations (or private equity) and have <15 min per patient.
Under time constraints like that, the best first pass will pretty much always be something like "first line, EBM / do what the LLM says." Yes, the first pass approach doesn't work all the time, and that's when you have to do more tests, try second and third lines, and eventually do differential diagnosis to figure things out.
Sadly, that will all happen roughly 3 years down the line from when the patient first saw you, with average appointment waits. And you'll have interacted with the patient or thought about their history, labs, and case a grand total of <90 min over that 3 years. Hope their problem wasn't urgent!
The problem is much less "EBM" and much more "the incentives and infrastructure are so borked it takes 3 years to dedicated 90 minutes to figuring out a given individual patient's case."
I enjoyed your book and your blog. I’m inclined to agree with the criticisms of the maxxing epidemic and culture-wide over-optimization, but I’m having a little trouble with some of the nuances. Correct me if I’m wrong, but my reading of your stance is that optimization isn’t bad per se. However, optimization has or can have unintended consequences in a mechanical sense or a cultural sense if: 1. The target of the optimization is not properly aligned with the actual goal, 2. The second and third order effects of the optimization are not captured, or culturally if: 1. The existence of a ‘number’ to optimize causes hyperfixation and leads to practitioners ignoring the second or third order effects, 2. A misallocation of resources in the event we are doing the above (ie, having RCT’s focus on causal pathways instead of outcomes), 3. A false sense of confidence specific optimization will lead to a better life / outcomes.
The other part I’m having trouble with is the elevation of ‘narrative’ and some of the criticisms of Meehl and Tversky/Kahneman. I understand that when Meehl did his work there was a limit to ‘machine readable’ and the defined outcome must definitionally favor statistics over clinicians. However, that paradigm is different now – machine readable is just about everything. The pieces of information an algorithm has access to is nearly the same as the clinician, and it has more access, theoretically, to historical data and outcomes. So where does that leave us? And how do we evaluate the truth of narratives if we don’t use average as the basis of evaluation?
I think, as a non-practitioner, there’s a different rationale, other than medical conservatism, for the desire for evidence. Experts disagree.
When preparing for my first son’s arrival, a very experienced ob explained that she started her career with a bias for avoiding pain medication during delivery, but now recommends it because she has seen how pain can stall labor, leading a a series of dominoes falling and increasing the risk of needing to do a Caesarian. The lessons of 30 years of experience.
And spoke to a very experienced delivery nurse and doula who explained that she started her career with an openness to using pain medication during delivery but now recommends against it because receiving the meds can stall labor, leading a a series of dominoes falling and increasing the risk of needing to do a Caesarian. The lessons of 20 years of experience.
So regardless of conservatism, expert judgment is not always as useful to the consumer of the advice…
And don’t get me started on what stands behind the “expert” advice one then gets from parenting books, some of which try to convince you that your kids will be a sociopath of you follow the rival book…
I hear you about the disagreement here, but there's no way we could run a randomized trial to find out whether your doula or your obstetrician is right. Given their extensive experience, I'd argue that they are both right.
Agreed, it is a story that drives home why RCT are needed when they can help: medicine is not physics, we really don’t know much of what’s going on in the black box.
My point is that fear of iatrogenic damage is one reason to mistrust experts and want EBM, but there are other, very down-to-earth reasons for wanting evidence where it can be had.
<Given their extensive experience, I'd argue that they are both right.>
This seems doubtful to me. "Right" how? All I know is that my wife much preferred the pain relief from an epidural rather than the pain of a long labor. Anothe wife told me of the pain of labor during the period in teh 1980s when pain relief was not given. Anecdotally, pain relief is preferred.
Imagine if these 2 experts were arguing over pain relief in other contexts - dentistry, surgery, etc. It would be laughable to be told that the procedure would be better without pain relief, the pain for the patient not a consideration, just the outcome.
There is a further issue with this approach, especially in the age of generative AI. Medicine is the ultimate n-of-one science...As I tell every patient the first time I see them: "You are your own science experiment. I have seen thousands of patients before and have access to data on likely millions of other patients. But none of them are you. Together we will learn about you, informed but not controlled by what is known about all those other patients".
Population data, which is the basis of EBM, is nice to have but it is 100% clinical judgement as to whether or not it applies to the patient sitting in front of you now. As von Eye pointed out years ago (paraphrased): "If you know everything about every individual, you know everything about the population. But if you know everything about the population, you know not one single fact about any individual."
As an input to the decision-making process about a patient, EBM is important, of course. As an algorithm to be applied -- not so much.
Re your second to last paragraph, this book basically runs with that intuition to the extreme: https://academic.oup.com/book/6511
I would still love for you to set out an argument about how we know acupuncture and chiropractic medicine work. I get that you dislike the epistemic framework of the people who claim it doesn't, but you need a positive argument for how the people who believe in it are justified.
Coming sometime this summer, I promise! I also want to talk about dentistry and physical therapy.
Another reason to take Evidence Based Medicine with a grain of salt : James Phelps MD (https://psycheducation.org/author/drphelps/ ) has made the case that, for treatment of mild episodes of bipolar disorder, while there is MORE evidence supporting the "atypical antipsychotics" compared with evidence for old drugs like Lamictal (lamotrigine) or low-dose lithium, this abundance of data comes from short-term studies that track efficacy and largely ignore the long-term risks of antipsychotics. And bipolar disorder treatment is often long term. Moreover, the simple fact that there is MORE EVIDENCE is because drug companies only fund studies for their profitable new drugs. Old drugs like Lamictal and lithium can't keep up with racking up evidence.
EBM has been a great movement that discredits harmful treatments, but we have to consider limitations like the ones Phelps brings up.
The thing is, in my whole life as a guy going to the doctor, I have typically found that while doctors do read papers and care to some extent about the results of RCTs, they care more about their own experience and what they see as *my* approach and priorities towards my own health. In some sense, their attitude reminds me of mine as a teacher; research in pedagogy exists, and I've read and been influenced by some of it, but in the end I treat my teaching like something I do in my own way conditioned by my own experiences and by the indivduals in front of me, not like a set of actions rigidly determined by empirical studies. And I would be pretty resistant if somebody told me I had to do it in the latter way.
Are you, by any chance, the Jordan Ellenberg of How Not to Be Wrong?
Yup.
Pretty cool to find you here, what's your teaching philosophy? And which mathematician have influenced your perspective on math education?
great to see a statistician’s take on EBM! I wrote a similar series on tensions between ebm and what I deem “mechanism based medicine”, you might find it interesting
I'd love to read. What's the right place to start?
Love this post!
The core hazard is as always that the only thing in the ballpark of crappiness with having no statistics is having a naive confusion about what your abundance of stats actually means. My major frustrations in recent encounters with docs hasn't been that they weren't minding statistical evidence; it was that they weren't actually reasoning through the inherent physical implications of the empirical truths in their hands in that A to B, Sherlockian way that we're all taught has something to do with doctoring and instead were Goodhearting themselves in real time. If *most* patients on this medication don't get condition X, but some do, but that's the reason you don't do the actual test to see if they have condition X, then lo and behold that stat is only going to get stronger even as it gets wronger. Or even more fun- doing test A suggests a lower but non-zero probability of successful detection with test B, so we will never do test B!
I really don't understand this position. You lurch back and forth between "studies are good" to "lol EBM is terrible," without actually saying why EBM is bad beyond some vague handwaving about every patient being an individual, and not being able to do a hyperparameter sweep of reality with RCT's.
And obviously we're not able to do hyperparameter sweeps? But we've carved a known treatment-and-effects space out of reality over the years, and it's good to pay attention to both the effect sizes and quality of evidence in that space. At a high level, that's EBM.
Care sucks in the US because doctors are all managed by HMO's or corporations (or private equity) and have <15 min per patient.
Under time constraints like that, the best first pass will pretty much always be something like "first line, EBM / do what the LLM says." Yes, the first pass approach doesn't work all the time, and that's when you have to do more tests, try second and third lines, and eventually do differential diagnosis to figure things out.
Sadly, that will all happen roughly 3 years down the line from when the patient first saw you, with average appointment waits. And you'll have interacted with the patient or thought about their history, labs, and case a grand total of <90 min over that 3 years. Hope their problem wasn't urgent!
The problem is much less "EBM" and much more "the incentives and infrastructure are so borked it takes 3 years to dedicated 90 minutes to figuring out a given individual patient's case."
Performative bafflement indeed!
I enjoyed your book and your blog. I’m inclined to agree with the criticisms of the maxxing epidemic and culture-wide over-optimization, but I’m having a little trouble with some of the nuances. Correct me if I’m wrong, but my reading of your stance is that optimization isn’t bad per se. However, optimization has or can have unintended consequences in a mechanical sense or a cultural sense if: 1. The target of the optimization is not properly aligned with the actual goal, 2. The second and third order effects of the optimization are not captured, or culturally if: 1. The existence of a ‘number’ to optimize causes hyperfixation and leads to practitioners ignoring the second or third order effects, 2. A misallocation of resources in the event we are doing the above (ie, having RCT’s focus on causal pathways instead of outcomes), 3. A false sense of confidence specific optimization will lead to a better life / outcomes.
The other part I’m having trouble with is the elevation of ‘narrative’ and some of the criticisms of Meehl and Tversky/Kahneman. I understand that when Meehl did his work there was a limit to ‘machine readable’ and the defined outcome must definitionally favor statistics over clinicians. However, that paradigm is different now – machine readable is just about everything. The pieces of information an algorithm has access to is nearly the same as the clinician, and it has more access, theoretically, to historical data and outcomes. So where does that leave us? And how do we evaluate the truth of narratives if we don’t use average as the basis of evaluation?
Thanks so much
I think, as a non-practitioner, there’s a different rationale, other than medical conservatism, for the desire for evidence. Experts disagree.
When preparing for my first son’s arrival, a very experienced ob explained that she started her career with a bias for avoiding pain medication during delivery, but now recommends it because she has seen how pain can stall labor, leading a a series of dominoes falling and increasing the risk of needing to do a Caesarian. The lessons of 30 years of experience.
And spoke to a very experienced delivery nurse and doula who explained that she started her career with an openness to using pain medication during delivery but now recommends against it because receiving the meds can stall labor, leading a a series of dominoes falling and increasing the risk of needing to do a Caesarian. The lessons of 20 years of experience.
So regardless of conservatism, expert judgment is not always as useful to the consumer of the advice…
And don’t get me started on what stands behind the “expert” advice one then gets from parenting books, some of which try to convince you that your kids will be a sociopath of you follow the rival book…
I hear you about the disagreement here, but there's no way we could run a randomized trial to find out whether your doula or your obstetrician is right. Given their extensive experience, I'd argue that they are both right.
What does this mean?
Agreed, it is a story that drives home why RCT are needed when they can help: medicine is not physics, we really don’t know much of what’s going on in the black box.
My point is that fear of iatrogenic damage is one reason to mistrust experts and want EBM, but there are other, very down-to-earth reasons for wanting evidence where it can be had.
<Given their extensive experience, I'd argue that they are both right.>
This seems doubtful to me. "Right" how? All I know is that my wife much preferred the pain relief from an epidural rather than the pain of a long labor. Anothe wife told me of the pain of labor during the period in teh 1980s when pain relief was not given. Anecdotally, pain relief is preferred.
Imagine if these 2 experts were arguing over pain relief in other contexts - dentistry, surgery, etc. It would be laughable to be told that the procedure would be better without pain relief, the pain for the patient not a consideration, just the outcome.