Great piece--I really like this thread you're on. One thing this made me think of is the role of professional ethics and codes of ethics in more subjective approaches like PT. If our standard is "only things that pass RCTs are good interventions, so just do those" then we don't really need ethics. If we're opening up a role for judgment, the ethics of the person applying that judgment become important.
The thumb rule I have followed is discard statistics + any claim of observational causal inference in these ITT scenarios but look at the purported mechanism of action and see if that mechanism can overwhelm my skepticism. The show-me-the-derivation unfortunately becomes show me a plausible explanation the more out of depth I am in that field. And these chatbots are ridiculously good in generating endless drivel of seemingly plausible explanations.
I think a lot of this boils down to a certain kind of 'sciencepilling' that takes statistics as the only good reason for believing something true is true. We saw it during the pandemic with masking- a certain breed of insufferable commentator would not that there weren't any RCTs saying masks would help. Well, the reason it's clear masks would help is that the thing that spread COVID was a little particle, and putting shit in the way of the particles would stop them. If anything the RCT was going to muddle the waters by producing a failure rate that consisted of idiots doing dumb things rather than the failure of the physics of the situation.
A math teacher I read (Michael Pershan) made a comment in one of his books differentiating between 'evidence-based' teaching and 'evidence-informed' teaching, suggesting the first is routinely both impossible and a nightmare and the second is what actually happens in a feedback-and-variable dense environment (which is pretty much anything involving human beings). He compared it to hiking with a map. The evidence of the map makes it clear that some approaches are likely to be better than others, with some ruled out categorically. But you still gotta hike, and your interests, physical capabilities, the weather and vegetation and rockfall, are going to be dictating where your feet go, and that's as it should be.
Broadly agree with your points here! I'm curious if you have thoughts on https://www.painscience.com/. I've overall found this site both interesting and helpful. It's "science-ish" but not dogmatic in my experience?
I agree with your overall message. And yes, opioids/opiates (natural or synthetic) are not universally helpful in musculoskeletal pain. My lumbar pain is an affirmation of that.
But regarding the combinatorial explosion of treatments. I thought Latin Squares were a method to reduce the number of treatment variables. Shouldn't it work to reduce treatment tests even with more than a couple of variables?
In addition, Judea Pearl's "The Book of Why" demonstrates that one can make inferences even with reduced data on variables. (which seems similar to the Latin Squares method but applied to incomplete coverage by the available data). I have no reason to doubt his group's mathematical rigor. Is he wrong?
Nice thoughts. Reminded me that the "intention to treat" bucket of folks who opt-out creates effects very similar to "response-rate bias" in polls/surveys results. This phenomenon cuts across pretty much all non-coercive interventions (as dramatically played out during Nate Silver's downfall around predicting the 2016 election for Clinton based on polls with substantial response-rate biases, in that would-be Trump voters had lower response rate, biasing the survey results in favor of Clinton, e.g. in Michigan). Maturing how we handle this response heterogeneity (e.g. in some kind of Bayesian framework that integrates over some plausible range of response-rate-to-outcome correlation), might help across the board. Further, I'm not sure why it's hard to imagine that the suite of outcomes could include measures such as, "is your life better than it was before?" and take the responses seriously, especially in N-of-1 style trials. I tend to push back against the "EBM is flawed so the answer is to reduce the emphasis on trials". Is your take that we need to puncture EBM's cult-like and and somewhat simplistic over-emphasis within administrative/bureaucratic contexts?
Great piece--I really like this thread you're on. One thing this made me think of is the role of professional ethics and codes of ethics in more subjective approaches like PT. If our standard is "only things that pass RCTs are good interventions, so just do those" then we don't really need ethics. If we're opening up a role for judgment, the ethics of the person applying that judgment become important.
The thumb rule I have followed is discard statistics + any claim of observational causal inference in these ITT scenarios but look at the purported mechanism of action and see if that mechanism can overwhelm my skepticism. The show-me-the-derivation unfortunately becomes show me a plausible explanation the more out of depth I am in that field. And these chatbots are ridiculously good in generating endless drivel of seemingly plausible explanations.
I think a lot of this boils down to a certain kind of 'sciencepilling' that takes statistics as the only good reason for believing something true is true. We saw it during the pandemic with masking- a certain breed of insufferable commentator would not that there weren't any RCTs saying masks would help. Well, the reason it's clear masks would help is that the thing that spread COVID was a little particle, and putting shit in the way of the particles would stop them. If anything the RCT was going to muddle the waters by producing a failure rate that consisted of idiots doing dumb things rather than the failure of the physics of the situation.
A math teacher I read (Michael Pershan) made a comment in one of his books differentiating between 'evidence-based' teaching and 'evidence-informed' teaching, suggesting the first is routinely both impossible and a nightmare and the second is what actually happens in a feedback-and-variable dense environment (which is pretty much anything involving human beings). He compared it to hiking with a map. The evidence of the map makes it clear that some approaches are likely to be better than others, with some ruled out categorically. But you still gotta hike, and your interests, physical capabilities, the weather and vegetation and rockfall, are going to be dictating where your feet go, and that's as it should be.
Broadly agree with your points here! I'm curious if you have thoughts on https://www.painscience.com/. I've overall found this site both interesting and helpful. It's "science-ish" but not dogmatic in my experience?
I agree with your overall message. And yes, opioids/opiates (natural or synthetic) are not universally helpful in musculoskeletal pain. My lumbar pain is an affirmation of that.
But regarding the combinatorial explosion of treatments. I thought Latin Squares were a method to reduce the number of treatment variables. Shouldn't it work to reduce treatment tests even with more than a couple of variables?
In addition, Judea Pearl's "The Book of Why" demonstrates that one can make inferences even with reduced data on variables. (which seems similar to the Latin Squares method but applied to incomplete coverage by the available data). I have no reason to doubt his group's mathematical rigor. Is he wrong?
Nice thoughts. Reminded me that the "intention to treat" bucket of folks who opt-out creates effects very similar to "response-rate bias" in polls/surveys results. This phenomenon cuts across pretty much all non-coercive interventions (as dramatically played out during Nate Silver's downfall around predicting the 2016 election for Clinton based on polls with substantial response-rate biases, in that would-be Trump voters had lower response rate, biasing the survey results in favor of Clinton, e.g. in Michigan). Maturing how we handle this response heterogeneity (e.g. in some kind of Bayesian framework that integrates over some plausible range of response-rate-to-outcome correlation), might help across the board. Further, I'm not sure why it's hard to imagine that the suite of outcomes could include measures such as, "is your life better than it was before?" and take the responses seriously, especially in N-of-1 style trials. I tend to push back against the "EBM is flawed so the answer is to reduce the emphasis on trials". Is your take that we need to puncture EBM's cult-like and and somewhat simplistic over-emphasis within administrative/bureaucratic contexts?