Never Ending Math Equation

Myopic evidence-based medicine can't see whether physical therapy works.

Of the many cursed segments of vertical video, few annoy me more than the cottage industry of lunatics ranting over screenshots of PubMed pages to claim authority for whatever therapy, training program, or peptide they are selling. These people are all full of shit. They promise to cure your pain, get you thin, and make you stronger. All through SCIENCE.

These influencers will tell you that the scientific is better than the not scientific, and might even go so far as to say that the remainder is pseudoscience or quackery. But we unfortunately live in a narrow-minded world where scientific too often means “proven efficacious in a systematic review of randomized controlled trials.” You’ll be hard-pressed to find anything in the realm of treating musculoskeletal limitations that fits the bill.

It’s hard to shoehorn these sorts of therapies into the starting requirements for randomized trials. We’d need to start with clean definitions of an intervention and an outcome. What would these be for the management of pain by physical therapy?

Let’s start with the intervention. If we are being dogmatic evidence-based practitioners, the intervention in any physical therapy trial is the invitation to have therapy. According to the intention-to-treat principle, the invitation is the only thing that can be randomized. We can’t look only at the people who comply with every instruction and make it to the end of the rigorous therapy regimen. The model patients who diligently comply might differ from those who don’t, and our statistical signal will be biased if we include only the former. The only way to avoid these selection biases is to count everyone who was randomized, regardless of what happens between randomization and the final assessment. Both committed patients and no-shows contribute to the measured average efficacy.¹

But what does it even mean for patients to follow the protocol? Physical therapy is far more complex than taking a drug. There’s no simple unit of treatment applied. Each interaction with a physical therapist involves a conversation about how things have been going, a plan for moving forward, some sort of interactive intervention in the office, and a discussion about what to do once the patient goes home. Each step here introduces a new branch in a deep decision tree. And every PT I’ve interacted with has been different, even when performing similar range-of-motion tests or manual therapies. Moreover, the treatment of any session depends on the entire history of the treatment so far. On top of this, every physical therapist I’ve seen has assigned daily exercises to do between sessions. This is part of the treatment, too! There is no way to perfectly isolate and randomize a single component of these complex treatment protocols.

A multi-stage protocol is an exponentially large collection of interventions. I made this point on the blog a few weeks ago in the context of anticoagulant trials for heart disease: “If you want to compare the effect of three different timings and three different dosages of a single drug, you need nine arms in your trial. If you want to additionally see if a second drug is helpful, you need 18.” Physical therapy is arguably much more complex.²

What about the outcome? In drug trials, we might get grim, unambiguous, objective outcomes like mortality. In vaccine trials, we might get unambiguous outcomes, such as a PCR diagnosis. Unfortunately, in pain management, the outcome is necessarily subjective. You can measure changes in range of pain-free motion, but there is too much heterogeneity to definitively stake out what a good outcome would be. Instead, pain therapies are most commonly evaluated based on improvements on the Numeric Ranking Scale. Studies ask participants at admission how their pain is on a scale of 0 (no pain) to 10 (worst pain imaginable). They ask them again at the follow-up. Statistical protocol then dictates computing the mean of the differences in treatment and control and running a t-test. You can try to remove the heterogeneity in how people respond to these questions, but these adjustments are based on subjective clinician calls. No matter what you do, pain is hard to mathematize. Doing statistics on these “numbers” and coming away with strong conclusions is a fool’s errand.

Beyond the treatment and outcome, all sorts of investigator biases make randomized trials even messier. You can blind the patients and the clinicians who assess outcomes, but you can’t blind the people applying physical therapy. It’s impossible to say what effect this sort of bias has on the scientific record. Even when well-intentioned, a clinician who believes in PT can subtly give away the secret assignment to their patients during a session.

I’ve never read a single study in this space that’s been compelling, and I don’t know why we hope that a narrow view of therapy can help us out of it. This fuels the fire of debate with people using studies to attack each other’s practices. There are countless articles and videos castigating stretching, massage, or cupping as not backed by evidence. These are all denounced as pseudoscience by a medical establishment that prescribed OxyContin like candy for two decades. Boy do I have some bad news for people who think there is great evidence that opioids work for pain management.

If we want to understand best practices for “wellness,” we need a different language around it. Maybe this language will need to lean on biomechanical plausibility or biochemical pathways. That certainly wouldn’t hurt. But more importantly, the language will have to prioritize discussions of craft, practice, and the cultivation of expertise. Whatever the case, the narrow definition of evidence-based needs to be reimagined. Healthcare is far more than a collection of unambiguous interventions with unambiguous outcomes.

1. No one likes to talk about this, but the no-shows introduce their own tricky bias. If the patient drops out of the study, the intention-to-treat principle insists we make up some number for them and include it in our average.

2. If you wanted to test a complex protocol like this, you’d have to use something more akin to reinforcement learning. Reinforcement learning promises to find optimal protocols by running many randomized scenarios that hone in on the specific effects of specific interventions on specific conditions. But we know that the number of scenarios you need to test in classic tabular reinforcement learning can be in the millions, even when you have only a few interventions and states. It’s fine for board games, impossible for anything that touches physical reality.