Learning from dynamics

Randomized measurements beyond the RCT

The N-of-1 trial design swapped a population of “individuals” for a population of time steps. But it was still asking the same question as a standard RCT: “Does taking this drug today work better than doing nothing?” Yet there is so much more information we can glean from this experiment.

In N-of-1 designs, one of the key assumptions is the effect has a short lifetime. Trialists will explicitly rule out treatments that are too long-lasting. In N-of-1 experiments, we have to assume something about the drug’s dynamics in order to probe whether it does anything. Implicitly we are assuming something about the drug’s effects over time. Maybe the response looks something like this? (I just grabbed a random figure from my Zotero library…)

Whether we are implicitly or explicitly modeling, we can learn a lot more about the effects of a drug than a single point on a dose-response curve.

Let me reframe the N-of-1 experiment in engineering language. My body responds to a drug over time. I am interested in understanding what this response looks like. I can take a pill and record how I feel every hour rather than every day. I can make a plot and see a complex response, not a simple binary.

This belabored description highlights how experiment design is the same as what we call “system identification” in control theory or “time series estimation” in statistics. We want to model how a system responds to its input.

Sharply contrasting against science, in engineering, our null hypothesis is there is an effect. We wouldn’t be messing with this fancy new piece of gear if we assumed it didn’t do anything. Of course we expect these systems to do something. We should be surprised when they don’t do anything.

If we’re assuming there’s some effect, then what exactly are we interested in? We have to figure out what some particular device can do in the context of the problem we’re trying to solve.

As a concrete example, let’s say I have speakers in a recording studio. Because of reflections off walls, the room might amplify certain frequencies. I want to figure out how to place my speakers or add EQ to a mix so that I hear a neutral mix without the room artifacts.

More generally, we need to identify enough about a system to understand what it will do when we put it out in the world in feedback with all sorts of other processes. In system identification, one of the most popular methods is exactly the same as the N-of-1 trial. The input to the system is random white noise: it is either equal to 1 or 0 at every step with equal probability. For linear systems, we can extract a full model of the dynamics from such inputs. Even for nonlinear systems, random inputs are the simplest way to capture the possible space of inputs and give statistically efficient estimates of the parameters.

Back to my audio engineering example. One way to estimate how a room amplifies or diminishes frequencies is to measure the response from a loud, singular burst. But if you want to save your ears, you can feed white noise out of the speakers instead. This trades off signal time against signal volume. The room will color the response of the white noise, and this coloring lets you identify the frequency response. You can then adjust the position of the speakers or set the EQ on your output mix to compensate for the room’s sonic idiosyncrasies.

We measure the response of a room to understand how to have concerts or record albums. We measure the parameters of a vehicle to more masterly deploy sophisticated automated maneuvers. We identify properties of a building to tune a heating system to be more energy efficient. Every experiment is motivated to understand what the system will do when we use it. Even in the N-of-1 trial I described yesterday, we could investigate dosages and dose-timing rather than just the simple binary of “Does this make your pain less severe?” These sorts of use questions are already available if we let ourselves be a little more thoughtful about the purpose of our experiments.