We spent the first half of this semester thinking about making predictions. But of course, at some point we need to do something with these predictions. What’s the right way to make decisions from data-driven patterns? There are so many factors that come into play. We have to model how past patterns reflect future outcomes. We have to quantify the consequences of acting upon incorrect predictions and the cost of inaction when our prediction is correct. And we have to formalize how to assemble these pieces into policy.
Let me set the stage with a sadly timely example. Suppose you have cold-like symptoms. You pull out one of your trusty covid tests you got for free in a conference gift bag. Your test is negative. Should you ask your doctor for a prescription for Paxlovid? Unless you are an addicted reader of the New York Times, it probably seems like taking an antiviral drug after testing negative for the associated infection is a bad idea. But those cheap covid tests aren’t very sensitive. Sometimes they only start testing positive well after symptoms onset. And you’ve seen those Pfizer commercials during football games arguing Paxlovid works best if you start taking it early.
Certainly, if you somehow knew for certain you had a covid infection, your decision-making would be easier. But you are in this situation where it’s plausible that you have covid and plausible that you don’t have covid. There must be some way to use knowledge of past events to make an educated guess of the course of best action.
You might appeal to some sort of inductive heuristic. You find some paper on the medRxiv that says that 75 out of 100 times, people whose cheapo covid test is negative don’t actually have covid. What does that fraction tell you about your infection status? The most common interpretation is that your own personal odds of infection are one to three. This is an inductive leap, assuming your test outcome is somehow linked to the outcomes of the 100 people in the study, and this informs your decision-making. That is, you model your own individual chances on the statistics of others in the past. I apologize, my dear readers, we are forced into thinking about probability, and I don’t like it.
There are a few ways to approach this probabilistic relationship between the measurement and the underlying state we’re interested in. For each possible measurement, we could count how frequently the state occurs. This would give us an estimate of conditional probabilities:
Pr[state | measurement] = fraction of times the state occurs when we see the measurement
In the testing case, the measurement is a test result (positive or negative), and the state is the infection (have covid or don’t have covid). Hence, in my hypothetical reference above
Pr[infection | negative test] = 25%
I emphasize again that these conditional probabilities are just capturing past proportions of individuals. To use them, we must explicitly assume that past experiences are indicative of the future. Then, when we see more individuals with the same measurements, we can hypothesize that the same proportion should have the associated state. If that state is bad, and we know that it occurs 9 times out of 10 when a particular measurement occurs, then we should be concerned when we observe that measurement. This all may feel reasonable so far, but I’m hammering on this point, as I’m always cautious when invoking stochastics. We make strong, implicit modeling assumptions when we use probabilities to make predictions.
There’s also an inverted way to set up the probabilistic modeling. We could assume that the measurement process is random. In the notation we just used, we build a mathematical model of Pr[measurement | state]. These models are easier to estimate when it comes to medical tests. People run studies comparing the outcomes of cheap tests to more expensive gold standard tests. Assuming the gold standard test is ground truth, they can then provide tables recording the number of times an individual tests negative when a more precise test confirms their infection. You could build up a whole table for the different scenarios of true positives, false positives, true negatives, and false negatives. This probabilistic model is attractive as tests hopefully don’t change their sensitivity and specificity over time. Unfortunately, we’ll see tomorrow that this model alone is insufficient to make decisions.
Let me note that it would be unacceptable to model the state as mechanistically depending on the measurement. That is, even if we could find a good fit of past data, we have no good reason to believe that
state = f(measurement) + noise
Where noise is a random variable, this is inverting the causation in the problem, and though it can build a perfect conditional probability model Pr[state | measurement], implausible models only beget implausible conclusions. You likely agree with me, but don’t forget this next time you see someone fit a logistic regression model predicting state from measurement.
Now, assuming that we are fine with our probabilistic modeling and quantification of uncertainty, I still haven’t told us what to do with this model. We need to connect the actions available to us with the potential outcomes of taking these actions under the uncertainty model. This naturally leads us to posing an optimization problem to balance costs and benefits. Tomorrow we dig into what it might mean to maximize expected returns under our probabilistic models.