The food pyramid vs the evidence pyramid

Peanut allergies, the LEAP RCT, and when experts issue guidance under pressure.

A popular claim in causal inference is that findings of randomized trials often contradict the consensus of observational studies. It’s actually quite hard to find RCTs where this is the case. The conclusions of the most notable example, the Women’s Health Initiative’s finding of harms of estrogen replacement therapy, remain hotly debated. I was recently pointed to an example that seemed unequivocal: the LEAP RCT demonstrating that peanut consumption by young children reduced peanut allergies. The actual story here was far more interesting than a simple ranking of forms of statistical evidence. The RCT was performed to halt an epidemic of panic.

Let’s first look at the results of the study. Between December 2006 and May 2009, the team enrolled 640 infants at high risk of peanut allergy. In the treatment group, the parents were instructed to feed their babies Bamba peanut puffs. The control group was instructed to have their babies avoid all peanut consumption. The outcome was whether a child had a peanut allergy at age 5.

The results did not need statistics. Only 1 of the 305 children who ate Bambas developed a peanut allergy. By contrast, 54 out of 313 in the control group developed a peanut allergy at some point.

That is a shocking result. A miracle cure. In high-risk children, consuming peanut protein resulted in a 50-fold reduction in peanut allergy. Had this been a vaccine trial, Bamba stock would have rocketed to the moon.

The trial was diligent about adhering to the staid statistical liturgy of evidence-based medicine. They used an intention-to-treat analysis, which the clerics have decided is the fairest way to evaluate medical interventions. In an intention-to-treat analysis, every subject who is randomized must be counted.

In the consumption group, seven subjects tested positive for peanut allergy at the beginning of the study and were advised to avoid peanuts. They get counted in the analysis. 2 of the babies in the treatment group had caregivers who admitted they decided to ignore the instructions and avoided peanuts. They get counted in the analysis, too. 5 subjects were lost to follow-up and had unknown status at the end of the trial. If we include this missing data and pessimistically assume those children developed allergies, the total number of peanut allergies counted in the treatment group is 15. In the control group, seven children had missing outcomes. The most pessimistic assumption for the trial would be that these kids didn’t develop allergy. Putting these extra counts together, 15 of 320 children randomly assigned to treatment may have developed an allergy, as compared to the 54 of 320 children who tested positive for peanut allergy after being randomly assigned to the control group. Even in this worst-case analysis, “randomly assigning parents to feed their kids peanuts if they can tolerate them” results in a 3.5-fold reduction in peanut allergies.

To rib my frequentist friends, you might ask what the p-value is for this overly conservative analysis. The paper reports p<0.001. That’s very modest of them. The p-value, even in this worst-case, intention-to-treat analysis, is in fact less than 0.000001. As I said, this study didn’t need statistics. For those keeping track at home, the z-score of the pessimistic intention-to-treat analysis is 5. I can’t say it enough: medicine is the only human-facing science that routinely finds 5-sigma interventions. Here’s another one to add to my list.

Reading this study disturbed me. This was a tremendous failure of preventative medicine. How on earth could we not have known of such an effective treatment? Why did we even need to do a trial in the first place? It’s not as if peanuts were a new food introduced in the 1990s. I had assumed there must have been some clinical equipoise, with substantial observational data suggesting peanut consumption was associated with allergy. But that’s not what I found at all. The actual story is far more interesting.

Peanut avoidance was a socially contagious panic rubber-stamped by experts. Starting with parent advocacy groups in the 1990s, news articles started popping up about scary stories of anaphylaxis from food allergies, even though deaths by peanut remained rarer than deaths from lightning strikes. Miranda Waggoner details a history of the rise of the allergy narrative and the sensationalist tone of the associated media coverage. In response to the advocacy, the American Academy of Pediatrics changed its dietary recommendations in 2000. They advised parents with children at “high risk” should not feed their children “peanuts, nuts, and fish until 3 years of age.” This advice ran counter to the conventional wisdom of allergists and immunologists, who believed consumption was important for priming the immune system.

After these guidelines were published, peanut allergies exploded, jumping more than threefold in the US between 1997 and 2008. It was at this point that observational studies emerged, including an influential one led by investigators from the LEAP team. These showed that peanut consumption was strongly anticorrelated with allergy. Freaked out by the sudden rise and realizing they had made a huge mistake, the AAP abruptly reversed its anti-peanut stance in 2008. This is two years before the LEAP trial finished its enrollment, three years before they observed any outcomes, and seven years before their findings were published in the New England Journal of Medicine.

The problem is that once you acquiesce to fear-mongering, no one notices your retraction. Despite overwhelming evidence, some people still don’t realize that parents have not been recommended to avoid peanuts for 20 years. It’s a fascinating study of the limits of evidence and a cautionary tale about the power of expert recommendations and political advocacy.