The Higgs Discovery Did Not Take Place
Some noncontroversial blogging during midterm break
In a post last week about statistical testing, I asked, "What are the grand discoveries that we wouldn’t have made without an understanding of null hypothesis testing?"
Nico Formanek replied, “Ok, I'll take the bait. Why wouldn't the discovery of the Higgs Boson count?”
Ah, now this is a fun example! I briefly replied to him in a comment and thought I’d write a quick follow up post last Friday. But in trying, I ended up down a deep rabbit hole trying to figure out what exactly the null hypothesis was.
It’s shocking that to properly explain the Higgs experiment, I’d have to write an entire book. The basic idea is that if you take two streams of subatomic particles, in this case Hydrogen ions, and accelerate them to near the speed of light, they cause explosions when they collide. Since E=mc^2, these explosions emit a lot of energy. If you look at the patterns in the explosions, you’ll see characteristics of exotic matter that exist for the tiniest fraction of time. If you generate trillions of these sorts of explosions, you’ll eventually see everything that could possibly exist in the energy range of your collider.
Since each event is an explosion, we can’t expect to see exactly the same explosion every time we bang the ions together. Explosions are inherently stochastic, and physicists have ornate ways of calculating the probability of seeing various proposed fireworks displays. This makes the sub-subatomic particles inherently statistical objects. The only way that we know how to make them is to build billion-dollar statistical casinos and check whether the statistical numerology of the Standard Model predicts an appropriate run of events. Physicists calculate the theoretical odds of their particle roulette wheel, and then they run a bunch of spins to make sure the odds match their expectations.
Except it’s a bit more complicated than that. In a well-functioning roulette wheel, the probability of every number is equal, and hence computing the probabilities can be done by only knowing the number of possibilities. In particle physics, the model is much more complex. I can’t explain it to you in a blog post, and neither can physicists. Particle physics is so annoyingly complex that I can’t even explain it to you unless you’re willing to finish a PhD. Physicists love to proclaim that no one understands quantum mechanics. The spontaneous symmetry breaking of gauge fields and flavor changing neutral currents of particle physics are a whole other business of weird.
The mathematics of Quantum Field Theory is a bizarre amalgamation of variational calculus, representation theory, and a bunch of wacky statistical tricks that turn infinities into zeros. Michel Talagrand tried to simplify the theory for mathematicians and ended up writing an 800-page book. Quantum chromodynamics adds extra complexity on top of this. If you are willing to toil through learning all of this ornate mathematics, you can calculate what you might expect to see when two protons are smashed together at near the speed of light.
Well, almost. It turns out that this very ornate and complex theory has a lot of unspecified parameters that are unknown in advance. Notably, the masses of all of the fancy subatomic particles, with jokey names inspired by James Joyce, have to be fit to data. The couplings between the various particles must also be estimated from data. All in all, there are 30-ish “free parameters” that have to be fit experimentally. On top of these fundamental parameters, there are apparatus specific parameters as well. These parameters concerning the vagaries of supercolliders also have to be incorporated to account for various noise processes that arise when bashing Hydrogen ions against each other at four teraelectronvolts.
Even without the elaborate mathematical theory, a probabilistic model with free parameters is challenging to validate from data. Imagine a roulette wheel where I tell you now that black and red have different probabilities. But those probabilities change depending on the day of the week. And you have to adjust the probabilities if the TGV goes by.
If only it were just about calibrating a roulette wheel! Instead, it’s a ten billion-dollar apparatus with thousands of scientific staff members. No single staff member knows how the whole apparatus works, of course. Instead, there is a shared community of trust. I recently attended a talk by Peter Gallison, who described the complex governance structures involved in experiments like the Large Hadron Collider. The CERN collaboration establishes parliamentary rules to decide upon scientific truth. Reality is validated by majority vote. Physicists love to talk a lot about how they are probing the very nature of the universe, but they do this by a lot of boring committee meetings.
One of the many CERN committees is a statistical committee. It is the one that argues about the best frequentist statistics to be used. The CERN statistics committee tells us that the p-value of seeing what they saw if the Higgs wasn't there is less than 0.0000006. But, as you might expect from anything decided by committee, it’s not that simple. Here’s what Science Magazine included in their glossary about the Higgs Discovery:
“Significance and the look-elsewhere effect: The probability for a background fluctuation to be at least as large as the observed maximum excess is termed the local P value, and the probability for an excess anywhere in a specified mass range is the global P value. This probability can be evaluated by generating sets of simulated data incorporating all correlations among analyses optimized for different Higgs boson masses. The global P value (for the specified region) is greater than the local P value, and this fact is often referred to as the look-elsewhere effect. Both the local and global P values can be expressed as a corresponding number of standard deviations using the one-sided Gaussian tail convention. For example, a 5σ significance tells us that the probability of the background alone fluctuating up locally by the amount observed or more is about 1 in 3 million. In particle physics, this criterion has become a convention to claim discovery but should not be interpreted literally.”
Should not be interpreted literally? Are you serious? “Trust us. We’re the smartest people on the planet. There’s no way we’d miscalculate 5σ events.” Where have we heard that before? Oh, right.
Despite accepting what the open science community would deride as Highly Questionable Research Practices (lots of p-hacking, HARKing, and multiple hypothesis testing), by adopting the 5σ convention, the Higgs statistics community tells us it shouldn’t matter. When they say “5σ,” they mean five standard deviations from the estimated mean background at exactly one mass location in their complex model. They check all of the possible bins and compute a background model for each bin. This results in the following figure:
See the bump that goes outside of their green error bars? That’s the Higgs Boson. (insert shrug emoji)
This brings me back to my original reply to Nico: we have an "object" that has been "observed" at a single location on Earth. This observation was done in an experiment where no single person understood the entirety of the procedure. It requires well over 6 years of graduate study to fully understand what was supposed to be seen in the most ideal experimental situation. Under these idealized conditions, which no one fully understands, the CERN folks tell us that the p-value of seeing what they saw if the Higgs wasn't there is less than 0.0000006. But this p-value is corrupted by all sorts of standard complaints about questionable research practices, and we are told not to take it literally.
OK, so what’s the point of all this? Am I just trying to be a science denier who casts uncertainty at all fundamental discoveries? I’ll confess: a little. But I also think we can learn something about modern science and engineering by digging into a few follow-up questions. What does it mean that the Higgs Boson was discovered? And why would we be in a different situation without that p-value?
The answers fit nicely in the argmin oeuvre.
First, it was certainly not the case that digging into the Higgs made me feel better about the utility of statistical testing. Whether or not the Higgs exists has no bearing outside the insular world of particle physics. If you don’t have a four teraelectronvolt supercollider, you can’t make a Higgs. The Higgs field has no bearing on any physics at any scale anyone would ever care about. So I don’t care either way if physicists think they found a Higgs. It has zero bearing on my existence.
Nico thought this view too pragmatic. But pragmatism too often gets short shrift in the history and philosophy of science. The philosophy of engineering remains underexplored! There has to be something we can do with substantive causal theories for them to be real.
Now to the second question. When we ask, “Does the Higgs Boson exist?” we are not asking about the material reality of an object. We're asking about our belief in a system. Do we believe that a collection of determined, over-credentialed scientists can organize themselves, through their democratic, participatory decision-making schemes, to decide upon the connection of data and theory? Do we believe that such institutions produce trustworthy procedures and rituals so that if they say they did something, then no one else has to check? Do we believe that their presented statistical counts represent a close enough facsimile of experimental conditions to corroborate an ornate, impossible to understand theory? Do we believe their committees properly adjudicate statistical practice and preregistration plans?
All of these questions are substantially less romantic than those popularized on YouTube or in Quanta magazine. They are questions about people, their budgets, and their committees. This is why statistics is so vital to the Higgs discovery. Just like how we need RCTs as a regulatory mechanism for drugs, big science needs 5σ as a way to set standards for pushing papers through their mass of review boards. Statistics is most useful in regulatory standards, providing compressed, crisp rules for stochastic approval. “5σ,” it turns out, is not different than “p<0.05.” It is another mindless convention needed for a well-functioning collaboration.
And well-functioning CERN was! With 10 billion dollars, they confirmed a model of particles that connects the crowning achievements of 20th-century physics. Through a massive collaborative government, they achieved a democratic consensus about the fundamental building blocks of physical theory, a united nations of physics. The Higgs Discovery is a celebration of modern bureaucracy, not a revelation about material reality.
Additional References of Statistics Philosophy Sickos
The 3000-author Science paper declaring the Higgs to be found.
An Annual Reviews paper on the statistics of the Higgs by David van Dyk
A discussion of the eccentricity of the statistical procedures by Tony O’Hagan
I hope this is written tongue-in-cheek, because otherwise, it's one of those cases where someone manages to instantly shatter their credibility. Going forward with this blog would require not so much Gell-Mann amnesia, but a Gell-Mann lobotomy.
I'm very tempted to go into full rant-mode, but it's late so I'll limit myself to the following:
1. While it was discovered at a single collider, the signal for the Higgs was found by two separate experiments (i.e. independently designed detectors) at different locations along the LHC ring. Sure, systematics can be correlated within the same location but it would be quite unlikely for the same outdoor rumble to show up in this particular signal and in no other search.
2. The LHC experiments did not stop their work on Higgs related physics after its discovery -- even now, more precise measurements of its mass and decay width are being made, see https://arxiv.org/abs/2409.13663 .
3. The Higgs field gives rise to the mass of elementary particles, and thus contributes to things like the stability of the proton, and rate of radioactive decay (and hence why the sun is able to shine continuously). The fact that the Higgs field couples to known particles in proportion to their masses may mean it couples to otherwise undetectable massive particles as well, which may be one way of ascertaining what dark matter is.
4. The properties of the Higgs boson and field as measured are to say the least, not ideal in terms of theory. In some sense, it would have been nicer if it hadn't been found at all! (It might yet be that these properties are explained by the Higgs being a composite particle itself.)
To put it another way, there is a tremendous incentive for particle physicists to similarly produce the discovery of SUSY via an LSP candidate, or indeed any new signal post-2012. Have the bureaucrats been too busy with congratulatory backslapping to cook something up?
I'm astounded by the small-mindedness of it all by someone professing to be so learned. We've been clever enough to discover things that are far removed from our every day experience. But there remoteness does not preclude them from ever being useful in our lives, though it may well take time for the seeds of discovery to bear the fruit of utility.
Impressive levels of irreverence here. Probably more useful to re-direct this energy and momentum to pick apart big pharma RCTs, where neither data nor analyses are out in the open. Particle physics receives a lot of sunlight, so to speak, and other uses of statistics, with arguably more direct (and negative) consequences, ought to be scrutinized with this level of motivation more often.