Thou Shalt Not Overfit

Venting my spleen about the persistent inanity about overfitting.

Most of the feedback I received about the last post concerned a footnote I forgot to format properly before posting. Let me post it here as a properly formatted pull quote.

“overfitting” in the way it is colloquially used in data science and machine learning doesn’t exist at all.

In the comments, Tucker Hermans attributed the following to Charles Isbell: "When the training error continues to decrease, but the testing error increases.” But Tucker was not happy with this definition or any of the definitions out there. Nico Formanek took it a step further, writing, “I once searched troves of stats, ML and phil sci literature for a general definition of overfitting. There is none.”

Indeed. There is none.

Time to bite the bullet and go in on this one again. I’ve written about overfitting extensively, both on this blog and in papers. A central line of my research for the last ten years has been motivated by the observation that overfitting doesn’t exist. We need to abandon the term. Let me try to motivate a different view through the lens of evaluation.

Let me first establish a sense of the colloquial definition of overfitting. I poked around and found some definitions from our friends in the tech industry.

• According to Google: “Overfitting means creating a model that matches (memorizes) the training set so closely that the model fails to make correct predictions on new data."

• According to IBM: "When an algorithm fits too closely or even exactly to its training data, resulting in a model that can’t make accurate predictions or conclusions from any data other than the training data."

• According to Amazon: “Overfitting is an undesirable machine learning behavior that occurs when the machine learning model gives accurate predictions for training data but not for new data.”

• According to Claude Sonnet 3.5: “Overfitting occurs when a machine learning model learns the training data too precisely, including its noise and random fluctuations, rather than learning the true underlying patterns. Think of it like memorizing exact answers to test questions instead of understanding the concepts.”

• According to the Oxford English Dictionary: "Overfitting is the production of an analysis which corresponds too closely or exactly to a particular set of data, and may therefore fail to fit additional data or predict future observations reliably."

OK, all of these definitions are saying the same thing.

1. An analysis works well on one data set.

2. The analysis works poorly on another data set.

The error on training data is small. The error on some other data is not. There is no mechanism or causal effect in the definition. It’s just that the evaluation didn’t work out the way you wanted. Overfitting is post-hoc rationalization of a prediction being wrong.

Calling this “overfitting” is decidedly unhelpful because predictions can go poorly in many ways. Perhaps you forgot to include a salient feature. Perhaps there was a feedback mechanism where the predictions impacted the outcome (e.g., The Lukas Critique). Perhaps we saw a black swan. There are a multitude of ways our reference narrative about the present can fail to capture future reality. When the future turns out not to be like the past, machine learning can’t work!

Though the definitions are all descriptive, we use overfitting to imply the analyst did something wrong. Someone needs to get scapegoated. The data scientists might as well be first. Our conventional sense of overfitting is that analysis keyed in too hard on one data set and thus missed the important concepts needed for prediction. In particular, we always blame overfitting on methodology, not data. In our books, we seldom blame the data for being a poor representation of the future. It’s your sinful practices that overfit.

Popular books still argue that you can prevent overfitting by following careful practices. Just like the metascientific statistical clerics come down from their ivory tower and tell people that as long as they preregister their analyses, they’ll find scientific truths, the same folks come down and tell us that as long as we follow some rules, we can’t overfit.

1. Thou shalt fear the curse of dimensionality.

2. Thou shalt not evaluate your test error more than once.

3. Thou shalt not interpolate the training data.

4. Always remember to regularize.

5. Honor the tradeoff between bias and variance.

6. Thou shalt not use nonconvex optimization.

7. Thou shalt not build large models.

8. Thou shalt not memorize.

9. Thou shalt not have larger training error than testing error.

10. Thou shalt not covet your neighbor’s GPUs.

THIS IS ALL TERRIBLE ADVICE. If you believed these religiously, you would have been crushed by deep learning (2014 me says hello).¹ They are all myths and are invalidated empirically, both in the lab and in the real world. This canonical view of overfitting is ACTIVELY HARMFUL. Statistical learning liturgy overfits to the large deviation inequalities of mathematical statistics. You can certainly construct simple cases where the bad practices of conventional wisdom do overfit. Lookup tables are terrible at prediction. High-degree polynomials interpolate one-dimensional noisy data. Textbooks and lecture notes and Medium posts are full of these contrived statistical models. They are not representative of any of the applications of machine learning today.

I’m curious if anyone disagrees with any of this. There is ample evidence that none of the machine learning commandments were true in the 1960s. But by 2025, I don’t think anything I wrote here is controversial. While Altmannian hyperscaling is an absurd, cynical marketing pitch, large models trained on all possible expected instances of reality perform well on lots of tasks. But it’s funny that classes still teach the ten commandments of machine learning, even though evidence has shown them to be invalid from the beginning.

So what do we do to move forward? In the next post, I’ll describe some overfitting case studies and how overfitting is always an issue of evaluation: Our articulated expectations of machine learning models failed to capture their actual performance. Perhaps that will lead us to a new term so we can leave “overfitting” in the bin of archaic words in the OED.

1. Bizarrely, even the deep learning people would quote these commandments to me as if they were true. The cognitive dissonance in the early 2010s was palpable.