Overfitting

The Generalization article ended on a specific, alarming gap: a model scoring 99.9% on training data but only 65% on test data. That gap has a name, and it’s one of the most important failure modes in all of Machine Learning: overfitting.

The simple definition

Overfitting happens when a model learns its training data too closely — including its noise, quirks, and coincidences — rather than learning the genuine underlying pattern. An overfit model looks fantastic on the exact data it trained on, because it has essentially memorized it, but performs noticeably worse on new data, because what it memorized doesn’t generalize the way a real pattern would.

This word has come up repeatedly throughout this glossary already — in the Dataset, Training, Epoch, and Parameters articles, among others — always as a warning, never fully explained. This article is that full explanation.

Why this happens: too much capacity, too little real signal

Recall from the Parameters article’s discussion of “how many parameters is enough”: a model with more capacity than its task genuinely requires has room to represent something far more specific than the real underlying pattern — including the accidental, meaningless quirks of its exact training examples. Given enough capacity and enough repeated exposure (too many epochs, as covered in the Epoch article), a model can essentially find a way to get every single training example “right” — not by learning what actually makes a house expensive or an email spam, but by, in effect, memorizing the answer key for that specific set of examples.

flowchart LR
    A[Model with excess capacity] --> B[Too many training epochs]
    B --> C[Model fits training data almost perfectly]
    C --> D[Including noise and coincidences unique to that data]
    D --> E[Performs poorly on new, unseen data]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a student memorizing the exact answers to last year’s exam questions, word for word, instead of learning the underlying subject. On a repeat of that exact exam, they’d score perfectly. On this year’s exam — covering the same subject, but with different specific questions — they’d likely do poorly, because they never actually learned the material, just the specific answers.

Where this breaks down: A student consciously chooses to memorize rather than understand, often as a shortcut. A model doesn’t choose this — overfitting is an emergent, mechanical consequence of training too long, with too much capacity, on too little or too repetitive data, as described throughout the Training Mechanics phase, not a deliberate strategy the model settles on.

What overfitting actually looks like while it’s happening

This is genuinely useful to recognize in practice, since it has a specific, visible signature. Recall the training/validation loss comparison first introduced in the Epoch article: during training, both training loss and validation loss typically fall together at first. At some point, training loss keeps falling — the model keeps getting “better” at its training data — while validation loss stops improving and starts rising instead. That divergence, visible as two lines splitting apart on a loss chart, is the visible signature of overfitting actively happening, and it’s exactly the signal that triggers early stopping.

flowchart LR
    A[Epoch 1-10: training and validation loss both fall] --> B[Epoch 10+: training loss keeps falling]
    B --> C[Validation loss stops falling, then rises]
    C --> D[Overfitting has begun]

A concrete example, layered

For a simple beginner example: a model predicting house prices trained on just 20 houses, using a highly flexible algorithm, might learn to perfectly predict the exact price of every one of those 20 houses — including quirks like “house #14 sold for an unusually low price because the seller needed to move quickly,” a fact with zero predictive value for any other house, but one the overfit model has effectively baked into its predictions anyway. For a production example: the well-documented wolves-vs-huskies image classifier from the Pattern article — the model that learned to detect snowy backgrounds rather than actual wolf features — is a real, well-known illustration of overfitting to an accidental correlation present in a specific training set, rather than learning the genuine visual pattern that actually distinguishes the two animals.

Follow overfitting epoch by epoch

Imagine a model learning to classify 1,000 photos as cat or dog. Lower loss is better.

Epoch	Training loss	Validation loss	What is happening?
1	0.80	0.84	The model has only started learning.
5	0.42	0.46	Both losses improved.
10	0.23	0.29	Best validation result so far.
20	0.10	0.38	Training improves, but unseen-data performance worsens.
30	0.04	0.55	The model is strongly overfitting.

At epoch 30, the training loss of 0.04 looks impressive by itself. The validation loss reveals the real problem. The most useful checkpoint may be the one saved near epoch 10, not the final epoch.

Training accuracy   = 99%
Validation accuracy = 72%
Gap                 = 27 percentage points

The gap is evidence to investigate. It is not a universal rule that a particular gap always means overfitting; acceptable gaps depend on the task, metric, data size, and cost of mistakes.

How engineers reduce overfitting

Collect more varied, representative training data.
Use Data Augmentation when realistic new variations can be created safely.
Reduce unnecessary model complexity.
Apply Regularization such as weight decay or Dropout.
Stop training when validation performance stops improving.
Check results across multiple splits with Cross-Validation when the dataset and compute budget make that practical.

How overfitting connects to everything else in this glossary

It’s worth being explicit about how many earlier articles were quietly built around defending against exactly this problem. Test Data exists specifically to catch overfitting honestly, since training performance alone can’t reveal it. Validation Data and early stopping exist to catch it during training, before it gets worse. Cross-Validation, covered later in this phase, gives an even more robust way to detect it. And the next several articles — Regularization and Dropout — cover specific techniques built to actively prevent it from happening in the first place, rather than just detecting it after the fact.

Common misconception

A frequent beginner assumption: that overfitting means a model is somehow “too smart” or “too powerful,” and that the fix is always to use a simpler model. Model capacity is only one contributing factor — the amount and diversity of training data matters just as much, echoing the Training Data article’s emphasis on data quality and breadth. A large, powerful model trained on a large, genuinely diverse dataset can generalize excellently; the same large model trained on a small, narrow dataset is at serious risk of overfitting. It’s the mismatch between capacity and available real signal that causes overfitting, not raw model size on its own.

Where this fits in what comes next

Overfitting is one of the two core failure modes this phase names precisely. The next article, Underfitting, covers the opposite failure — a model that never learns the pattern well enough in the first place, even on its own training data — completing the full picture of what can go wrong on either side of good generalization.

In one sentence

Overfitting is what happens when a model memorizes its training data — including its noise and coincidences — instead of learning the real pattern underneath it, and recognizing its telltale signature (a widening gap between training and validation performance) is one of the most practically important skills in all of Machine Learning.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026