The Underfitting article closed with a promise: a family of techniques designed to help a model land in the productive middle ground between overfitting and underfitting. The main such family has a name: regularization.
The simple definition
Regularization is any technique that deliberately discourages a model from fitting its training data too perfectly, in order to improve how well it generalizes to new data. It sounds almost paradoxical at first — making a model slightly worse at its training data, on purpose, to make it genuinely better overall — but that trade is exactly the point, and it’s one of the most important, widely used ideas in practical Machine Learning.
Why deliberately holding a model back actually helps
Recall from the Overfitting article that overfitting happens when a model has more capacity than its real signal requires, and uses that excess capacity to memorize noise and coincidences rather than genuine pattern. Regularization works by adding a cost to that excess flexibility, nudging the model toward simpler, more conservative solutions that are less likely to be chasing noise. A model that’s gently discouraged from fitting every last training-data quirk perfectly ends up more likely to land on the genuine underlying pattern instead — the whole strategy is to trade away some training-data perfection in exchange for real-world reliability.
flowchart LR
A[Loss Function measures prediction error] --> C[Total training objective]
B[Regularization penalty discourages excess complexity] --> C
C --> D[Training balances both: accurate AND simple]The most common technique: weight decay (L2 regularization)
The most widely used form of regularization, often called weight decay or L2 regularization, works by adding an extra term to the Loss Function covered in the Training Mechanics phase — a penalty based on how large the model’s weights are overall. Recall that training’s whole job is minimizing loss; if large weights now also increase that loss slightly, training will naturally prefer smaller, more modest weight values whenever they achieve similar accuracy, rather than letting any single weight grow disproportionately large to perfectly capture one specific training quirk. Smaller, more evenly distributed weights tend to produce smoother, less erratic predictions — exactly the kind of behavior associated with a genuinely learned pattern rather than memorized noise.
ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a company setting a strict budget cap for a project, on purpose, even though an unlimited budget could theoretically produce a more elaborate, seemingly “better” result. The cap forces the team to focus on what actually matters most, cutting unnecessary extras, and often produces a more robust, maintainable outcome than an unconstrained effort would have.
Where this breaks down: A project team consciously decides what to cut under a budget constraint, applying judgment about what’s essential versus excessive. Weight decay applies its constraint uniformly and mechanically across every weight, via the loss function penalty, with zero judgment about which specific weights are “essential” versus “excessive” — it simply makes large weights costlier across the board, and lets the ordinary training process, described throughout the Training Mechanics phase, work out the rest.
Types of regularization worth knowing
Weight decay (L2) isn’t the only approach, and it’s worth knowing the landscape briefly:
- L1 regularization works similarly to weight decay but tends to push some weights all the way to exactly zero rather than just shrinking them — effectively removing certain features from the model’s consideration entirely, which can be useful when you suspect many features are genuinely irrelevant.
- Early stopping, first introduced properly in the Epoch article, is itself a form of regularization in a broader sense — it doesn’t touch the loss function directly, but achieves a similar goal (preventing the model from over-fitting to training data) by simply not letting training run long enough to memorize noise.
- Dropout, covered in its own dedicated article next, takes a very different, structural approach — randomly disabling parts of a network during training, rather than penalizing weight size directly.
Weight decay with a small numerical example
Imagine two house-price models make equally good predictions on the training data:
Model A weights: [10, 0.1]
Model B weights: [ 3, 2.0]
Prediction loss for both models: 4.0Suppose we add an L2 penalty using this simplified rule:
total loss = prediction loss + λ × sum of squared weights
λ = 0.01For Model A, the total loss is 4.0 + 0.01 × (10² + 0.1²) ≈ 5.00. For Model B, it is 4.0 + 0.01 × (3² + 2²) = 4.13.
Both models started with the same prediction loss, but regularization makes Model A more expensive because it depends heavily on one large weight. The optimizer is encouraged toward smaller weights when they predict nearly as well.
Here, λ (lambda) is a Hyperparameter. If it is too small, overfitting may remain. If it is too large, the model may become too restricted and Underfit.
flowchart LR
A[Too little regularization] --> B[May memorize noise]
B --> C[Balanced regularization]
C --> D[Simpler useful pattern]
D --> E[Too much regularization]
E --> F[May underfit]Regularization strength must be selected using validation results, not by repeatedly checking the final test set.
A concrete example, layered
For a simple beginner example: training the multi-feature house price model without regularization might let the weight for an irrelevant, coincidentally-correlated feature (like “house number ends in a lucky digit”) grow unreasonably large, since nothing discourages it from doing so; adding weight decay makes that large, unjustified weight costly, pushing training toward keeping it small unless the feature earns its influence through genuinely strong, consistent predictive value. For a production example: OpenAI’s published GPT-3 training recipe used the AdamW optimizer — Adam combined specifically with decoupled weight decay, as covered in the Optimization article — meaning weight decay regularization was directly built into the very training process that produced GPT-3’s 175 billion parameters, not treated as an optional afterthought.
Common misconception
A frequent assumption: that regularization is a fix applied only when a model has already been observed overfitting, almost like a repair. In practice, weight decay and similar techniques are typically built into the training process from the very start, as a standing precaution — exactly as GPT-3’s use of AdamW illustrates — rather than something bolted on reactively after a problem is already spotted. Regularization is best understood as a preventive design choice baked into how training happens, not an emergency patch.
Where this fits in what comes next
You now understand the general strategy of discouraging excess model complexity to protect generalization. The next article, Dropout, covers one of the most widely used and structurally distinct regularization techniques specifically for neural networks — one that works by randomly disabling parts of the network during training, rather than penalizing weight size directly.
In one sentence
Regularization deliberately discourages a model from fitting its training data too perfectly, trading a small amount of training accuracy for a meaningfully better chance at genuine generalization — and it’s a standard, preventive part of how most real models are trained, not a reactive fix applied after the fact.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed