The Model Architecture article closed on a promise: holding architecture roughly fixed and systematically studying how performance changes with scale. This article covers exactly that body of research: scaling laws.
The simple definition
Scaling laws are predictable, mathematical relationships showing how a model’s performance improves as you increase its parameter count, training data, and compute — typically following a smooth, power-law curve rather than a chaotic, unpredictable one. Recall from the Large Language Model article’s discussion of scale producing genuinely new capability. Scaling laws are the actual, published research quantifying that relationship precisely — not just “bigger tends to be better,” but a specific, testable mathematical curve researchers can use to predict a model’s performance before ever actually training it at full scale.
Why discovering this relationship mattered so much
Recall from the Training article’s enormous, real costs — training a frontier model can cost tens of millions of dollars in compute alone. Before scaling laws were well understood, deciding how to spend a fixed compute budget — a bigger model? more training data? — was largely guesswork. Scaling laws turned this into a genuinely answerable, mathematical question: given a fixed budget, precisely how should it be split between model size and data to get the best possible resulting performance.
flowchart LR
A[Fixed compute budget] --> B{How to split it?}
B --> C[More parameters, less data]
B --> D[Fewer parameters, more data]
B --> E[Scaling laws predict which split performs best]The real story: a landmark paper, and the correction that overturned it
This is worth telling in full, because it’s a genuine, consequential scientific story with a real twist. In 2020, OpenAI researchers led by Jared Kaplan published influential scaling laws suggesting that, given more compute, you should prioritize growing model size dramatically over growing training data — their formula recommended increasing model size by roughly 5x for every 10x increase in compute, with data size growing much more slowly. This finding shaped how the field trained models for years, including GPT-3’s own design. Then, in 2022, DeepMind researchers led by Jordan Hoffmann published a paper studying over 400 models and reached a strikingly different conclusion: model size and training data should actually scale at roughly the same rate, not model size dominating. Their headline finding, stated directly: most large language models at the time — including celebrated ones — were significantly undertrained, meaning they had far more parameters than their training data could actually put to good use. To prove it, DeepMind trained a new model called Chinchilla, with 70 billion parameters — four times smaller than their own earlier 280-billion-parameter Gopher model — but trained on four times more data, using the exact same compute budget. Chinchilla convincingly outperformed the much larger Gopher on nearly every benchmark tested. A later 2024 paper by Microsoft and MIT researchers found the actual cause of the Kaplan-Chinchilla discrepancy: Kaplan’s original study had counted parameters slightly differently and been run at smaller scale, biasing its conclusions — reaffirming Chinchilla’s finding as the more reliable one.
ANALOGY vs. TECHNICAL REALITY
Analogy: Think of deciding how to spend a fixed education budget for a student — do you hire more tutors (more “model”), or buy more books and practice material (more “data”)? Kaplan’s original finding was like recommending hiring far more tutors than buying materials; Chinchilla’s finding was that a more balanced investment, tutors and materials growing together, actually produces a better-educated student for the same total spending.
Where this breaks down: A student’s education involves genuine, adaptive human learning. Scaling laws are precise, empirically fitted mathematical curves, derived from training hundreds of real models at different sizes and measuring their actual loss, as covered throughout the Loss Function article — a statistical relationship discovered through systematic experimentation, not an intuitive judgment about balanced investment.
Why this correction reshaped the entire industry’s practice
It’s worth being direct about the real, downstream consequence of the Chinchilla finding. Following its publication, major AI labs visibly shifted toward training smaller models on proportionally much more data — Meta’s Llama family, for instance, was explicitly trained following compute-optimal, Chinchilla-informed principles, prioritizing extensive training data over simply maximizing parameter count. A single, well-designed research paper, correcting an earlier, influential mistake, measurably changed how the entire field allocated enormous training budgets going forward.
A concrete example, layered
For a simple beginner example: given a fixed budget sufficient to train either a 10-billion-parameter model on a small dataset, or a 3-billion-parameter model on a much larger dataset, Chinchilla-style scaling laws would predict the smaller model, given more data to learn from, likely performs better on the same compute budget — a genuinely counterintuitive result to anyone assuming “bigger model” always wins. For a production example: DeepSeek’s own published research, covered throughout this phase, explicitly cites scaling law considerations in explaining its architectural and training efficiency choices — and Nvidia’s own public statement following the DeepSeek market reaction referenced “pre-training and post-training” scaling laws directly, alongside a newer concept called test-time scaling, covered back in the Test-Time Compute article, showing how central this research has remained to how the field understands genuine performance improvement.
Why scaling laws aren’t a perfectly settled, final answer either
It’s worth being honest about this being a genuinely still-active area of research, not a fully closed question. A 2025 paper reexamining Chinchilla’s own methodology found real ambiguities in exactly how the original study counted parameters, and ongoing research continues refining and stress-testing these relationships — scaling laws are a powerful, genuinely useful predictive tool, but, like the Kaplan-to-Chinchilla correction itself demonstrated, not necessarily the final word.
A small numerical mental model
Scaling-law experiments train multiple smaller models, plot training resources against loss, and fit a curve. Engineers can then estimate a larger run before spending the full budget.
More parameters ─┐
More data ────────┼──→ usually lower validation loss
More compute ─────┘ but with diminishing returnsThe original 2020 study reported power-law trends across more than seven orders of magnitude. This does not mean ten times the compute gives ten times the intelligence. A power law predicts a smaller, smooth improvement in loss as resources grow.
The later Chinchilla work changed the practical lesson: a large model can be undertrained if it sees too little data. It recommended scaling model parameters and training tokens together rather than spending almost all extra compute on parameter count. Sources: Scaling Laws for Neural Language Models and Training Compute-Optimal Large Language Models.
How companies use scaling laws
Before a costly training run, a lab can train smaller models, estimate how loss changes, and choose a promising allocation. DeepSeek’s LLM report describes fitting scaling curves and uses the approximation C ≈ 6ND, where N is parameter count and D is training-token count. It is a planning approximation, not the exact electricity bill or hardware runtime. Source: DeepSeek LLM.
Common misconception
A frequent beginner assumption: that scaling laws simply confirm “bigger models are always better,” with nothing more precise to say. As the Kaplan-to-Chinchilla story demonstrated, scaling laws are specifically about the optimal balance between model size and data for a given compute budget — and getting that balance wrong, as much of the field initially did, means wasting enormous amounts of real compute training models that are larger than their data can actually support.
Where this fits in what comes next
You now understand the predictable mathematical relationship between scale and performance, and the real research correction that reshaped how the industry actually trains models. The next article, Emergent Abilities, covers a related but distinct, and genuinely more contested, phenomenon — capabilities that seem to appear suddenly at certain scales, rather than improving smoothly along the predictable curves scaling laws describe.
In one sentence
Scaling laws are mathematical relationships predicting how performance improves with model size, data, and compute, and the real 2022 Chinchilla correction — proving a 70-billion-parameter model could beat a 280-billion-parameter one simply by training on proportionally more data — genuinely reshaped how major AI labs allocate their training budgets to this day.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed