The Emergent Abilities article closed with a promise: a more settled, practical technique with its own genuine controversy. That technique is distillation.
The simple definition
Distillation is a training technique where a smaller “student” model learns to mimic the outputs of a larger, already-trained “teacher” model, compressing much of the teacher’s capability into a much smaller, faster package. Recall from the Transfer Learning phase’s broader theme of reusing prior learning rather than starting from scratch. Distillation is a specific, powerful instance of that idea — instead of training a small model directly on raw data and hoping it learns well, you train it to match a large, already-capable model’s actual behavior, which turns out to be a significantly more effective teacher than raw data alone.
Why mimicking a teacher’s outputs works better than training on raw data alone
This is worth explaining precisely, since it’s the real, published insight behind the technique, originating in a 2015 paper by Geoffrey Hinton and colleagues. Recall from the Probability Distribution article that a model doesn’t just produce a single correct answer — it produces a full distribution of probabilities across every possible option. A teacher model’s full probability distribution carries much richer information than a single correct label would: if a teacher model says an image is “90% cat, 8% dog, 2% everything else,” that 8% “dog” signal tells the student something genuinely useful — that cats and dogs are visually similar — information a simple “correct answer: cat” label would never convey. The student trains specifically to match this rich, full distribution, not just the teacher’s single final answer.
flowchart LR
A[Large, trained Teacher model] --> B[Teacher's full probability distribution on training examples]
B --> C[Small Student model trains to match that distribution]
C --> D[Compact model, much of the teacher's capability retained]ANALOGY vs. TECHNICAL REALITY
Analogy: Think of an expert chef training a promising junior cook — not just handing them a written recipe with a single correct outcome, but having them watch and try to replicate the master chef’s actual technique, including subtle judgment calls (a little more salt here, slightly less time there) that a simple recipe card alone would never capture.
Where this breaks down: A junior chef’s learning involves genuine, conscious observation and skill development over time. A distillation “student” is trained through the exact same gradient descent and loss-function-minimization process covered throughout the Training Mechanics phase — a mathematical optimization matching probability distributions, not an apprentice consciously studying a mentor’s technique.
The real, quantified success story: DistilBERT
This deserves to be told with its actual, published numbers, since it’s one of the clearest, most successful real-world demonstrations of this technique. In 2019, Hugging Face researchers released DistilBERT, a distilled version of Google’s BERT model — and the published results were genuinely striking: DistilBERT has 40% fewer parameters than BERT, runs 60% faster, while retaining 97% of BERT’s language understanding performance on standard benchmarks. That’s a remarkably favorable trade — giving up a small amount of raw capability in exchange for a dramatically smaller, faster model, something that made deploying capable language understanding on more limited, real-world hardware genuinely practical.
The real, unresolved controversy: did DeepSeek distill OpenAI’s models?
This is worth telling as a genuine, ongoing dispute, connecting directly to the MoE article’s DeepSeek story. When DeepSeek’s models produced results competitive with OpenAI’s far more expensive offerings, OpenAI publicly suggested DeepSeek may have used distillation — training DeepSeek’s models to mimic outputs generated by OpenAI’s own models, accessed through OpenAI’s API, without authorization. This would mean at least part of DeepSeek’s apparent efficiency breakthrough came not purely from architectural innovation, but from a large, capable teacher (OpenAI’s models) doing much of the underlying heavy lifting first. DeepSeek has denied wrongdoing. The dispute remains genuinely unresolved publicly, but it’s a real, direct illustration of a serious question this technique raises: if a company’s terms of service prohibit using their model’s outputs to train a competing model, is that a case of ordinary, legitimate distillation, or something closer to unauthorized copying of learned behavior — a real, live legal and ethical question the field hasn’t settled.
A concrete example, layered
For a simple beginner example: a small student model trained to classify movie reviews by mimicking a much larger teacher model’s confidence scores — not just “positive” or “negative,” but the teacher’s nuanced 73%-positive, 27%-negative breakdown — learns a more refined sense of sentiment than training on simple positive/negative labels alone would provide. For a production example: DistilBERT’s real-world numbers — 40% smaller, 60% faster, 97% of BERT’s performance — made it genuinely practical to run capable language understanding directly on mobile devices and other resource-constrained hardware, exactly the kind of on-device application the original paper specifically demonstrated as a proof of concept.
Why distillation has real, honest limits
It’s worth being direct about a genuine trade-off, not presenting distillation as a free win. A student model is fundamentally capped by what its teacher actually knows — recall from the DistilBERT numbers that even a well-executed distillation retained “only” 97% of the teacher’s performance, a small but real gap. Distillation compresses and transfers capability efficiently; it doesn’t exceed the teacher’s own ceiling, and a student trained to mimic a flawed or biased teacher will typically inherit those same flaws.
Follow one training example with numbers
For the sentence “The puppy chased the…”, imagine the teacher returns:
ball: 0.62 cat: 0.18 car: 0.08 other tokens: 0.12A hard label would reveal only ball. The full distribution also teaches the student that cat is more plausible than car here. A distillation loss measures the gap between teacher and student distributions, and gradient descent changes the student’s weights to reduce it.
A verified Google production example
Google released Gemma 2 in 2B, 9B, and 27B sizes and states that the 2B and 9B models were trained using knowledge distillation from the 27B model.
Gemma 2 27B teacher
↓ produces richer training targets
Gemma 2 9B and 2B students
↓
smaller deployment choicesGoogle reports that distillation improved the smaller models compared with training them from scratch using the same number of training tokens. Source: Google’s Gemma 2 explanation.
Common misconception
A frequent beginner assumption: that distillation is the same thing as fine-tuning, covered throughout the Training Paradigms phase. As this article has explained, they’re genuinely different — fine-tuning continues training an already-pretrained model on new, task-specific data; distillation trains an entirely separate, typically much smaller student model to mimic a different, larger teacher model’s behavior, a fundamentally different relationship between the two models involved.
Where this fits in what comes next
You now understand both the genuine technical success and the real, unresolved ethical controversy this technique has produced. The final article in this phase, Pruning, covers a related but distinct compression technique — not training a new, smaller model to mimic a larger one, but directly removing unnecessary pieces from an existing model instead.
In one sentence
Distillation trains a smaller student model to mimic a larger teacher model’s full output behavior rather than just its final answers, a genuinely successful technique — proven concretely by DistilBERT’s 40%-smaller, 60%-faster, 97%-as-capable real results — that has also become the center of a real, unresolved 2025 dispute over whether DeepSeek used it to learn from OpenAI’s models without authorization.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed