What model distillation is — how big AI models teach small ones to cut costs
Model distillation is a technique in which a small, cheap student model is trained to imitate the outputs of a large, expensive teacher model. The student learns not only the final answers but the probability the teacher assigns to every alternative, so it can approach the performance of the teacher with less data and a much smaller size. Geoffrey Hinton, Oriol Vinyals and Jeff Dean formalised the method in 2015. A well-known example, DistilBERT, kept about 97 percent of the language understanding of BERT while being 40 percent smaller and 60 percent faster. Distillation underpins the move of top-model capability into cheaper models, and it is also at the centre of disputes over training on the outputs of a rival
The three lines
- Definition — a small student model learns to imitate a large teacher, including the teacher's full probability distribution
- Effect — DistilBERT kept about 97% of BERT's performance while 40% smaller and 60% faster; it is how cheap models catch up
- Dispute — training on a rival's outputs breaches most terms of service; OpenAI raised this about DeepSeek in 2025
Key questions
- What does model distillation mean
- **A large model acts as teacher and a small model as student.** The name comes from distilling a liquid down to its essence. | | Standard training | Distillation | |---|---|---| | Learns from | human-labelled answers | **the teacher model's outputs** | | Shape of the answer | cat | cat 90%, tiger 7%, dog 3% | | What the student gains | what is right | **how similar the options are** | | Data needed | a lot | comparatively less | **The second row is the key.** Learning only cat treats tiger and dog as equally wrong. Learning the teacher's probabilities also teaches that **a tiger is closer to a cat than a dog is.** Hinton called this **dark knowledge.**
- How much smaller and cheaper does distillation make a model
- **Representative figures:** | Case | Teacher | Student | Result | |---|---|---|---| | DistilBERT (2019) | BERT | DistilBERT | about **97%** of performance, **40%** smaller, **60%** faster | | DeepSeek-R1 distilled (2025) | DeepSeek-R1 | small Qwen- and Llama-based models | reasoning transferred to small models | | September 2026 price war | each lab's top model | mid-tier and lightweight models | claims of near-flagship quality at lower prices | **Read the last row carefully.** Anthropic says Opus 5.5 matches its top model Fable 5.1 on most tasks, and OpenAI presents GPT-6 Sol and Luna as offshoots of its flagship Astra. **Neither says whether distillation was used.** But moving top-model capability into a cheaper model is exactly the problem distillation solves (see 「Claude Opus 5.5 vs GPT-6 Sol pricing」).
- Why is distillation controversial
- **Because anyone's model can be the teacher.** Distillation does not require access to the teacher's internals — only its answers. Any model sold through an API can be queried. | Issue | Detail | |---|---| | Terms of service | major AI companies **prohibit using outputs to train competing models** | | January 2025 | OpenAI said it was investigating whether DeepSeek had distilled its models | | Counterpoint | the original models were themselves trained on others' work from the internet | | Defences | monitoring output patterns, blocking high-volume accounts | **Distillation is the most efficient way to build a cheap model and the easiest way to copy one.** Inside a company it is cost-cutting; applied to a rival it becomes a terms-of-service fight.
- Where do distilled models fall short
- **Outside what the teacher taught them.** | Weakness | Reason | |---|---| | rare questions | areas never queried are not learned | | long reasoning | multi-step reasoning depends heavily on size | | inherited errors | the teacher's mistakes are copied too | | safety behaviour | whether refusals and safeguards transfer must be tested separately | **That is why companies tier their models by task** — cheap models for volume, top models for hard reasoning. OpenAI's split between Sol for developer work and Luna for bulk summarisation reflects this. When a price list says same performance, ask **on which tasks.**
The most common way for a cheap AI model to catch up with an expensive one is to learn from it. That is model distillation.
1. Definition — teacher and student
Knowledge distillation trains a small, light student model to imitate the outputs of a large, capable teacher model. Geoffrey Hinton, Oriol Vinyals and Jeff Dean set it out in their 2015 paper Distilling the Knowledge in a Neural Network.
| Role | Trait | Cost |
|---|---|---|
| Teacher | large, accurate | expensive to run |
| Student | small, fast | cheap to run |
| Distillation | transfers the teacher's judgment | paid once, at training |
The cost is paid once; the savings recur every time the student runs.
2. How it works — probabilities, not just answers
| Method | Student sees | Information gained |
|---|---|---|
| label training | cat | right or wrong |
| distillation | cat 90% · tiger 7% · dog 3% | how close each option is |
Hinton called the information in those probabilities dark knowledge — absent from the label but already known to the teacher. The paper used a temperature setting to soften the distribution so the student could learn small differences.
For large language models the practice differs slightly. Teams often query the teacher at scale, collect its answers and reasoning, and train the student on them.
3. Cases
| Year | Case | Result |
|---|---|---|
| 2015 | Hinton et al. | method established |
| 2019 | DistilBERT | about 97% of BERT, 40% smaller, 60% faster |
| 2025 | DeepSeek-R1 distilled models | reasoning moved into small Qwen- and Llama-based models |
| 2026 | mid-tier price war | claims of near-flagship quality at lower prices (distillation not disclosed) |
The September 22, 2026 price list sits on this trend. Anthropic says Opus 5.5 matches Fable 5.1 on most tasks; OpenAI's Sol and Luna branch off Astra. Neither disclosed whether distillation was used, but the goal of moving top-tier capability into cheaper models is the same.
4. The dispute — using someone else's model as teacher
Distillation needs no access to the teacher's internals — just questions and answers. Any model available through an API can serve as a teacher.
| Issue | Detail |
|---|---|
| Terms of service | leading labs bar using outputs to train competing models |
| January 2025 | OpenAI said it was investigating possible distillation by DeepSeek |
| Counterpoint | the originals were trained on others' work from the internet |
| Defences | detecting bulk queries, account bans, output-pattern monitoring |
The same technique is efficiency inside a company and copying outside it.
5. Limits
| Limit | Explanation |
|---|---|
| Coverage | the student cannot learn what the teacher was never asked |
| Depth | long multi-step reasoning depends heavily on model size |
| Inherited errors | the teacher's mistakes transfer too |
| Safety | whether safeguards transfer must be tested separately |
So when a vendor claims equal performance, ask on which tasks.
6. Common questions
Is distillation the same as fine-tuning? No. Fine-tuning further trains an existing model on specific data; distillation trains a model using another model's outputs as the target. Distillation is often implemented as a form of fine-tuning (see 「What fine-tuning is」).
Are many distilled models open source? Yes. Because large models are hard to run on personal hardware, the open-model community releases many distilled versions (see 「What on-device AI is」).
7. What remains unconfirmed
- Whether the models released in September 2026 used distillation is undisclosed.
- The outcome of the DeepSeek distillation inquiry was not verified here.
- How safety properties transfer under distillation is not covered.
Sources
- arXiv — Hinton, Vinyals, Dean, Distilling the Knowledge in a Neural Network (2015)
- arXiv — Sanh et al., DistilBERT, a distilled version of BERT (2019)
- arXiv — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025)
- Wikipedia — Knowledge distillation
- Fortune — What slowdown? OpenAI, Anthropic release dueling models as AI price wars heat up