Contents
At a glance
- Category
- Agent concept - model adaptation
- Origin
- Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network, 2015
- Practical method
- Fine-tune a small model on a large model's outputs for one task
- Best dataset source
- Your own production logs, which match the inputs you really see
- Pays off when
- Volume is high, the task is stable, and latency matters
- Check first
- Provider terms on using model outputs to train other models
What distillation is
Knowledge distillation trains a small "student" model to imitate a large "teacher" model. The technique goes back to work by Hinton and colleagues in 2015, and the modern practical version is straightforward: run the big model over a large set of realistic inputs, keep the outputs, and fine-tune a small model on those input-output pairs. The student learns the teacher's behaviour on that task without ever seeing the teacher's weights.
The pitch is narrow and real. On one well-defined task, a distilled small model can approach the teacher's quality while costing a small fraction per call and responding much faster. It will not match the teacher across tasks - it has traded generality for efficiency, which is the entire point.
How a business would actually do it
The realistic sequence is: ship the automation on a frontier model first, log every input and output through your observability layer, and after a few thousand real runs you have a distillation dataset assembled from production traffic rather than invented cases. Fine-tune a small model on it, then run both side by side on your eval set and compare quality, cost and latency.
Because the data is production traffic, it covers the distribution you actually see, including the awkward inputs that synthetic datasets never contain. That is why distillation from your own logs tends to outperform distillation from a generic dataset by a wide margin on your task.
Cost, licensing and when it pays
Distillation pays when volume is high and the task is stable. A workflow running a few hundred times a month will not repay the effort - the model bill was never the problem. A classification or extraction step running hundreds of thousands of times a month is exactly where the arithmetic works, and where the latency improvement may matter as much as the cost.
Check the licence before you start. Several providers' terms restrict using their model outputs to train competing models, and the rules differ by provider and change over time, so read the current terms of service for the specific teacher model rather than relying on what was true last year. Distilling into an open-weight student you host yourself also has to respect that student's own licence.
What you should know
Distillation narrows the model. The student will be good at what you distilled and unreliable outside it, so it is a poor choice for a general assistant and a good choice for a single step in a pipeline: classify this enquiry, extract these fields, decide whether this is urgent. A common production shape is a distilled small model doing high-volume routing with a frontier model handling the small share of cases the router flags as hard.
Keep the teacher available. When your task changes, you regenerate the dataset and re-distil, and your eval set is what tells you the new student is at least as good as the old one before it takes over.
Distillation FAQ
What is model distillation?
Training a small model to reproduce a larger model's outputs on a specific task. You run the big model over realistic inputs, keep the outputs, and fine-tune the small model on those pairs. The result is much cheaper and faster on that task and weaker everywhere else.
How is distillation different from fine-tuning?
Distillation is a use of fine-tuning where the training labels come from a larger model rather than from humans. Ordinary fine-tuning teaches behaviour from examples you author or collect; distillation teaches it from a teacher model's outputs, which is much cheaper to generate at scale.
When is distillation worth doing?
When one step runs at high volume, the task is stable, and you have production logs to distil from. At a few hundred runs a month the model bill was never the bottleneck. At hundreds of thousands, the cost and latency savings can be substantial.
Where does the training data come from?
Ideally your own production traffic, captured through your observability layer while the frontier model runs the task. That data matches the real input distribution including the awkward cases, which is why it consistently beats synthetic or generic datasets.
Is it legal to distil from a commercial model?
It depends on that provider's terms of service, several of which restrict using outputs to train competing models, and the rules change. Read the current terms for the specific teacher model before you start, and check the licence of any open-weight student model you plan to host.
Will a distilled model be as good as the original?
On the narrow task you distilled, often close. Everywhere else, no - you traded generality for efficiency deliberately. Use it as one step in a pipeline, and keep a route to the larger model for the cases your router flags as hard or unusual.
Primary source
Distilling the Knowledge in a Neural Network (arXiv)
Checked August 2026. Anything priced or rate-limited moves, so verify against the source before you budget.
Want this built rather than explained?
Knowing what distillation means is one thing; having it running in your business is another. We scope one workflow at a time, quote a fixed price in writing after a free consult, build it, and hand it over with documentation you own.
