Distilling an LLM into an SLM for a held-out task is a second project, not a first one. Train a smaller student to match a larger teacher only after that teacher already meets a written bar on unused work and you have a real reason to shrink — latency, device, or serving cost you can name. Distilling a teacher you have not measured produces a smaller uncertain model. Write both licences before anyone generates traces. A public distilled checkpoint is someone else’s student, not this run.
By Shen Pandi, Towards AGI. Reviewed September 2026. This page answers: distilling an LLM into an SLM for a held-out task.
Who this page is for
Read this when the brief says “distill it down” and nobody can show the teacher’s result on unused documents. Typical readers are a platform owner who wants a smaller footprint, a sponsor who saw a published DeepSeek student and thought that was the engagement, and counsel who have not yet read the teacher’s grant. This is not a compression tutorial. It will not publish an llm.build size-versus-quality curve.
If the large model has not yet been measured on your task, stop and do that first — usually SFT plus unused evaluation, not a student. If you only need adapters on the teacher, read LoRA. If you meant “use a published small checkpoint as a base”, that is fine-tuning or evaluation, not distillation. This URL owns teacher-first order, both licences, and the difference between a public student and your run.
Teacher first, student second
If the large model fails your extract, the student will fail faster. Measure the teacher on unused work, then decide whether latency, cost or device limits justify a student. “We cannot keep the teacher warm on the box we operate” is a reason. “Students are fashionable” is not. “A blog shrank a chatbot” is not a result on your book.
Write the bar the teacher already meets: extract completeness, exception rate, attribute agreement — whatever the workflow uses. Write the footprint reason in one sentence an SRE would recognise. If either sentence is missing, you are not ready to generate traces. You are ready to evaluate the teacher or to drop the shrink story.
Licence follows both models
A student trained on teacher outputs can inherit duties from the teacher and from the student’s own base. A student trained on a closed API may violate that API’s terms. Write both grants down before anyone starts generating teacher traces. Those two reads are the product. A later “we shrank it” slide does not repair a forbidden trace.
DeepSeek’s published story is a useful split, not a blanket. The DeepSeek-R1 repository states the code and model weights are MIT-licensed, support commercial use, and allow modifications and derivative works including distillation for training other LLMs. The DeepSeek-V3 repository states the code is MIT-licensed while use of the V3 Base/Chat models is subject to the Model License. Those are different grants. Do not treat “DeepSeek” as one permission. Read the family page for serving a 671B-class teacher; this page only needs the grant that covers traces.
Teacher card: may we use outputs to train another model?
Student card: may we adapt and distribute that smaller base?
Cross-family stop: Llama traces into a non-Llama student.
Closed-API stop: terms that forbid training on outputs.
Llama traces into another family are a stop
The published Llama agreements forbid using Llama materials or outputs to improve a non-Llama large language model. That clause is the usual stop when someone later proposes distilling traces into a Qwen or Mistral student. Internal research that never leaves the building still needs counsel if the outputs might later train another family. The adapter is not a new legal species and neither is a student.
This page will not reprint the Community Licence. The surviving duties live on the Llama family page. If the teacher is Llama and the student is not, treat that as a stop until counsel has read the current agreements. Do not generate traces “just to see” while that read is pending.
A public student is not your run
A published distilled checkpoint is someone else’s student, trained on someone else’s teacher traces, for someone else’s objective. Using it as a base is fine-tuning or evaluation. It can be a reasonable SLM candidate. It is not “we distilled our LLM”. If you need a student that matches your teacher on your documents, that is a scoped engagement with your traces and your unused pack.
Filing a downloaded student as a distillation deliverable is a category error. Operators will think they can regenerate it. They cannot. They will think the teacher’s bar transferred. It did not. They will think both licences were reviewed for your traces. They were reviewed for a public recipe, if at all. Say the honest sentence: we evaluated a published small model.
What changes in the run
The teacher produces traces or targets on your permitted inputs. The student is trained to match those targets. You pay teacher inference plus student training. Serving later can be cheaper if the student holds the written bar. The student still needs its own unused-work score. Matching train traces is not the same as matching the teacher on reserved documents.
Write both licences before anyone generates traces. Llama’s clause is a stop if the student is another family — see /models/llama. An R1 MIT teacher is a different grant from a V3 Model License teacher. A closed API is often a stop. The student base has its own card. None of those sentences are optional extras for a “quick shrink”.
Insurance: teacher must already meet the extract bar
No traces until the large extract already works
insurance
A wording team wants a smaller student because the large teacher is expensive to keep warm. Distillation can enter only if that teacher already meets a written extract bar on unused policies under handler review. If the teacher has never been scored on those policies, do not generate traces. You would be shrinking a guess. If the teacher meets the bar, generate permitted traces, train the student, and score the student on the same unused pack. If the student misses, you still have the teacher.
If counsel will not allow the teacher’s outputs to train another family, stop before the first trace. If someone points at a public DeepSeek student and calls it the deliverable, correct the filing: that is a candidate base, not this run. Coverage authority stays with the people you already named. A smaller file does not settle a claim.
Retail: shrink for latency after attributes already hold
Attribute teacher holds; the store-adjacent box is slow
retail
A catalogue team has a large teacher that already agrees with merchandisers on unused SKUs. They want a smaller student because the current serving path misses a latency budget on a store-adjacent device. Distillation is in scope only if the teacher traces are permitted on that device’s student family, and if the device plan is an operating plan, not a slide. Score the student on the same unused SKUs. A student that misses colour and fibre is not “faster”. It is unfinished.
If the honest constraint is region — product copy must not leave a boundary — generating teacher traces on a closed API can break that rule even if the student later runs locally. Write the path. If they only wanted a small published checkpoint, evaluate that checkpoint. Do not relabel the evaluation as distillation.
When distillation is the right conversation
Use it when the teacher already meets the bar, the shrink reason is written, both licences allow traces into the student, and you will score the student on the same unused pack. Then the run is teacher inference plus student training plus a second evaluation. It is not a shortcut around measuring the large model.
Skip it when the teacher fails, when a published student is actually what they wanted, when the API terms forbid training on outputs, or when the Llama-to-other-family clause is a stop. Those are not modelling puzzles. They are reasons to keep the teacher, change the student family, or drop the shrink story.
Decision prompts, not scores.
When
Then
Teacher meets the written bar; footprint reason is named; both licences allow traces
Distillation can be scoped. Budget teacher inference plus student training plus a second score.
Teacher has not been measured on unused work
Do not generate traces. Evaluate the teacher first.
The deliverable is a downloaded public student
Call it evaluation or fine-tuning of that checkpoint. Do not file it as your distillation.
Teacher outputs may not train the chosen student family
Stop. Change family, keep the teacher, or drop the shrink. Llama-to-other-family is a stop.
Count teacher inference as part of the bill
People compare “serving the student” with “serving the teacher” and forget the week of teacher tokens used to build traces. That week is part of the project. If the teacher is a closed API, you also pay their meter and you may be buying a terms problem. If the teacher is a 671B-class MoE, you are paying a serving plan you claimed you could not afford — just earlier.
Write the trace budget before you start. Write how many inputs, how long the completions are, and who may regenerate traces. An unbounded “keep sampling until it looks good” job is how distillation becomes more expensive than keeping the teacher. If you cannot name the token budget, you do not yet have a shrink plan. You have a hope that serving the student will erase the bill you have not written down.
What distillation will not replace
It will not replace measuring the teacher. It will not replace a serving plan for the student. It will not replace the SLM page’s question — is smaller even the engagement. It will not turn a closed API into an open student against the terms. Those are other doors. This page only shrinks a teacher you already trust, under grants you already read.
If the brief is “we want DeepSeek quality in a small box”, you have named a brand, not a run. Open the DeepSeek family page for the R1 versus V3 grant, then decide whether you are evaluating a public student or generating traces from a teacher you operate. Quality is not a transferable noun. Unused work is. A public student that happens to look small is still someone else’s objective, trained for someone else’s bar, under someone else’s traces.
Handover for a distillation run
The pack is the teacher identifier and its unused-work notes, both licence lines, the trace recipe, the student checkpoint, the student’s own unused-work notes, and the sentence that explains why the teacher was too large to serve. Include the grant that allowed traces — R1 MIT, V3 Model License, a closed-API refusal, or a Llama stop — as a written exhibit.
If you cannot show the teacher’s bar, you do not have a distillation handover. If you only have a downloaded student, you have a different engagement. We will ask for the teacher’s result before we treat a smaller file as this run. A shrink without that exhibit is a download with a story attached. File it as evaluation if that is what it was.
Sources
DeepSeek, DeepSeek-R1 repository (Accessed September 2026). DeepSeek-R1 states the code repository and model weights are MIT-licensed, support commercial use, and allow modifications and derivative works including distillation for training other LLMs. R1 and R1-Zero are listed as 671B total / 37B activated with 128K context.
DeepSeek, DeepSeek-V3 repository (Accessed September 2026). DeepSeek-V3 states the code repository is MIT-licensed while use of the V3 Base/Chat models is subject to the Model License. The published table lists 671B total / 37B activated parameters and 128K context. The series supports commercial use.
Questions we hear first
Is distillation cheaper than fine-tuning the large model?
Serving the student can be cheaper. Creating it costs teacher inference plus student training. Count both, then compare with adapting the large model or a small base directly.
Can we distill a closed API into an open student?
Often the API terms forbid using outputs to train another model. Read the terms. We will not plan a project that ignores them.
Does a student inherit sovereignty?
No. A smaller model still runs somewhere, and someone still operates it.
Is a published DeepSeek distilled checkpoint our distillation?
No. It is someone else’s student. You may evaluate or adapt it as a base. Your run needs your teacher, your traces and your unused pack.
Can we use Llama traces to train a Qwen student?
The published Llama agreements forbid using Llama materials or outputs to improve a non-Llama large language model. Ask counsel before any traces. See the Llama page.
Is an R1 teacher the same grant as a V3 teacher?
No. DeepSeek-R1 states MIT on code and weights including distillation for other LLMs. DeepSeek-V3 states code is MIT while Base/Chat use is under the Model License. Read the cited repositories and the family page.
Can we skip scoring the student if the traces look clean?
No. Matching train traces is not the same as matching the teacher on reserved documents. Score the student on the same unused pack you used for the teacher.
Does the student automatically meet the teacher’s bar?
No. Score the student on the same unused work. Matching traces is not a transfer of the teacher’s result.
Can we skip measuring the teacher if we already bought the student GPU?
No. A box for the student does not create a teacher bar. Measure the teacher first or drop the shrink.
Measure the teacher before you pay for a student.
If the brief is “distill it down”, we will ask for the teacher’s held-out result and the licence on both models first.