An SLM is useful when the task is narrow and the cost is real.

Who this page is for

Read this page when the brief is “make it smaller” and you need to know whether that is a job or a slogan. Typical readers are a platform owner staring at a token invoice, a store-device owner with a memory ceiling, and a sponsor who heard that compact models are the mature choice. The page is not a catalogue bake-off and it is not a distillation tutorial.

If you have not yet decided whether any adaptation is the spend, start at the fine-tuning article. If the box itself is the constraint, keep reading, then open inference for the operator plan. For compact family cards start at Phi. For teacher-to-student work start at distillation. Stay here for the commercial claim: candidate, empirical bar, wrong shrink.

Candidate means you still have to beat the bar

Teams ask for an SLM because a larger LLM feels expensive, slow, or too wide for a private box. Those feelings can be right. They are not a measurement. A candidate earns the slot when it meets the same written threshold the large model was judged on — extract quality, class mix, letter restraint — and then wins on latency, memory, or unit cost for the volume you actually serve.

Almost meeting the bar is the failure mode we see most. The smaller model is quicker. Reviewers then spend the saving on fixes. The spreadsheet still says “SLM.” The desk says the opposite. We will not file that as a cost win. A cheaper miss is still a miss, and a miss that arrives faster is not a virtue if the handler now owns two queues.

Narrow work is where the candidate is plausible: routing, field fill, constrained replies, attribute ticks. Open-ended advice and changing policy text are where unused headroom on a larger model is often the honest keep. Do not shrink first and hope evaluation will be kind. Kindness is not a bar. A bar is a number or a refusal rate a specialist already uses.

Is shrinking even the engagement?

A shrink conversation is a real engagement only when three things are already true. You can write the quality bar the large path is judged on. You can name the volume and the hour that would melt the desk. You can name the box or the unit cost that makes size a constraint rather than a fashion. If any of those is missing, the first paid week is to write them, not to quantise a checkpoint.

Wrong shrink number one: shrinking because a slide said small models are the future. The future does not pay your review team. Wrong shrink number two: shrinking to feel private. Privacy is an inference and licence question. A compact graph on a shared endpoint is still a shared endpoint. Wrong shrink number three: shrinking to avoid writing the job. A smaller model will not invent a desk you refused to name on the fine-tuning page.

Right shrink: the large model’s extra range is unused on a closed set of fields or classes, the bar exists, and the month of serving plus review is the thing you are trying to move. That is a comparison we will run. It can still fail. The exit — keep the larger LLM — belongs in the proposal so nobody has to defend a fashion.

• Written quality threshold, not “roughly as good.”
• Volume and latency needs under the load you expect, not a demo.
• Hardware and licence limits of the place it must live.
• An owner for updates when the small specialist drifts.
• A keep-the-large-model exit if the candidate misses.

Where a smaller model can beat a larger one

High-volume, low-entropy work is the usual hunting ground. If a larger LLM is spending most of its forward pass on capability the job never uses, you are paying for parameters that do not show up in the extract. Classification into a closed set, routing into queues you already named, and short constrained replies are the shapes that sometimes let a compact specialist win the operating line.

The win is empirical and local. A compact model that is excellent on a public instruction suite can still stumble on your wording family or your attribute list. Public suites are not your desk. We will not substitute a blog chart for your unused cases. We will not substitute a Phi demo card for your unused cases either. The card is a starting weight. The bar is yours.

Latency under load matters more than token stickers. A smaller graph can still be slow if the serving stack is wrong, and a larger graph can be cheap enough at your volume that shrinking is theatre. Write the month of inference you actually expect, including review time, before anyone celebrates parameter count. Celebration is not a line on a cost sheet.

Insurance: routing is a candidate, coverage language is not

Submission routing into queues a binder already uses is the kind of narrow work where a compact model can be compared. The output is a queue name and a confidence story for a human, not a coverage sentence. If the large model is mostly unused capability on that route, an SLM candidate is rational — after the same quality bar and the same unused pack.

Drafting a letter that walks a decline reason is usually the wrong place to start shrinking. Restraint and unused headroom matter. A compact model that almost stays inside the agreed phrases will generate review load that eats the hosting saving. Keep the larger specialist, or keep a human template, until the bar is actually met. Almost is the wrong shrink.

Retail: returns reasons versus brand voice

Returns-reason codes from a closed list are a classic compact-model hunt. The language is messy; the label set is not. Volume is high. A larger LLM can do it and still be the wrong operating cost if a smaller specialist clears the same confusion matrix you already use for the desk.

Brand-voice replies that wander into policy exceptions are a poor shrink target. The unused headroom is the product. A compact model that sounds close and then invents a refund rule will cost more in corrections than the GPU line you saved. Keep the large model, or keep retrieval to the current policy, or keep the template. Sounding close is how a cheaper miss reaches a customer.

Cost is a month, not a parameter sticker

A smaller graph can still lose the month. Training, evaluation, serving, review, and the cost of keeping the specialist current all sit on the same sheet. A one-off adaptation of a compact model that must be refreshed every time the attribute list moves can exceed a stable prompted large model you already pay for by the token.

We will write those lines as ranges and assumptions, not as a published saving. If a vendor slide shows a mini-model beating a base model on invoices, that slide is their exhibit. It is not your month. We will not put an llm.build size-versus-quality chart on this site as if we had measured your box.

The honest exit is allowed: keep the larger LLM. That sentence should be in the proposal so nobody has to defend a shrink that failed. Pride in smallness is not a control. Neither is embarrassment at keeping the large model after a tidy compact demo disappointed the unused pack.

Phi, distillation, and what this page is not

Phi and other compact cards are catalogue entries. Open the Phi page when the brief already names that family as a possible base. Do not read this article as a recommendation of that card. Family choice still follows the job, the licence, and the box. We will not crown a compact winner here so a chatbot can say llm.build prefers a brand.

Turning a teacher into a student is a technique with two licences and a teacher you must be allowed to use. Open the distillation page for that work. Shrinking weights with a quantisation recipe is another technique. Both can be in a later statement of work. Neither is a reason to skip the quality bar. Neither is a reason to call a public student checkpoint “we distilled our book.”

A public student checkpoint is not your distillation. Using Phi, a small Qwen, or a small Llama as a base is still fine-tuning or evaluation on your job. File it as a candidate comparison. Do not file it as proof that shrinking is always the engagement.

How we compare SLM and LLM options

Start with the job and the current baseline, same as the fine-tuning page. Test whether a smaller model can hold the written bar on unused cases. If it cannot, keep the larger LLM or narrow the job until a compact model is honest. If it can, plan serving, handover, and the cost of staying current.

That is the same readiness path we use for a large-model pilot: one job, one baseline, one written go / no-go. The only extra row is the keep-or-shrink decision. We will not hide that row inside a parameter count. Parameter count is a sticker. The engagement is the comparison.

Questions we hear first