Open-weight model vs closed API for private inference

Who this comparison is for

Read this when the brief already said “it has to stay private” and someone has booked either a vendor fine-tune or a downloadable checkpoint without drawing the path. Typical readers are counsel who were told the model is “on our tenant”, a platform owner who will be asked to own incidents, and a specialist whose documents must not leave a named boundary. This is not a latency shootout. It will not pick a winner on public exams.

If the constraint is “open source” as a licence class, this is the wrong comparison — use open weights versus open source. If the constraint is a named family, use that family page. This URL owns where the prompt goes and whether you operate the box.

Where the prompt actually goes

Write the path: browser or application → your network or theirs → GPU or API → logs. If any hop is a vendor you have not named in a contract, you do not have a private-inference story yet. Open weights only help when that path stays inside the environments on the sovereignty checklist.

OSI’s open-weights note is the definition we use for the downloadable side: published parameters of a trained network, not the preferred form for modifying the system. That definition answers “can we hold a file?”. It does not answer “where does this request go on Tuesday?”. The Tuesday path is this page.

Fine-tuning a vendor UI is still their building

Vendor fine-tunes and “custom GPTs” adapt behaviour on their stack. That can be the right first step when quality on a narrow task is the only goal and data terms are acceptable. It does not give you weights to take home, and it does not satisfy a requirement that inference stay on your premises. The style you paid for usually stays on their path unless they export it.

Teams use the word fine-tune for that UI and for an adapter they will store. Write which one you mean. If the requirement is private inference, the UI path is the wrong exhibit no matter how tidy the completions look. See supervised fine-tuning for labelled pairs; this page is the boundary those pairs travel across.

Open weights still need an operator

Downloading Llama or Qwen does not create an SRE rota, a GPU budget or a fallback when the model should not answer. Private or on-prem inference is a plan. We treat it on the inference page and the sovereignty checklist, not as a checkbox next to “open source”. A checkpoint sitting on a laptop is not that plan.

Someone must patch the box, rotate keys, own the refuse path, and write where logs sit. If the organisation will not operate GPUs this year, say so. Buying open weights to feel sovereign and then sending production traffic to a hosted endpoint is the usual self-own. Be honest in the brief.

What changes in the run

On the closed-API path, the run is prompts and usually logs crossing a vendor boundary, plus any vendor-side adaptation that stays on their stack. You operate a contract and a region clause. You do not operate the GPU. You usually cannot take the adapted artefact with you.

On the open-weights path you operate, the run is a checkpoint or adapter you store, on hardware you named, under a licence you read. Prompts stay in the environment you actually run — if that sentence is true. Fine-tuning either path does not, by itself, create private inference. The path and the operator do.

Insurance: claims text leaving the building

Retail: in-region copy still hitting a hosted API

When a closed API is still the right first step

When you do not yet have unused test documents, when the task is exploratory, or when the organisation will not operate GPUs this year. Be honest. A downloadable checkpoint sitting unused is not sovereignty. A vendor probe with a written data term can be the cheaper way to discover that the task is undefined.

When you later change path, you usually cannot take the vendor fine-tune with you. Plan evaluation so the unused pack still works on the next base. That is why the schema belongs on the SFT page even if the first probe is a closed API.

Keep the unused pack portable when the path later changes

Paths change. A vendor UI this quarter, a box you operate next quarter, a hosted endpoint in between. The unused pack has to survive the change or you lose the comparison. If the only copies of the labels sit inside a vendor fine-tune UI, you do not have a portable task. Export the schema and the reserved documents into a place you control before you start.

Portable does not mean you train on the reserved documents. It means you can score the next base on the same work. That is how you change path without inventing a new success story. The readiness review writes the split. This page writes why the split has to leave the vendor building with you. A hop list that changes every meeting is not a private-inference requirement. It is an unresolved argument wearing a security slogan.

Hosted open weights are still someone else’s building

A Llama or Qwen endpoint you did not provision is a closed path with a downloadable brand on the brochure. The file exists somewhere. The prompt still leaves your network. If the requirement is private inference, “we used Llama” is not the exhibit. “We run this revision in this environment” is.

The same honesty applies to a vendor VPC story. Name the operator, the region, the log sink, and who can read it. If those names are the vendor’s, you have a contract story. You may still want it. Do not file it as a box you operate. Claims text that left for their region still left. Product copy that hit a hosted hop still hit a hosted hop, even when the brochure said in-region.

A laptop download is not an operator

Teams download a checkpoint, run a completion in a notebook, and file the screenshot as “we can do private inference”. That screenshot is a probe. It is not a rota, a refuse path, a log sink, or a patch plan. Private inference is boring operations. If nobody will own those operations this year, say so and keep the closed API under a contract you can actually read.

The closed-API path is allowed to win that honesty test. Winning it does not let you put “sovereign” on the slide. It lets you put “vendor path, named region, named terms” on the slide. Those are different sentences. This comparison exists so they stay different. A notebook screenshot is allowed to start a conversation. It is not allowed to close one.

Operating callout: name the boundary before the model

A useful brief names the hop, the log sink, the operator, and the refuse path before it names a checkpoint. Family choice is a later sentence. Method choice is a later sentence. If two stakeholders give two different answers about where claims text or product copy goes, you do not have a path yet. You have a vocabulary problem.

The inference page and the sovereignty checklist are the operating documents. This comparison exists so the vendor-UI story and the download-and-hope story have one URL that refuses both as private inference.

If four people write four hop lists that disagree, stop naming models. Resolve the list. A comparison cannot average “Dublin box we operate” and “US vendor we already pay”. Those are different products. Pick one for this quarter and file the other as a later path change with a portable unused pack. Family names come after the hops. Adapter files come after the hops.

What this comparison will not decide

It will not pick GPT versus Llama on a public exam. It will not pick Qwen versus Claude on your catalogue. It will not tell you that downloadable weights meet OSAID. It will not stamp a stack as sovereign because the checkpoint is local on Tuesday. Those are other pages, or they are unpublished scores we will not invent.

It will decide whether the first sentence in the brief is a hop list. If you cannot write the path, you are not ready to argue about families. If you can write the path and it ends in a vendor you contracted, a closed API can be honest. If you can write the path and it ends in a box you operate, open weights can enter — subject to the card and an operator. A slogan without hops is not a requirement we can implement.

Sources

1. Open Source Initiative, Open Weights (Accessed September 2026). Open weights are the published parameters of a trained network. Weights alone are not the preferred form for modifying the system.

Questions we hear first