Inference is where control, cost and latency become real.
Inference is where control, cost and latency become real, because that is where the prompt, the output and the logs actually go. Training adapts weights. The running system is what your team lives with: location, operator access, retention, failover, and the price of a month under load. A capable model that can only sit on a shared endpoint can fail a confidentiality requirement even when a notebook looks strong. Scope the operating plan before you commit to a family or a fine-tuning path. This page is that plan — who runs the box, how slow is too slow, and what a month costs — not a slogan about private AI.
By Shen Pandi, Towards AGI. Reviewed September 2026.
Who this page is for
Read this page when the live question is where the running system lives, not whether the extract is tidy in a notebook. Typical readers are a platform owner who will be paged, counsel who asked where snippets go, and a sponsor who was told a fine-tune would keep everything in-house. The page is not a sovereignty essay and it is not a closed-API feature list.
If you need the four sentences that make independence a requirement, open sovereignty and then fill the checklist. If you need to compare a downloadable weight with a hosted API as a product choice, open the closed-API comparison. Stay here for the runbook: location, operator, logs, latency, and the month.
The prompt leaves someone’s hands. Write whose.
Every completed job is a prompt, an output, and usually a log. Those three objects have a location, an operator, a retention clock, and a legal entity attached. If you cannot name those four, you do not have an inference plan. You have a demo. Demos do not get paged at 2 a.m.
Buyers often treat “private” as a checkbox on a vendor page. Private is a chain: the application that builds the prompt, the network path, the box that runs the weights, the store that keeps traces, and the people who can open those traces when something breaks. A gap in that chain is where control actually fails. A green checkbox on a marketing site is not a chain.
This page is the operating conversation. It is not a sovereignty slogan and it is not a training method. If you came here because someone said fine-tuning would keep data in-house, that sentence is false. Adaptation does not move the endpoint. Open the sovereignty page for the wider choice, then come back here for the roster and the clock.
Logs are the quiet leak
Sponsors argue about weights and forget traces. Prompt logs often contain the document snippet, the customer line, the claim reference, the SKU that has not launched. Outputs can contain the draft that legal has not seen. If a platform default keeps those traces in a region or a tenancy you did not name, the model’s quality is not the incident. The store is the incident.
Write who can read production traces, for how long, and what is redacted before they land. Write whether a vendor operator can open a ticket and see the same objects. Write whether evaluation traffic uses a different store. Evaluation that copies live personal data into a shared notebook is still an inference path. Calling it research does not move the snippet.
Retention is a product decision. “Keep everything for debugging” is how a short-lived prompt becomes a long-lived file. “Keep nothing” is how you cannot investigate a bad extract. Pick a clock and an owner. Put both in the proposal. A proposal that is silent on traces is an incomplete operating plan, however tidy the adapter handover.
Where prompts are built, and which systems they pull from.
Where weights execute: managed, your cloud, on-premises, or edge.
Where traces sit, who can open them, and the retention clock.
Which legal entities and subprocessors are on that path.
What happens to a prompt when the model must not answer.
Location options are a short list. Treat them as one.
Managed cloud, your own cloud project, a room you own, or a device at the edge. Those are the usual four. Each one changes who can see the prompt and what a bad month costs. Mixing them without writing the mix is how a “private” pilot still phones a shared endpoint for embeddings or for the fallback model. The interesting hop is usually the one that never made the architecture slide.
A hosted endpoint can be the right pilot tool. It is the wrong production path if the requirement is that the snippet never leaves a named tenancy. An on-prem box can be the right production path. It is the wrong first invoice if nobody owns incidents, patches, or GPU failure. Edge can be the right path for a store device. It is the wrong path if the licence on the card forbids that device or the memory does not fit.
Fallback is part of location. When the specialist must stay silent, where does the work go — a human queue, a rules engine, a larger hosted model? That last option can undo the residency story. Write it before go-live, not after the first silence. Silence that phones another region is not a graceful degrade. It is a second running system you did not budget.
Decision prompts, not scores.
When
Then
The snippet or the log must not leave a named tenancy or room.
Treat hosted shared endpoints as a pilot-only option unless the contract names the same tenancy and access rules.
You need a result this month and residency is a preference, not a written rule.
A managed endpoint can be the first path. Write the preference so it does not get sold as a control.
Nobody on your side owns incidents or patches for a local box.
Do not buy on-prem as a slogan. Buy an operator plan or stay managed.
Silence must not fall through to a hosted model in another region.
Write the human or rules fallback. Do not leave a hidden API in the catch block.
Questions to settle for each location. Not a ranking.
Location
What you must be able to name
Managed endpoint
Region, subprocessors, log retention, whether vendor staff can read traces
Your cloud project
Account owner, network path, who holds keys, which services still leave the project
On-premises
Hardware owner, patch clock, licence to run the weights, incident roster
Edge or on-device
Memory fit, update path, licence for the device, what is stored locally
The operator is a person, not a logo
Someone patches the box, rotates the keys, reads the traces when a letter goes wrong, and decides whether a model update ships on a Thursday. If that someone is only a vendor brand, write the brand as the operator and accept what that does to independence. If that someone is “platform” with no roster, you do not have an operator. You have a hope that the desk will wait.
On-prem without an operator is how a sovereignty slide dies in the first GPU fault. Managed without a named vendor role is how a private-sounding VPC still has staff you cannot list. Write the role, the hours, and the escalation. The checklist asks whether you require that roster. This page is where the roster is drafted so a later incident is not a discovery exercise.
Updates are part of operating. A specialist that never gets a new base, a new adapter, or a new refusal rule will drift as the desk changes. Write who may push an update and how you will know it did not silently reopen a hop you had closed. An update path that uses a shared notebook in another region is a new inference location wearing a maintenance label.
Insurance: the letter draft is also a record
A handler pastes a snippet of a first-notification file into a prompt and receives a draft letter. That prompt is part of the claim file whether you meant it to be or not. If the endpoint stores the snippet in a region counsel has not accepted, you have created a second claims system with worse controls than the first. Quality of the draft does not repair that second system.
On-prem or a private project can be the right path for that draft. It still needs an incident owner at 2 a.m. when the box is down and the queue is not. Fine-tuning the letter style does not create that roster. The operating plan does. A beautiful adapter on a box nobody will patch is a file, not a running system.
Where the FNOL snippet and the draft sit
insurance
Write the path: workbench to prompt builder to weights to trace store. Name the region and the people who can open a trace. Name the clock after which the snippet is gone. If any hop is a shared vendor log you cannot name, the path is not private yet. Naming the hop is the work. Hoping the vendor “does not train on our data” is not a location.
If the fallback is “call the public API when our box is busy,” write that as a residency exception or remove it. A quiet catch block is how a private letter leaves the building on the first busy Friday. Busy Fridays are load. Load belongs in the month you price, not in a footnote.
Retail: catalogue prompts still leave the building
A merchandiser sends supplier copy and a schema. The output is a filled attribute row. The prompt often includes unreleased product names and seasonal prices. A shared endpoint that keeps those prompts is a leak of the next catalogue, not a theoretical privacy topic. The leak happens at inference time. Training later will not unsay the prompt.
Serving a compact model in your own project can close that hop. It does not close the hop if embeddings, moderation, or “just this one fallback” still travel to a third party. Draw the whole graph. The interesting edge is usually the one nobody put on the slide. Closed-API compare is useful when that edge is a hosted product you are pretending is plumbing.
Unreleased SKUs in the prompt path
retail
List every service that sees the supplier paragraph. If a hosted moderation API sees the unreleased name, treat that as an inference location. Either permit it in writing or take it off the path before go-live. Permitting it is allowed. Forgetting it is how a private project still publishes the spring line to a vendor log.
Store devices that run a local specialist still need an update path and a rule for what they keep overnight. A local cache of yesterday’s prompts is a store of tomorrow’s catalogue. Set the clock. A clock that says “until we debug” is not a clock. It is a drawer.
Latency and unit cost without a fake score
The task tells you whether a two-second draft is fine or whether a queue will melt. Load tells you whether a pretty single-user demo survives the afternoon. We will ask for expected completed jobs and the worst hour you care about. We will not invent a tokens-per-second figure and put it on this site as if we measured your box. Unpublished llm.build latency tables do not exist because we have not sat on your load.
Unit cost includes review. A cheap endpoint that produces letters a handler rewrites is not cheap. A dearer local box that stays inside the agreed phrases can win the month. Write both sides. The inference page is where that arithmetic belongs, not the fine-tuning page and not a parameter sticker on the SLM page. Mixing the arithmetic into a training invoice is how a team buys adapters when they needed a roster.
Availability is an operating line. Who is paged. What the desk does when the model is silent or the box is down. A sovereignty story that dies with the first GPU fault was never an operating plan. It was a slide. This page exists so the slide has to name a person.
How we scope an inference conversation
Bring the job, the expected volume, and the environment you want the weights to live in. We compare feasible hosting options against location, access, latency, and cost, then write the limits into the proposal. Model artefacts are only useful if they can run where you need them. That constraint belongs in the first scoping call, not after a pilot notebook. A stranded adapter is an expensive souvenir.
If the honest answer is “managed endpoint for the pilot, private project only if the bar is met,” we will write that sequence. If the honest answer is “this family cannot live in the room you named,” we will say so before you buy a training week. Read the closed-API comparison when the alternative is a product endpoint rather than a box you operate. Read the checklist when the alternative has to survive a security review as written lines, not as a diagram.
Questions we hear first
Can we deploy in our own environment?
Share your infrastructure, residency and access requirements. Deployment feasibility, dependencies and responsibilities need to be agreed for your chosen model and environment before an engagement is committed.
Does fine-tuning make our AI sovereign or secure?
Fine-tuning alone does not establish sovereignty or security. Those depend on where data is processed, who operates the infrastructure, access controls, licensing and operational practices. We scope these requirements alongside model adaptation.
Is a VPC endpoint the same as on-premises inference?
No. A private network path to a vendor-operated service is still vendor-operated. On-premises means you operate the box, or a named entity you contracted to operate it, on infrastructure you control. Write which one you mean.
Do you run production inference for us?
The usual engagement is a plan, artefacts where the licence allows, and a handover. Ongoing operation is a separate scope if you want it. We will not imply that a pilot includes a 24-hour desk we have not staffed.
Can we keep prompts out of logs entirely?
Sometimes, at the cost of harder debugging. We will write the trade: redaction, sampling, or a short clock. “No logs” and “we will investigate every bad extract” do not fit in the same paragraph.
Does a closed API always lose on residency?
No. Some product endpoints name a region and an access rule you can accept. Some do not. Compare them on the closed-API page, then write the hop on this runbook. A closed API that meets the written lines can be the running system. A downloadable weight that sits on a laptop is not.
If we already fine-tuned, is the operating plan too late?
No. Write the path the adapter actually uses. If a hop fails a line, you have a production exception or a reason to move serving. Training first and planning later is more expensive. It is not a reason to skip the plan.
Can we reuse last year’s latency demo as the month?
No. Last year’s single-user demo is not the worst hour you will serve. Bring expected completed jobs and the hour that would melt the desk. We will not invent a tokens-per-second figure to fill the gap.
Start with one task and a written scope.
Bring the workflow, the constraints and where inference should run. We use that brief to decide whether LLM fine-tuning, an SLM, retrieval or a readiness review is the right first step.