For a specialist business task, start with supervised fine-tuning unless you already have a stable SFT model and preference pairs a specialist will stand behind. Direct Preference Optimization and related methods teach a model which of two answers is better. Rafailov et al. show DPO can align from those pairs without a separately trained reward model. That pair cost comes after an SFT baseline. It is not a first run on a messy inbox, and it is not a way to invent facts that were never in the context.
By Shen Pandi, Towards AGI. Reviewed September 2026. This page answers: DPO vs SFT for a specialist business task.
Who this page is for
Read this when someone proposed DPO, ORPO or “a bit of RLHF” as the first training activity on an extract, a classification job, or a pile of unread mail. Typical readers are a specialist who will have to author chosen-versus-rejected pairs, a sponsor who heard that preference methods are what the labs use, and a platform owner who has not yet shipped an SFT baseline. This is not a TRL tutorial. It will not rank DPO against PPO on a public chatbot table.
If you cannot yet write one good answer, read supervised fine-tuning. Preference work needs two plausible answers plus a reason. If the artefact you care about is an adapter file, read LoRA after the SFT baseline exists. This URL owns pair cost after that baseline.
Preference data is a different artefact
SFT needs “this is the answer”. DPO needs “A is better than B, and we agree why”. The second artefact is expensive in regulated document work. A handler can often mark an extract. Asking the same handler to invent a worse extract that is still plausible, then defend the ranking, is a different hour of their week. If specialists already disagree on a single target, they will not produce clean pairs. They will produce two politics.
Write the reason into the pair. “A is nicer” is not a reason a conduct team can reuse. “A quotes the source clause; B invents a limit” is. “A stays inside the published product-copy rule; B overclaims a fabric” is. Without that sentence, later operators cannot tell a style preference from a safety preference. The model will mix them.
Two completions on the same input, both plausible enough to teach from.
A chosen side a named specialist will defend.
A rejected side that fails a written rule, not a vibe.
A reason that would survive a conduct or merchandising review.
Pair cost after an SFT baseline
Collecting preference pairs before SFT is usually a waste of specialist time. If the model cannot yet produce two drafts that look like the job, you are ranking noise. Get a supervised baseline that already formats the extract, the letter, or the product copy. Then spend pair hours on the failures that remain: tone, refusal style, ranking of otherwise-correct drafts that still miss a written bar.
Count the hours honestly. Each pair is two completions plus a reason plus a review. In insurance correspondence that can be more expensive than the original extract labels. If the budget only covers one artefact, buy SFT labels. Preference work is the second cheque, not a fashionable replacement for the first.
RLHF is not the default enterprise path
Classic RLHF trains a reward model on pairs, then a policy against that reward. That is two training jobs, two artefacts to version, and a second model to incident. DPO folds the preference into a single optimisation on the policy, using the pairs directly. Rafailov et al. published that simplification. It is simpler operationally. It is not automatically safer for your customers or your files. Simpler ops is not a conduct control.
Both families assume you already have a model that can emit two plausible answers. They are alignment tools. They do not define the task. ORPO and KTO change the loss shape; they do not change the artefact you must collect or the rule that unused evaluation work stays unused. We will not pick a lab acronym for you from a blog date. We will not treat PPO-style RLHF as the grown-up default for a specialist desk.
Not a first run on a messy inbox
A messy inbox is a readiness problem. It is not a preference problem. If the mail is uncleared, unlabelled, and full of cases nobody has judged, ranking two guesses will copy the mess more expensively. Write a task, clear the files, and get one defended target per input before anyone authors a rejected twin.
Sponsors sometimes hope preference training will “sort out” disagreement between specialists. It will not. If two people cannot agree on a single gold draft, they will not agree on a ranking. Fix the standard. Then, if two drafts both meet the standard and one still fails a written tone or refusal bar, pairs can enter.
What changes in the run
The unit of data is a pair: chosen versus rejected, plus a reason a specialist will stand behind. The loss pushes the model toward the chosen completion and away from the rejected one. There is no separate reward-model training step in DPO. You still start from an SFT (or otherwise capable) checkpoint. The foundation licence does not change. Inference location does not change.
Preference training does not add facts that were never in the context. Ranking two guesses will not create a source quote. Leak the reserved comparison set into pair collection — because someone wanted hard negatives from the unused pack — and you have no comparison left. Treat pair authoring with the same split discipline as SFT labels.
Insurance: ranking two already-correct drafts
Both letters already extract; the bar is ranking
insurance
A claims desk has an SFT model that already fills agreed fields from a wording family. Two drafts on the same case are already correct on those fields. One is factually aligned but brusque; the other is polite but offers a next step the handler cannot promise. Preference pairs can enter here. Each pair is two already-correct drafts, a chosen side, and a reason a conduct reviewer would sign.
If the desk cannot yet extract the fields, DPO is the wrong first step. Ranking two broken extracts teaches ranking, not extraction. If two handlers already fight about a single gold letter, they will not author clean pairs. Fix the letter standard first. Unused test documents stay out of pair authoring. Coverage authority stays with the people you already named.
Retail: tone of product copy
Two attribute-faithful blurbs, one overclaims
retail
A catalogue team has an SFT model that already fills agreed attributes. The remaining failure is product-copy tone: two blurbs both mention the right fibre and fit, but one implies a performance claim merchandisers will not stand behind. Preference pairs can teach that ranking. The reason must cite the copy rule, not “sounds punchier”. Punchier that overclaims is the rejected side.
If merchandisers have not yet agreed the attribute schema, do not spend pair hours on blurbs. The first artefact is still one target per input. If product copy must not leave the region, pair authoring on a vendor UI is still their building — see the closed-API comparison. Preference files do not create a private-inference story.
When preference work is in scope
After SFT, when tone, refusal style or ranking of otherwise-correct drafts still fails a written bar, and you can collect pairs without leaking unused test documents. Not as the first training run on a messy inbox. Not as a way to skip a judge for a single extract. Not as a way to invent facts. Not as a reason to stand up a reward-model stack because a lab blog used RLHF.
If the remaining failure is missing knowledge, retrieve or change the documents. If the remaining failure is an undefined task, go back to SFT readiness. If the remaining failure is “the lab blog used DPO”, that is not a failure of the model. It is a failure of the brief.
Decision prompts, not scores.
When
Then
SFT baseline exists; remaining failures are ranking of otherwise-correct drafts
Preference pairs can enter. Budget specialist hours for reasons, not just ticks.
No single gold answer exists yet, or the inbox is still messy
Do not collect pairs. Read supervised fine-tuning. Two politics are not a dataset.
The gap is missing facts or changing handbooks
Retrieval or better documents. Ranking two guesses will not create a source.
Someone wants RLHF as the default enterprise path
Refuse the default. DPO is the simpler ops story if pairs exist. Neither replaces SFT.
Why labs and desks collect different pairs
Public preference sets are often chatbot helpfulness, harmlessness, or a style a crowd worker will click. A specialist desk needs pairs a named person will defend in review. That is a smaller set and a slower set. Do not import a public preference mix and call it your letter standard. The mix was not collected for your book.
If you mine production logs, permission, privacy and split discipline still apply. Logs are not automatically a preference set. Someone still has to mark chosen versus rejected and write why. An implicit click is not a reason a conduct reviewer can reuse.
Product-copy tone is still a written rule
Retail pairs on product copy fail when the reason is “punchier” and nobody can point at the claim the desk is allowed to make. Tone is not a vibe you collect after the fact. Tone is a rule that already exists in brand or legal, turned into a chosen-versus-rejected pair. If that rule is not written, you are back on the SFT gate: one target, not a ranking.
Insurance pairs on already-correct drafts fail the same way when the reason is “nicer”. Nicer that promises a next step the handler cannot give is the rejected side. Write that. If you cannot write that, you do not have preference data. You have two letters and a meeting.
A reward-model stack is a second product
Classic RLHF asks you to host, version and incident a reward model as well as the policy. That can be the right research path. It is the wrong default for a specialist desk that already struggles to version one adapter. If someone wants the reward-model path, they need a reason that is not “labs do it”. DPO exists so that reason has to work harder.
ORPO and KTO will keep arriving in vendor slides. Treat them as loss-shape cousins, not as a way to skip pairs or skip the SFT baseline. The artefact you cannot afford is still the specialist hour, not the acronym. If the slide cannot name the SFT checkpoint you would start from, the slide is early.
Handover for a preference run
The pack is the SFT checkpoint you started from, the pair file with reasons, notes from unused test documents, the exact base revision, any adapter produced, and the licence line from the family page. Include the written bar that the pairs were meant to move. Include who is allowed to author the next pair so a later intern does not invent a new taste.
If you cannot show the SFT baseline and the reasons, you do not have a preference handover. You have a ranking experiment. We will ask for the baseline before we treat DPO as an engagement artefact. Pairs without that baseline are specialist hours spent ranking noise. Pay for one gold target first.
Almost never as the first step on a specialist extraction or classification job. Get SFT and unused test documents first.
Is DPO safer than RLHF?
It is simpler operationally because there is no separate reward model. It is not automatically safer for your data or your customers. RLHF is not the default enterprise path.
Can preference training fix missing facts?
No. If the facts are not in the context or the weights, ranking two guesses will not create a source. Use retrieval or better documents.
Do we need a reward model for DPO?
No. Rafailov et al. align from preference pairs without a separately trained reward model. You still need the pairs, the reasons, and a capable starting checkpoint.
Can we mine pairs from production logs?
Only if permission, privacy and split discipline allow it. Logs are not automatically a preference set. Someone still has to mark chosen versus rejected and write why.
Is ORPO or KTO better than DPO for our letters?
We will not publish an llm.build winner among those losses. They still need pairs or equivalent judgements. Pick from ops simplicity and a scoped comparison, not from last month’s acronym.
Does preference alignment change the foundation licence?
No. You are still adapting the same base. Family duties survive. Read the family page.
Can we start preference work on a messy inbox to save time?
No. A messy inbox is a readiness problem. Ranking two unjudged drafts copies the mess. Write one gold target first.
Do not collect preference pairs until SFT has a baseline.
If someone proposed DPO first, bring the task and the labels you already have. We will say whether preference work is even in the conversation.