Readiness is the decision to train, not a pile of extra files.

Who this page is for

Read this page when someone has a share of documents and a date they want training to start. Typical readers are a data owner who has been asked to “just send the files,” a specialist who already knows the labels disagree, and a buyer who was told volume is the same thing as readiness. The page is not a generic data-quality audit and it is not a model-training week wearing a discovery costume.

If you already have permission, coverage, and a locked unused pack for one named job, you may be past this engagement — go to the fine-tuning article to see whether adaptation is even the spend. If the live question is where those files may sit while you review them, open inference and the checklist first. Stay here when the missing artefact is the decision itself.

The product is the unused pack and the permission to stop

People bring us a share of documents and ask when training starts. The honest product is earlier: a written yes, a written not-yet, or a written no. Yes means you may use the examples, they cover the job, labels are stable enough to learn, and a comparison pack will stay unused until the final scoring. Not-yet means we know what to fix. No means training would be theatre.

The unused test set is the part sponsors try to skip. It feels like waste. It is the only reason a later number means anything. This page uses “unused test documents” for that pack. The older phrase is a held-out test set: reserved for final evaluation, never shown to the optimiser, and not used to steer the next change. If you tune against those cases, you no longer have an independent result. You have a rehearsal. We will not call a rehearsal a go / no-go.

Permissions sit above quality. A beautiful label on a file you cannot use is not readiness. It is a liability. We start with source, ownership, and permitted training and evaluation uses. Then we look at whether the labels are worth learning. Quality work on a forbidden corpus is how a team spends a month making an unsafe pile tidier.

The three sets that have to stay separate

Training examples adapt the model. Validation examples help you choose settings and candidates while you still work. The unused test set is reserved for the final comparison with the baseline. Those three jobs cannot share the same rows. If a near-duplicate of a test wording sits in train, the comparison is contaminated even if the filenames differ. Filename hygiene is not methodology.

Separation is not a random shuffle of a zip file. Related policies, the same customer, the same product family, or the same week can leak. Insurance wordings that share a schedule, retail SKUs that share a supplier paragraph: those need a split rule you can explain to a later analyst who was not in the room. We will write that rule before anyone samples. A rule that only lives in a notebook cell will be lost the week someone “just adds a few more rows.”

If you later use the unused test numbers to guide another change, you need a fresh independent pack before you claim a final figure. Validation exists so you can steer without spending the unused pack. Spending the unused pack because the first number disappointed you is how a rehearsal becomes a press sentence. We will not help you write that sentence.

• Training: learn the task.
• Validation: refine and select while you still work.
• Unused test: evaluate once, then stop or commission a fresh pack.

How the unused test documents stay unused

Lock the identity of the unused pack before experiments begin. That means a pin: which files, which versions, which split rule. People who may run trials should not be able to browse those documents “to get a feel.” Feeling is peeking. Peeking is how a later extract looks familiar for the wrong reason.

Access control is part of methodology, not only security theatre. If the unused pack sits in the same shared drive as the train pile, someone will open it. Put it in a place the training path cannot read. Write who may open it, and only at the final scoring. If that sounds heavy for a pilot, the pilot is asking for a number it has not earned.

Near-duplicates are the quiet failure. Two wordings that differ by a schedule letter, two SKUs that share a paragraph, two letters that are the same template with a new name: if one side sits in train and the other in unused test documents, you are scoring memory of a sibling. The split rule has to follow the text and the family, not the primary key. We will sample for that leakage. We will not pretend a hash of the filename is enough.

After the final comparison, stop. If you want another change, commission a new unused pack or accept that you no longer have an independent figure. That is the whole point of reserving the set. A dashboard that refreshes “test accuracy” every week is a validation dashboard wearing the wrong name.

What the review actually checks

Generic twenty-eight-question lists bury the insurance and retail problems. Who owns the documents. Which uses are permitted. Whether two specialists already disagree. Whether related products leak across the split. Whether sensitive fields can be minimised before anyone copies a corpus. Those are the checks that change the decision to train. A spellcheck score does not.

Quality is specialist work, not a lint pass. Conflicting labels are a task problem. Coverage holes are a sampling problem. A model cannot invent a document family you never showed it and still be judged fairly on that family. We will name the holes. We will not fill them with synthetic rows and pretend the desk looks like that. Synthetic rows, if they appear later, stay on the train side after a quality pass. They do not enter unused test documents.

Versioning is how a later reader traces a number to a file. Dataset pin, transformation pin, model pin. Without those, a tidy table is a story. The sample report shows where those pins sit. The cells for scores stay empty until a separately scoped evaluation exists. This review does not fill those cells. Filling them is a different engagement with a different statement of work.

• Source, ownership and permitted training and evaluation uses.
• Label quality and specialist disagreement.
• Coverage of languages, document families and hard cases.
• Sensitive-data minimisation, access and retention.
• Near-duplicates and leakage across the three splits.
• Dataset and model versioning so a result can be traced.

Insurance: who may use the wording, and what must not leak

A binder’s PDF library is not a dataset until counsel says the use is allowed. Third-party wordings, cedent files, and personal data in a first-notification scan each have a different answer. We will not treat “we have SharePoint” as permission. The review names the source and the use, or it stops. A stop here is cheaper than a contaminated pilot that counsel later withdraws.

Leakage in this industry is often a shared schedule or a follow-on endorsement that sits in train while its sibling sits in unused test documents. A random split will not catch that. A family-aware split will. The unused pack has to be unused in substance, not only in filename. Substance means a later reader can see why those wordings were reserved together.

Retail: SKU families leak the same way

Supplier copy is often reused across colours and sizes. If every red variant sits in train and the unused pack is the blue variant with the same paragraph, you have not reserved a test. You have reserved a synonym. The split rule has to follow the text, not the SKU code. A code-wise shuffle is how a catalogue “passes” a comparison that a merchandiser would not recognise as unseen work.

Permissions here are brand, supplier, and sometimes a marketplace contract. A scrape of a competitor’s catalogue is not a cute extra. It is a use we will flag and usually refuse. Readiness includes saying no to a corpus that would make the later model awkward to defend. Awkward to defend is a commercial fact, not a taste.

What you get, and what you do not

The engagement is a written review for one task. Depth, fees, and dates are agreed before data is shared. You get findings, a split recommendation, a list of holes, and a go / not-yet / no. You do not get a trained model unless you buy that later. You do not get a score. The sample report on this site is the structure of the document. Performance numbers exist only after a separately scoped evaluation. We will not pre-fill a result cell “for illustration.” Illustration is how unpublished numbers become quotes.

Synthetic rows may fill a named gap later. They do not fix a permission hole and they do not replace representation of the desk. Origin, quality, and leakage checks still apply. That is also a shared FAQ. We will not expand it into a synthetic-data product on this URL. If you need a synthetic-data engagement, that is a different brief with a different honesty problem.

How to start without sending the corpus

Bring a high-level task description and a data inventory: where the files live, who owns them, roughly how many, which languages, which families. Agree depth and dates in the proposal. Then, and only then, we agree a transfer path that matches the sensitivity. The contact form is not that path. A zip on the first email is a readiness defect before we have opened it.

If the inventory already shows you cannot permit a use, you do not need to transfer anything to get value. The decision to wait is the product. That is what this heading promised: readiness is the decision, not extra files. Extra files without a decision are how SharePoint becomes a superstition about being “almost ready.”

Questions we hear first