← Back to productsShipped · In use
Shipped · In use

Closed-Loop Evals

The classifier can tell you a ticket was safe for an AI copilot. Nothing in it could tell you whether that call survived contact with the customer. This system scores the highest-confidence tier against real ticket outcomes — replies, reopens, resolution — and when the tier stops holding up, it writes its own correction and commits it.

Prediction to self-correction

Inputs
Classified ticketsSupport platformSample + rate floors
Pipeline
01
Select Predictions
Narrows to the tickets the classifier called safe for a copilot to answer
02
Pull Outcomes
Fetches replies, reopens, channel and status in batches, only for tickets settled long enough to judge
03
Score Hold-Up
Joins prediction to outcome per subcategory and computes how often the tier actually held
enough evidence?
Correction Gate — a thin sample or a healthy rate leaves the pipeline here
04
Write Correction
Caps the tier for that subcategory and has GPT-5.4 draft the counter-example appended to the brand's prompt
05
Feed Next Run
The next classification run reads the cap and the new example — no human step in the loop
Outcomes
Correction + Commit
Override, prompt example and a commit carrying the stats that triggered it
Trend Logged Only
Below the sample floor, or holding up fine — recorded, nothing changed
Correction path
Correction gate
Observe only
Runs for
4 products
Validates
Copilot-ready tier
Ground truth
Live ticket outcomes
Audit trail
One commit per correction

Models

Models in the loop

No model scores the outcome
Whether a tier held up is computed from reply counts, reopens and channel. Deliberately arithmetic, because a model judging its own earlier call is what this system exists to replace.
GPT-5.4 drafts the correction
The only generative step: once the numbers trip the gate, it writes the negative example appended to that brand's prompt file.
Same provider swap
It shares the classifier's client, so the Claude configuration — claude-sonnet-5 and claude-haiku-4-5 — applies here without separate wiring.
The cap is not generated
The tier ceiling itself is written as plain configuration, not model output, so every automated change stays reviewable and revertible in version control.

Design decisions

Automating a change to your own prompts

Ground truth, not a second opinion
Held up means the customer did not reopen and did not need to keep replying. Chat channels are excluded from the rate rather than counted against it, because a chat reply means something different.
A lag gate before judging
Only tickets closed long enough to have reopened are pulled. Score too early and every not-yet-reopened ticket flatters the tier.
Version control as the review step
There is no human approval gate. A minimum sample floor decides whether to act, every correction commits with the statistics that caused it, and revert is the rollback path.
Minimal blast radius
Exactly one functional change to the existing classifier — a tier cap it reads on the way out. Everything else is additive files and prompt text.
Scope held deliberately narrow
Only the top tier is validated. The middle tiers need drafted-versus-sent reply comparison, which is a different problem and is explicitly deferred.
Unknowns kept visible
The sample floor and hold-up threshold are provisional pending calibration, and any data-quality quirk in the platform counts as truth until it is coded around.