← Back to productsShipped · In use
Shipped · In use

AI Suitability Ticket Classifier

Before you can automate support, you have to know what support is actually doing. This pipeline reads a ticket export, extracts every customer intent, works out what data an answer would require, and assigns each ticket an AI-suitability tier — then audits its own tier calls with a second, deterministic method.

Export to executive report

Inputs
Ticket exportPer-brand prompt packTier + privacy rules
Pipeline
01
Clean Export
Strips personal data and markup from the raw export before anything reaches a model
02
Classify Tickets
Two GPT-5.4 calls per ticket — an actionability gate, then a strict-schema extraction of intents, required data and a suitability tier
03
Cluster Intents
mxbai-embed-large embeddings, UMAP plus HDBSCAN, then GPT-5.4 writes the category labels
04
Normalize Data Needs
A GPT-5.4 pass collapses free-text data requirements into consistent groups, keeping the raw wording
05
Re-derive Tier
Deterministic rules first, with all-mpnet-base-v2 similarity as the fallback — no model verdict is taken on trust
audit agrees?
Divergence Route — an independent read-only pass flags disputed tiers instead of overwriting them
06
Aggregate + Report
Rolls topics and tier distribution into the executive report
Outcomes
Dataset + Report
Per-ticket intents, data needs and tiers, plus the executive summary
Divergence Queue
Tickets where the two methods disagree go to human review
Main line
Divergence route
Human review
Runs for
4 products
Surface
Operator CLI
Models
GPT-5.4 + local embeddings
Output
Dataset + exec report

Models

Models in the loop

GPT-5.4 family
gpt-5.4-nano, gpt-5.4-mini and gpt-5.4, picked per role so cheap calls do the cheap work. Reasoning effort is tuned per stage rather than left at one setting.
Claude, switchable
claude-sonnet-5, claude-haiku-4-5 and claude-sonnet-4-6 sit behind a one-line provider swap, which is also how batch mode gets used to roughly halve spend.
mxbai-embed-large
Runs locally through Ollama to embed every extracted intent, so the clustering pass costs nothing per run and never leaves the machine.
all-mpnet-base-v2
Backs the independent tier audit: when regex bucketing cannot place a ticket, sentence similarity decides, under an explicit floor and margin.
UMAP + HDBSCAN
Not models but the shape of the taxonomy — dimensional reduction then density clustering, with unstable cluster IDs canonicalised across runs.
qwen3:14b
A local model kept available for offline labelling experiments, so prompt changes can be tried without spending on a hosted call.

Design decisions

What makes the output trustworthy

Never trust the model's own verdict
The suitability tier is recomputed from three yes/no facts the model must state, then re-derived a second time by a deterministic pass that flags disagreement rather than silently correcting it.
Brand-agnostic by composition
One universal rules block plus a per-brand examples file. Adding a brand is a prompt folder, not a code branch — and a linter enforces the shared wording rules after generation.
Cost and failure economics
Batch mode roughly halves spend, reasoning effort is tuned per role, and retries resend only unresolved tickets under a hard attempt cap with a bounded tolerance for stragglers.
A silent failure, encoded as guardrails
One run collapsed nearly two hundred labels into near-duplicates and reported zero cost. The fix — batched schema-bound mapping, per-item completeness checks and an explicit taxonomy status — is now permanent.
Stable IDs over unstable clusters
Density clustering renumbers itself on every rerun, so cluster identity is canonicalised per category and subcategory pair to keep comparisons across runs meaningful.
Known gap, stated plainly
Chunks are buffered in memory and written once, so a mid-run kill loses classifications already paid for. Runtime checkpointing is the next change, not a claimed feature.