
There is a specific kind of dashboard that appears about four months into an AI workflow project. It shows tickets processed, workflows deployed, models in production, and a large percentage labelled automation rate. Every line goes up. Leadership is pleased.
It is measuring activity, and activity metrics are easy to make go up. Here is what to measure instead, and — more importantly — what to capture before you ship, because several of these become unrecoverable once the system is live.
Take the baseline first, or you have nothing
The single most common irrecoverable mistake: shipping before measuring the before state.
You need four numbers, captured while the process is still entirely human:
-
Volume — transactions per month, with seasonality if it exists
-
Fully loaded cost per unit — not salary divided by tickets. Include supervision, rework, and the QA sampling that operations budgets usually bury elsewhere
-
Exception rate — what share already gets escalated or handled unusually today, which is your realistic ceiling
-
Cycle time — trigger to resolution, as a distribution rather than a mean, because the tail is where customers notice
Most organisations discover here that they genuinely do not know number two. That discovery is worth the exercise on its own, and it is also why so many pilots described internally as successful cannot survive the question compared to what.
Once the workflow changes, the before state is gone. You cannot reconstruct it later from a system that no longer works that way.
The five metrics that hold up
Cost per unit of work, before and after, on an identical unit definition. The definition is where this goes wrong. If "ticket" silently becomes "ticket excluding those routed to the specialist queue," the number improves without the work improving. Freeze the definition in writing and keep volume visible next to it so a mix shift cannot flatter you.
Resolution rate as a trend, never a point. The share handled without human involvement. Week one and month three differ substantially, because early escalations reveal failure modes you then fix. Anyone quoting a single accuracy figure for a production system without stating when it was measured has not watched one mature. Report it weekly with the trend line.
Rework rate. The share of automated outputs later corrected, reversed, or complained about. This is the metric most likely to be omitted and the one that determines whether the system is actually working, because a workflow that resolves fast and wrong is worse than the human it replaced. It also tends to move in the opposite direction to resolution rate when someone tightens thresholds too aggressively — which is exactly why both must be on the same dashboard.
Cycle time, as a distribution. P50 and P95. Automation usually improves P50 dramatically and can make P95 worse, because escalated cases now wait in a queue nobody has staffed properly. If you only report the mean you will miss a real degradation in customer experience while celebrating.
Token cost per transaction. Small but not negligible, and unlike traditional savings it grows with adoption. Track it as its own line, attributed per workflow at the egress boundary. It moves with provider pricing, prompt size, retrieval volume, and how often your self-check step fires. A prompt change that triples cost is otherwise invisible until the invoice arrives.
Three metrics that flatter you
Automation rate without a rework denominator. Ninety percent automated tells you nothing if a quarter of it is wrong. Always paired, never alone.
Time saved, computed as volume times an assumed per-unit saving. This is arithmetic dressed as measurement. Unless headcount, overtime, or a backlog actually moved, the saving exists only in the model — and avoided headcount that was never going to be hired is not a saving.
Any composite index. A single "efficiency score" combining several inputs exists to obscure a declining component. In this category the component most likely to decline quietly is rework quality, which is precisely the one you most need visible.
Instrument at the boundary
Practically, most of this falls out of one architectural decision: route all inference through a single internal service rather than calling providers from application code.
That boundary is where you attribute token spend per workflow and per transaction, enforce which fields may leave, pin provider and region, and — critically — persist the decision record before the caller sees the response.
For each execution, persist the input snapshot, everything retrieved, the prompt version, the model version, the raw output, each gate check with its individual result, the final verdict, and for escalations the human decision. Keyed to the business record.
That last element is what makes honest measurement possible at all. Every escalation is a labelled example: what the system would have done versus what a human actually did. After a few thousand pairs, threshold tuning becomes an empirical exercise rather than a product manager's intuition, and you can compute the cost of disagreements for any candidate threshold before changing it.
Retrofitting this after go-live is expensive precisely because the interesting transactions have already passed through unrecorded.
Evals: the metric behind the metrics
None of the above is stable without a regression suite of real cases with known-correct outcomes, run against every prompt, model, retrieval, and gate-rule change.
Use real transactions, not synthetic ones, and make every production failure add a permanent case. Fifty well-chosen cases beat five hundred generated ones, because the value lies in covering failure modes you have actually experienced.
Without it, four things change independently underneath you — prompt version, model version, retrieval index, gate rules — and any of them can move your numbers. Attributing a metric shift to the right cause becomes guesswork, and a provider deprecating a model version turns into an emergency rather than a scheduled regression run.
What the ninety-day output should look like
One sentence with a dashboard behind it: this workflow cost $X per transaction with a Y-day P95, it now costs $Z at N percent automated resolution with an M percent rework rate, at this volume.
That sentence is what unlocks further budget internally. Slide decks do not, and a pilot with no baseline cannot, because you have no defensible way to demonstrate the improvement was real.
Full context, including 2026 US cost benchmarks and the compliance constraints shaping the architecture: Digital Transformation Company in USA: How to Choose the Right Partner in 2026.
We build and instrument these systems at TechCirkle — agentic workflow development and LLM integration.

Frequently Asked Questions
What should I measure before deploying an AI workflow?
Four numbers while the process is still fully human: monthly volume with seasonality, fully loaded cost per unit including supervision and rework, current exception rate, and cycle time as a distribution. Once the workflow changes these become unrecoverable, and without them every later savings claim is unfalsifiable.
Why should resolution rate be reported as a trend rather than a single figure?
Because it rises substantially between week one and month three as early escalations reveal failure modes you then fix. A single number without a measurement date is close to meaningless, and quoting one is a reliable signal that the speaker has not operated a system through its maturing period.
What is rework rate and why does it matter so much?
The share of automated outputs later corrected, reversed, or complained about. It determines whether the system genuinely works, because fast and wrong is worse than the human it replaced. It also tends to move opposite to resolution rate when thresholds are tightened aggressively, which is why both belong on the same dashboard.
How do I track token cost per transaction properly?
Route all inference through a single internal service and attribute spend per workflow and per transaction at that boundary. Track it as its own line rather than folding it into a run-rate, because unlike traditional savings it grows with adoption and moves with prompt size, retrieval volume, and how often self-checks fire.
Why avoid composite efficiency scores?
Because a single blended score exists to obscure a declining component, and in AI workflows the component most likely to decline quietly is rework quality — the exact thing you most need visible. Five separate metrics reported monthly are harder to game and easier to act on.
How many eval cases does a production workflow need?
Fifty well-chosen real cases with known-correct outcomes beat five hundred synthetic ones. Every production incident should add a permanent case. The suite must run against changes to prompts, models, retrieval, and gate rules, since all four move independently and any can shift your metrics.
What makes "time saved" an unreliable metric?
It is usually computed as volume multiplied by an assumed per-unit saving, which is arithmetic presented as measurement. Unless headcount, overtime, or a backlog actually moved, the saving exists only in the model — and avoided hiring that was never budgeted is not a saving that any finance function will recognise.