Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

Your DORA Metrics Are Lying Now That the Team Uses AI

Deployment frequency and lead time still matter, but AI-assisted development broke how most teams measure them. Here is what to instrument instead.

techcirkle
4 min read
Cover image for Your DORA Metrics Are Lying Now That the Team Uses AI
Delivery metrics on a monitoring screen
Delivery metrics on a monitoring screen

The four delivery metrics are still the right spine for measuring software delivery. The problem is that the way most organisations instrument them now conceals exactly the constraint that AI-assisted development introduced.

This is a short post about three measurement mistakes and what to do instead.

Mistake one: measuring lead time from ticket creation

Lead time for changes is meant to capture how long it takes to get a change into production. Most teams measure it from when the ticket was created, because that is what the tracker makes easy.

That measurement is dominated by prioritisation and queueing before anyone writes code, which is a real problem but a different one. It also flatters you in precisely the wrong direction: when code authoring gets faster, ticket-to-production shortens slightly, and the metric reports an improvement even if the engineering path to production got worse.

Measure from first commit, or better, from merge to production deploy. Track backlog latency separately if you care about it — and you should, but not in the same number.

Mistake two: treating deployment frequency as the throughput measure

Deployment frequency is a capability measure: it tells you whether you can ship often. It is a poor throughput measure, because a team can deploy twenty times a day while a large backlog of reviewed-but-unmerged work sits behind a queue.

Since AI assistants started producing more change than review capacity can absorb, this gap has widened significantly. Deployment frequency stays flat or improves. Work-in-progress balloons. Nobody sees it because nobody instrumented the queue.

Instrument the queue. Specifically:

  • Open pull request count, tracked as a trend

  • Time from PR opened to first review

  • Time from first review to merge

  • Age of the oldest open PR

Splitting the two time measures is the crucial part. If time-to-first-review dominates, your constraint is review capacity, and no pipeline work will help. Most teams conflate the two and conclude, wrongly, that the problem is CI.

Engineers reviewing a change
Engineers reviewing a change

Mistake three: change failure rate that ignores silent rollbacks

Change failure rate is supposed to capture the proportion of deployments that degrade service. In practice most organisations count incidents that were declared, and a large fraction of failures are handled by someone noticing, rolling back, and saying nothing because it was resolved in four minutes.

That undercount matters more now, because the whole point of good delivery infrastructure at high change volume is that failures are caught early and reverted cheaply. If your process only counts declared incidents, an improvement in fast rollback looks identical to no failures at all — and you lose the signal that would tell you which categories of change are risky.

Count rollbacks as failure events automatically, from the deployment system rather than from a human process. It is a small instrumentation change with a large effect on the honesty of the number.

What to add for the AI era

Beyond fixing the four, three additions earn their place:

CI cost and duration per merge. Cost tracks change volume; duration tracks system health under that volume. Rising cost with flat duration is fine. Rising both means the system is straining.

Flake rate measured as re-runs. Count how often a pipeline is re-run without a code change. This is the honest measure and it is trivially available from your CI provider's API.

Percentage of services on the standard path. If you are investing in platform work, this is the adoption number that tells you whether it is landing. Everything else about platform value is downstream of this.

A note on gaming

Every metric that is used for evaluation gets optimised directly. Deployment frequency is trivially inflatable by splitting deploys. Lead time improves if you merge smaller, less meaningful changes. Change failure rate improves if you stop declaring incidents.

The defence is not better metrics; it is using them as instruments rather than targets, and pairing every throughput measure with a stability measure so that gaming one degrades the other visibly.

The full argument, including what to buy and how to vet a vendor claiming to fix all this, is at DevOps Services Company: How to Choose the Right Partner in the AI Era. For the consulting-engagement framing, our IT consulting services guide covers how these programmes are usually structured.

Frequently Asked Questions

Are DORA metrics still relevant?

Yes. The four remain the best available spine for delivery performance. What changed is that the dominant constraint moved to a stage the four do not measure directly, which is why pull request cycle time now belongs alongside them.

Where should lead time be measured from?

From merge to production deploy for engineering-path health, with backlog latency tracked separately. Ticket-to-production conflates prioritisation with delivery and moves for reasons that have nothing to do with your pipeline.

How do we measure flaky tests objectively?

Count pipeline re-runs on unchanged commits. It requires no test-level instrumentation, is available from any CI provider's API, and is much harder to argue with than a subjective sense that the suite is unreliable.

Should individual developers be measured on these?

No. All four are system measures. Applying them to individuals produces immediate gaming and destroys the collaboration that makes review and incident response work. Measure teams and systems.

What is a realistic target for pull request cycle time?

Under a day from opened to merged for typical changes, with time-to-first-review under a few hours. The absolute numbers matter less than the trend and than whether the first phase or the second phase dominates.

How often should these be reviewed?

Weekly for the operational numbers, quarterly for the trend conversation. Reviewing trends monthly encourages reaction to noise; reviewing them annually means you find out about a degradation a year late.

Subscribe to techcirkle