Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

The AI Readiness Assessment: Four Tests Before You Build

Before you build the AI feature you promised the board, test four things: data reality, permissions, evaluation and failure economics. In that order.

techcirkle
5 min read
Cover image for The AI Readiness Assessment: Four Tests Before You Build
Team assessing data readiness for an AI project
Team assessing data readiness for an AI project

The pattern is consistent enough to be a genre now.

A company promises the board an AI feature. An engineer builds a demo in a fortnight and it is genuinely impressive. It runs on a spreadsheet of hand-picked examples. Six months later the feature is not shipped, the team is on its third model, and nobody can explain to the board why something that worked in February does not work in August.

The demo was never the hard part. The four tests below are, and they run in dependency order — failing any one makes the later ones irrelevant, which is why running them in the wrong order wastes quarters.

Test 1 — Data reality

Does the data this feature needs actually exist, reachable, correct often enough?

Three separate questions hiding in one, and teams routinely answer only the first.

Exists is usually yes. Reachable is where it collapses: the data exists in a system owned by another department, behind an integration that does not exist yet, in a schema nobody has documented since the person who designed it left. A six-month integration project is not an AI project, and it needs to be named as a prerequisite rather than absorbed silently into the AI timeline.

Correct often enough is the one nobody tests. Pull 200 real records and check them by hand against ground truth. If your product catalogue is 82% accurate, no model will give you a 95% accurate answer built on it. This measurement takes a day and reprioritises more roadmaps than any other single activity in this list.

Test 2 — Permissions at retrieval time

Can the system answer "is this user allowed to see this?" for every retrieved item?

This is the failure that becomes a security incident rather than a bug report, and it is disturbingly common because the naive architecture makes it invisible.

The wrong design: retrieve everything relevant, generate an answer, filter what is displayed. The model has already read the salary review and the acquisition memo. It will not quote them directly — it will summarise around them, and the summary carries the information anyway.

The right design: authorisation is applied to the retrieval query itself, so the model never sees documents the user cannot access. This means your permission model must be queryable at retrieval time, at low latency, per user. If your permissions live in an application layer three services away from your vector store, that is an architecture problem to solve before the pilot expands beyond a friendly group.

Test 3 — Evaluation

Can you tell whether it is working before a customer tells you?

AI features are distributional rather than deterministic. A prompt change improves nine cases and quietly breaks the tenth. Without a systematic harness you find out from a support ticket, and by then several changes have compounded.

The method is unglamorous and it works:

  1. Collect 200–500 real cases with known correct answers

  2. Run every candidate change against all of them

  3. Track pass rate over time

  4. Never ship a change that lowers it

The strategic point is worth stating plainly: model access is a commodity your competitors buy at the same price. A curated evaluation set built from your domain, your edge cases and your definition of correct is not purchasable at any price. The eval set is the moat, not the model.

If you do one thing from this article, build this before you build the feature.

Test 4 — Failure economics

What does a wrong answer cost?

This determines the entire design, and skipping it is how teams end up with an architecture mismatched to their risk.

A wrong product recommendation costs a click. A wrong invoice total costs a customer relationship. A wrong eligibility decision costs a regulatory finding. These require different amounts of human review, different logging, different confidence thresholds and different escalation paths — and the cheapest architecture is the one matched to the actual cost, not the most cautious one available.

Write the number down. Not a severity label — a number, even a rough one. It converts an unbounded design argument into a bounded engineering decision.

The finding that repeats

Across these assessments the conclusion is nearly always the same sentence, differently phrased: the model is not the constraint.

The constraint is that the data lives in four systems, two of which nobody maintains. Teams that internalise this early ship working features. Teams that do not spend two quarters swapping models to solve a data problem, which no model can solve.

Frequently Asked Questions

How long does an AI readiness assessment take?

Two to three weeks for a single feature area. The data-accuracy sample and the permissions review are the time-consuming parts; the model evaluation itself is usually the quickest.

Do we need this if we are only using an off-the-shelf API?

If the feature operates only on public or already-clean data, you can go straight to a prototype. If it touches your proprietary data — which is where the business value usually is — the four tests apply regardless of which vendor's API sits behind it.

How many examples do we need for an evaluation set?

200–500 real cases to start, weighted toward edge cases and known-hard inputs rather than a random sample. Grow it every time production produces a failure you did not anticipate; that is the compounding asset.

What is the most common reason AI projects fail?

Data access and quality, by a wide margin. The second most common is the absence of an evaluation harness, which means the team cannot tell whether their changes are helping and starts making decisions by anecdote.

Should permissions be enforced at retrieval or at display?

Retrieval, always. Filtering at display time means the model has already read the restricted content and can leak it through summarisation without ever quoting it.

Can we skip evaluation for an internal tool?

Internal tools are where unevaluated AI does its quietest damage, because users assume competence and there is no customer to complain. If anything, internal tools need the harness more, because nobody else is checking.


The full guide — five consulting engagement types, pricing, SOW clauses and how AI shifted the economics — is here: Software Development Consulting Services: What You Actually Get for the Money.

We run these assessments alongside LLM integration and agentic workflow development builds. Get in touch for a scoping call.

Subscribe to techcirkle