Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

Build the Eval Set Before You Build the Feature

Model access is a commodity your competitors buy at the same price. Your labelled evaluation set is not purchasable at any price. Build it first.

techcirkle
5 min read
Cover image for Build the Eval Set Before You Build the Feature
Team reviewing AI evaluation results on screen
Team reviewing AI evaluation results on screen

Ask an engineering leader in 2026 what is blocking their AI roadmap. The answer is almost never "the models cannot do it".

The answer is: we cannot tell whether a change made things better.

That is a new class of problem and it deserves more attention than it gets, because it is the difference between teams shipping AI improvements weekly and teams shipping quarterly while arguing about whether the last release helped.

Why normal testing does not transfer

Traditional software is deterministic. Given the same input, you get the same output. A test asserts equality, it passes or fails, and CI tells you the truth.

AI features are distributional. The same prompt produces different outputs. More importantly, a change that improves the average can silently destroy a specific case that mattered.

You tighten a prompt to reduce verbosity. Nine cases improve. The tenth — the one where a customer asks about a refund on a cancelled subscription — now omits the condition that made the previous answer correct. Nothing failed. No test broke. You will hear about it from support in three weeks, by which point four more changes have shipped and nobody knows which one did it.

The method

Genuinely simple. The difficulty is entirely in doing it rather than in understanding it.

1. Collect 200–500 real cases with known correct answers.

Real, not synthetic. Pull them from your actual traffic, your support tickets, your historical human-handled cases. Weight toward edge cases and known-hard inputs — a random sample wastes most of its slots on easy cases you will never regress on.

2. Define what "correct" means, in writing.

This is the step teams skip and it is the one that carries the value. Frequently, writing it down reveals that two people on the team have different definitions, which explains a surprising number of previous arguments.

For some tasks correctness is exact match. For most it is a rubric: does the answer contain the required condition, does it avoid the prohibited claim, is the extracted amount right.

3. Run every candidate change against all of them.

Prompt changes. Model changes. Retrieval changes. Temperature. All of it. This has to be a script someone runs without thinking about it, not a manual process someone remembers to do when the change feels risky.

4. Never ship a change that lowers the score.

And when the score drops, look at which cases broke. The aggregate number tells you something is wrong; the per-case diff tells you what.

Why this is a moat, not just hygiene

Here is the part that is under-appreciated, and it is the strategic argument for funding this properly.

Your competitors can buy exactly the same model access you can, at the same price, today. Model capability is a commodity input. Whatever advantage a frontier model confers, it confers on everyone with a credit card.

What they cannot buy, at any price:

  • Your 400 real cases drawn from your domain

  • Your edge cases, discovered through your specific production failures

  • Your written definition of what a correct answer looks like for your customers

That set compounds. Every production failure you diagnose adds a case. Every ambiguity you resolve sharpens the rubric. Two years in, a competitor starting fresh cannot replicate it by spending money, because the input was time and incidents rather than budget.

In 2026 the eval set is the defensible asset. The model is not.

Practical notes from doing this

Version the set. When you add cases, the historical score becomes incomparable. Tag versions and keep the old ones runnable, or you lose the ability to say whether things have improved over a year.

Separate regression from exploration. One frozen set that must never go down, plus a growing set for discovering new failure modes. Mixing them means the bar moves every time you learn something.

Evaluate the retrieval layer separately. If your feature retrieves before it generates, build a labelled question-to-document set and measure retrieval precision independently. Otherwise you cannot tell whether a bad answer came from bad retrieval or bad generation, and you will tune the wrong stage.

Fund someone to own it. The genuinely new role of 2026 is a person responsible for maintaining the labelled sets that define correctness. Not a data scientist, not QA. Teams that funded this ship weekly. Teams that did not ship quarterly.

Start before the feature. The set is what tells you whether the feature is working at all. Building it afterwards means the first version shipped unmeasured, and you will be reverse-engineering a baseline from production complaints.

Frequently Asked Questions

How many cases do we need to start?

200–500 is a workable starting point for a single feature area. Fewer than 100 and the score is too noisy to act on; more than 1,000 at the start is usually procrastination.

Should the cases be synthetic or real?

Real. Synthetic cases test the failure modes you already imagined, which are not the ones that will hurt you. Pull from actual traffic, historical tickets, and human-handled cases with known outcomes.

Who writes the correctness rubric?

Whoever owns the outcome — usually product, with engineering. The exercise of writing it down frequently surfaces genuine disagreement about what the feature is supposed to do, which is valuable on its own.

How do we evaluate open-ended outputs?

With a rubric of checkable properties rather than an expected string: does it contain the required disclaimer, does it avoid the prohibited claim, is the extracted figure correct, does it cite a real source. Model-graded evaluation can help at scale, but the rubric still has to be written by a human.

Does this replace monitoring in production?

No. The eval set catches regressions before release; production monitoring catches the failure modes you did not anticipate. Every production failure should become a new eval case — that is the loop that makes the set compound.

Can we use the same set for a different model?

Yes, and that is one of its main benefits: it makes model swaps a measurable decision rather than a leap of faith. Build so you can change models in an afternoon and let the eval set decide.


The full 2026 picture — agents, cost curves, retrieval, regulation and org structure — is here: Top AI Developments for Business in 2026: What Actually Changed.

We build AI systems and LLM integrations evaluation-first. Get in touch.

Subscribe to techcirkle