There is a genre of post going around about typed decision engines: pull decision-making apart from execution, boil every fork down to a few primitives, put the engine at every bounded fork, and watch one slow expensive agent become an always-on decision layer that routes and scores in milliseconds.

The architecture is sound. I am not here to argue with it. I ran the thing against four jobs we already had working, kept the numbers, and the result was not the one I expected when I proposed the experiment.

It lost three of the four. And the interesting part is not that it lost. It is what each loss was made of.

We keep a file of claims this company has withdrawn and may never re-assert. It is enforced by a pre-push script built out of patterns. Each entry carries a probe line, the literal text a violation would look like, and often an antiprobe, an honest sentence that an earlier over-broad pattern wrongly flagged.

Those antiprobes are a better negative set than anything I could write, because each one is the exact sentence a previous pattern got wrong. Same subject, nearly the same words, opposite meaning.

Over ten probes and twelve antiprobes:

                probes caught    antiprobes cleared    false alarms
regex gate         10 / 10            12 / 12               0
typed engine        9 / 10            11 / 12               1

The regex is perfect here and free. The engine costs a call and scores 20 of 22.

One limit travels with that number, and it is not a footnote. This corpus is the regex's own gold standard, so by construction every probe is text those patterns were written to match. The comparison can show the engine is not better here. It cannot show whether the engine has more reach, because the case that would prove reach, a paraphrase no pattern covers, is exactly the case this corpus does not contain. Writing those paraphrases myself would be grading my own homework, so that question is open and I am not going to pretend otherwise.

The false alarm is the more expensive of its two errors. It flagged an honest cost breakdown as a withheld cost claim: the topic matched, the claim did not. A gate that fires on honest copy teaches people to route around the gate, which is worse than a gate that occasionally misses.

The posts make a specific promise about batching: many questions in one call is far faster and far cheaper than the same questions one at a time. That is correct, and it is the single most useful property of the whole design.

But in our claims run, the two questions in one call disagreed with each other.

One question asked whether the text asserts, as fact, any of the withheld claims. The other asked which one. On the violation it missed, the first read 0.91 and the second chose the none-of-these escape.

So it agreed the text asserted a withheld claim, and then failed to say which, and the combination let a real violation through. A single-question build would have hidden that: I would have seen one confident number and shipped.

One case, so take the size seriously: n is one. And my two questions were logically coupled, both about the same sentence, which is not the same shape as thirteen independent questions in a batch. So this is not a counter-example to the speed and cost numbers, which measure latency and spend and which I have no reason to doubt.

It is a second, separate argument for the same design. Batching is not only cheaper. It is also the only arrangement in which a model can be caught disagreeing with itself inside one decision, because the disagreement is only visible when both answers arrive together.

Because the missed violation scored 0.91 on that first question, triggering on that field alone looked like a free upgrade from 9 of 10 to 10 of 10. No new calls, data already on disk.

It is 10 of 10. It also fires on eleven of the twelve honest sentences.

trigger on assertion score >= 0.7    caught 10 / 10    false alarms 11 / 12

The honest sentences score 0.87, 0.88, 0.89, 0.90, 0.90. The cause is the question I wrote, not the model. I asked whether the text asserts as fact something a reader would take as one of those claims being true. What it answered is whether the text asserts something factual at all, which is true of every honest sentence in the file.

That is a true measurement of a quantity next to the one I claimed. A flattering headline was available, free, derived from data I already held. The only thing that stopped it was printing the negatives beside the positives.

Second job. Classify a building element from two measured numbers, thickness and run, on a few hundred cases with a known answer.

It produced five non-majority answers in two hundred and fifty two calls. All five were wrong. Its accuracy equalled the majority-class baseline exactly, which is another way of saying it answered the same label to everything.

The vendor's own documentation says this: perception is out of scope. Asking a decision engine to see a building from dimensions is perception. The overall accuracy figure is the trap here, because always answering one label scores the same. The diagnostic that matters is whether the answers vary with the input at all.

Third job. Given a task description, pick which specialist should take it.

Here it reads correctly and confidently, and I would use it. But the neighbouring half of the same job went the other way. Retrieving candidate matches from a catalogue of a couple of hundred entries looked like a job for the engine, and a local embedding model already running on the machine did it at eight out of eight on held-out cases, in under two hundred milliseconds warm, for nothing.

Wherever a deterministic answer already existed, the deterministic answer won. A regex. A local embedder. A count printed by a script.

That is not an argument against the design. It is an argument about where the boundary sits, and the boundary is narrower than the posts suggest. Every place we tried it, we kept finding the answer was computable, and a computed answer is free, instant, and identical on every run.

So we kept exactly one use: checking whether a cited source supports the claim made about it. The mechanical half of that, whether the quoted string exists at the cited line, needs no model at all. What remains is a linguistic question with no regex, and that is where a typed decision belongs.

Four jobs, one survivor, and a rule that will outlive the tool: before adding a decision engine to a fork, check whether the fork has a computable answer. Most of ours did.

metaend

Subscribe to receive the latest engineering benchmarks and insights directly in your inbox. Stay ahead with our curated analyses.

Sponsor / Ad

Access frontier and open LLM models via pay-per-prompt API or web interface: Explore NanoGPT.