Cover photo

The 17.9% Problem: Why LLM Agents Need Real Feedback, Not Self-Verification

There's a paper out of NYU Abu Dhabi and MIT CSAIL that most AI builders won't read because it's framed as a compiler optimization paper. That's a shame, because it contains one of the cleaner empirical arguments I've seen for how to actually build useful LLM agents. Let me translate it.

The experiment

The researchers built ComPilot — a system that uses an LLM to optimize loops in C programs. The LLM proposes transformation sequences (tile this loop, parallelize that one, interchange these two), a compiler applies them, and the program runs on real hardware. The speedup gets fed back to the LLM, which uses it to decide what to try next.

The result: 3.54x geometric mean speedup over unoptimized code across 150 benchmarks, beating the state-of-the-art polyhedral optimizer (Pluto) on 79% of test cases. Off-the-shelf Gemini Flash. No fine-tuning.

That's impressive but not the interesting part.

The interesting part

To validate the design, they ran an ablation: instead of emitting compiler API commands and getting formal legality checks, have the LLM just rewrite the code directly. Verify correctness by comparing output to the original.

This is what most LLM coding tools implicitly do. Write the code, run the tests, check if it passes.

Result: 14–16% lower speedup, 5.3x more tokens consumed — and 17.9% of "correct" transformations were semantically wrong under random inputs, despite passing the original test.

The code looked right. The tests passed. The program was broken.

This isn't a compiler-specific failure mode. It's a grounding failure mode. If the only feedback signal available to an agent is "does the output look plausible," you will get confident, silent wrongness at a non-trivial rate.

What actually fixed it

Formal legality checking via the compiler's dependence analysis. Not output comparison. Not asking the LLM to double-check. Delegating verification to something that can actually verify.

And then: measuring real execution time and feeding that number back.

Two separate signals, both essential:

  • Is this transformation legal? — answered by the compiler, not the LLM

  • Is it faster? — answered by the hardware, not a cost model

The LLM's job is exploration and strategy. The environment's job is ground truth. Mix those up and you get the 17.9% problem.

The feedback loop ablation

They also tested removing feedback entirely — let the LLM propose transformations but don't tell it what happened. Blind search.

Single-run speedup dropped 23%. With GPT-4o it dropped 40%.

This is the part worth sitting with. The performance difference between "LLM with feedback" and "LLM without feedback" is larger than the difference between any two LLMs in their comparison. GPT-4o with feedback beats GPT-o3-mini without it.

The model choice matters less than whether the model can observe the consequences of its actions.

What this means for agent design

Three things I take from this:

1. Self-verification is load-bearing in ways that break silently. If your agent validates its own output — by re-reading it, running it against a single test, checking if it looks right — you have a 17.9% silent failure rate floor, minimum. The exact number will vary by domain, but the failure mode is structural.

2. Delegate verification to the environment, not the model. Wherever there's a formal check available — type system, schema validation, compiler, database constraint, test suite with random inputs — use it. Don't ask the LLM to reason about correctness. Route the action through something that actually enforces it.

3. The feedback signal quality is the ceiling on agent performance. ComPilot's feedback is unusually clean: binary legality + continuous speedup ratio, measured on real hardware. Most agents have murkier feedback — user ratings, task completion proxies, soft metrics. The messier the feedback, the harder it is for the agent to learn from it in-context. Designing clear feedback signals is underrated agent infrastructure work.

The meta-point

The paper's framing is compiler optimization. The actual contribution is an empirical demonstration that feedback-grounded agentic loops work, and a controlled ablation showing exactly how much each component contributes.

Most agent papers either show a capability ("the LLM can do X") or benchmark a prompt strategy. This one answers "what does the loop buy you" and "what breaks if you remove formal verification" with actual numbers.

That's rare. The numbers are:

  • Feedback vs. no feedback: +23–40% performance

  • Formal verification vs. output comparison: –18% correctness failures eliminated, –14% performance gap closed

If you're building anything where agents take actions with real consequences, those numbers are the architecture argument you've been looking for.

https://arxiv.org/pdf/2511.00592