We run what behaves like a small company of AI agents: one orchestrator that plans and delegates, and six department heads that each own a single slice of a git repo. Seven agents in total, one manager, every consequential decision signed and written to an append-only audit log. The whole thing runs on omp (omp.sh / Oh My Pi), a coding-agent harness. (Aside, because the names collide and language models keep conflating them: omp here is the coding agent, not OpenMP, the shared-memory parallel-programming API. Different thing entirely.)
We showed how we run that company structure in Running a company as an org chart of AI agents, and how we added oversight in Keeping a human in the loop over a company of AI agents. Before we turned the company outward to chase revenue, we gave it one last internal tune-up. We pointed the agents at their single largest variable cost, remote inference, and gave them one rule before they were allowed to change anything: measure first.
That rule earned its keep immediately.
The plan was the obvious one. Remote frontier models are expensive and rate-limited. A lot of agent work is token-heavy but not intelligence-heavy: summarizing, classifying, extracting, digesting verbose tool output. Push that cheap-heavy tier onto a small local model on the desktop GPU, keep reasoning and code generation remote, and the bill should drop.
So the first milestone was pure telemetry, and it was a hard gate: nothing routed, nothing trained, until a baseline report existed. The harness already writes per-call usage to local session logs, so this was a read, not an instrumentation project. Over a 23-day window the company had made 5,846 remote model calls, burned 1.34 billion tokens, and spent $1,146.81, about $49.90 a day.
Then the number that rewrote the plan. Of those 1.34 billion tokens, 96.2% were cacheRead: context the system had already sent, being re-sent and re-billed on every step. Re-reading cached context alone cost $647.78. Actual new generated output, the part that looks like thinking, was 20.2% of the bill. And the tier the whole plan was built to capture, discrete low-complexity agent tasks, turned out to be 53 calls and $1.04, which is 0.09% of spend, already running on the cheap model.
The cheap-heavy work we set out to route away did not exist as a routable set of agents. The orchestrator was re-reading 228 cached tokens for every token it generated, peaking at 413x on one heavy day. The cost was not task mix. It was repeated payload inside reasoning sessions. The measurement caught the plan's core assumption being wrong before a single byte of traffic moved. That is the entire reason the gate exists.
The reframe pointed at compression, not routing: shrink the verbose tool outputs before they enter the context that gets re-cached every step. That still needs a competent local model, so we stood one up.
The pick was Qwen3-4B-Instruct-2507 at Q4_K_M, served by Ollama. On a consumer RTX 3070 Ti with 8 GB, it decodes at about 141 tokens a second, sits fully on the GPU at 6.3 GB with the context window we chose, and takes 2.5 GB on disk. Comfortably past the 50-to-100 tokens-a-second we expected from a small quantized model, because it never spills to CPU.
We deliberately chose the non-thinking instruct build, not a reasoning variant. For digestion work, a thinking model spends its generation budget on a private monologue and can return empty content. The dedicated instruct split emits no reasoning, so every output token is useful, and a whole class of failure cannot happen. More on that failure below, because we walked into a close cousin of it anyway.
The real work was a hook that fires on a successful tool result. If the output is verbose enough and on the allowlist, the local model digests it before it ever enters the cached context, with a prompt tuned to lead with errors and final status and to preserve every exit code, file path, line:col, URL, identifier, and number verbatim, dropping only repetition and progress noise.
On real bash output the mechanism works well. Measured against live token counts, it cut three log samples by 95%, 92%, and 84%, roughly 90% on average, and preserved every critical token: the TypeScript error codes, the failing test assertion, the SQLite error, the exact file.ts:line:col locations, the exit codes. Faithful, not just smaller.
The part we are most confident about is not the compression. It is the fail-safe. The hook ships off by default. The committed default is all-remote. It is allowlisted to bash only. It hard-excludes read, edit, and write, because those carry code and exact data the agent needs byte for byte. It runs a cached health check, and if the local model is unreachable it passes the original output straight through. We proved this with the desktop effectively off: the agent received the full, undigested output, the run exited 0, and there were zero manual steps. A cost optimization that can break a tool call is not worth having. This one cannot. When local is down, nothing about the company changes except the bill.
Then we measured what the hook would save in production, and it is small.
Across the entire 23-day history, only 12 bash outputs landed in the hook's size window, totaling about 145 KB. Bash output is overwhelmingly tiny: 88% of it is under 1 KB, things like git status and one-line pipelines. Bash is just 3.8% of the tool-output bytes worth digesting. The real volume is read output, 71% of it, and read is exactly what we exclude on purpose, because code and data have to survive verbatim. The big budget line, the 96.2% cacheRead, is mostly the system prompt, tool definitions, and reasoning history being re-read, not tool output at all.
Apply the measured 90% reduction to that thin slice and the projected saving at the committed default is well under $2 over a comparable 23-day window, under 0.2% of the bill, with no measurable reduction in rate-limit hits. The harness was also already doing most of this work: it compacts context de-deterministically, protects the newest tokens, and trims many command outputs before our hook ever sees them. We are picking up a residual on a residual.
So the win is not a smaller bill. The win is two things. One, a fail-safe pressure valve for the rare bash-heavy batch day, free and near-zero risk. Two, and this is the real prize, the measurement itself: we now know exactly where the money goes, which means we know that the only way to grow the saving is a faithfulness project on read and search digestion, where the volume is. That is a real project with a real risk (lossy-digesting code is dangerous), gated on the same per-task validation we used for bash. It is the honest next step, not a number we can claim today.
Where it works: bash-heavy batch sessions, as a safety valve, with full recovery of the original output always available. Where it does not: anything latency-sensitive (the local decode adds a few seconds), and the budget at large, because the digestible bash volume is small and the harness already captures most of it. The upgrade path is faithful read and search digestion, validated before it is trusted.
The fun of a project like this is in the traps. These are real, and grounded in the work.
The empty-output trap. Ollama's OpenAI-compatible endpoint silently ignores a per-request context size. The harness read the model's trained context of 262,144 tokens, Ollama's actual default was 4,096, so it quietly truncated and returned nothing at all. The model worked in isolation and produced empty output through the harness. The fix was to bake the real context window into the model definition itself rather than pass it per request. This is the cousin of the thinking-model trap, and the reason we chose a non-thinking model: two different roads to the same blank reply.
The memory model that was never local. The config pointed memory operations at a small ONNX model and had for a while, so on paper that work already ran locally and for free. It did not. That model is disabled in the current harness version (a broken operator in the runtime), so memory had been silently falling back to remote the whole time. "We already do this locally" was false, and only checking it surfaced that. We swapped in a model that loads.
The harness already ate the easy savings. Before our hook sees a tool result, the harness compacts context, protects the newest 40,000 tokens, and minimizes shell output. A 19.6 KB git log --stat arrived under 8 KB, so our hook correctly skipped it. Over the window the harness had already shaken 560 bash outputs down to pointers. Respect what your tools already do, or you will proudly reinvent it and measure a phantom win.
A model that vanished mid-build. The sharpest taste of the problem was first-hand. One of the milestone agents was wired to a remote model id that returned a 404, not found. The subagent did not degrade or retry, it died on spawn, and we re-ran that milestone on a working model. A model that is simply gone is the most extreme version of the availability pain the whole project was aimed at, and it happened to us while we were aiming at it.
The bill tasted its own medicine. The project ran into the exact friction it set out to study. The 23-day window logged three rate-limit exhaustion events plus one warning, with the five-hour quota window hitting 100% on the two heaviest days. The cost and the limit pressure are the same mass, repeated payload, on the same days. And the milestones of this very effort added about $30 of new spend, all of it remote, because the hook was correctly still off. We were paying the problem while we measured it.
The thing I find most interesting is not the hook. It is that the company improved a piece of itself without anyone losing control of it.
The orchestrator did not do the work. It fanned the work out to department-head agents, milestone by milestone, with a hard gate at the front: the baseline report had to exist before anything could route or train. Every shipped milestone carries a signed decision and a thread you can replay, recorded in a single records-office daemon that is the sole writer of the company's memory. Knowledge is append-only; a revised finding supersedes the old one rather than overwriting it, so the trail of what the company believed, including the wrong first assumption, is intact.
The constraint that made me trust it was the boundary. The agents doing this work are scoped to the repo. The routing and the hook all live inside the repo, project-local. Nothing was written to the harness's own machine-local config outside that boundary, by policy, because crossing into a human's machine configuration is the kind of step an autonomous agent should not take on its own. When the work brushed against that line, it stayed inside and noted it, rather than reaching across. Self-improvement is easy to demo and hard to keep safe. The brakes (a measurement gate, signed decisions, an append-only record, a folder boundary the agents respect) are what make it something you would run.
If you care about whether agents leave a checkable trail, that posture, verifiable decisions and honest limits over confident claims, is the same one we apply to the products we ship, including metaend Grade, our agent-readiness scanner. The pattern travels.
Four lessons, all boring, all the point. Instrument before you optimize: the baseline was the most valuable artifact here, and it killed the original plan in the first milestone, before we could spend weeks routing a tier that did not exist. Make the optimization fail-safe and off by default: this layer is reversible, self-disabling, and all-remote until you opt in, which is why we can ship it without a knot in the stomach no matter how small the saving. Respect what your tools already do: half our theoretical win was already captured by the harness, and measuring that saved us from claiming it twice. And correctness over cleverness: the clever version digests read output too and shows a bigger number, the correct version leaves code verbatim until a faithfulness project earns the right to touch it. We shipped the correct version and wrote down the path to the bigger one.
The result is not a headline cost cut. It is a proven, reusable mechanism, a fail-safe pressure valve, and a precise map of where the money goes. For a last tune-up before the real work begins, knowing the territory is worth more than a small saving we would have had to oversell.
I write about agent infrastructure, local inference, and privacy-first systems as metaend. If you want to reach me, build on any of this, or tell me where it breaks, the door is here:
More writing lives at paragraph.com/@metaend