Cover photo

A cheap, sandboxed coding agent vs Claude: 18 of 18 each, 23x apart on cost

A six-task benchmark tied on solves, then showed where the price gap came from and why the result is only a starting point.

I wanted to know what a cheap coding agent could do on a real project folder without getting access to change it. So I put mini-swe-agent in a throwaway rootless Podman container, pointed it at DeepSeek V4.1 Flash through NanoGPT, and compared it with the Claude harness I use day to day.

Both solved all 18 attempts. The mini-swe-agent setup cost $0.1918. Claude cost $4.4948. On this test, Claude cost about 23 times as much per solved attempt.

That is a result, not a verdict. The tasks were too easy to separate the agents. Both hit 18 of 18, which makes this a ceiling-effect benchmark. The useful part is the setup, what broke along the way, and what the run did and did not measure.

The setup

The stack was mini-swe-agent 2.4.6, LiteLLM 1.108.1, deepseek/deepseek-v4.1-flash-1017 through NanoGPT, and rootless Podman 6.1.4. The mini-swe-agent model name was nano-gpt/deepseek/deepseek-v4.1-flash-1017.

I installed the agent with its MCP dependency:

uv tool install mini-swe-agent==2.4.6 --with mcp

The test image was built from python:3.12-bookworm, with /testbed created in the image and pytest==8.4.2 installed. The agent config used Podman as the executable, localhost/mini-testbed:py312 as the image, /testbed as its working directory, and the NanoGPT model name above.

The folder launcher mounts the current directory into the container as a temporary overlay. The directory is visible to the agent, but its writes go into a separate upper layer rather than changing the source folder. Secret files such as .env and *.pem are shadowed with /dev/null. The launcher refuses risky paths, including home or /, paths containing : or ,, and folders inside git-crypt repositories.

That is the part I wanted to test in practice: a cheap agent can work against a real project tree while the host copy stays untouched. The container is disposable, and the source directory is not the agent's writable working copy.

Four things that broke

The first failure was ModuleNotFoundError: No module named 'mcp'. LiteLLM's MCP handler needed the package, so the fix was to install mini-swe-agent with --with mcp.

The second was harder to spot. Tool calls were silently stripped because the NanoGPT provider metadata in LiteLLM reported that the model did not support function calling. The model registry entry needed "supports_function_calling": true.

The third was Podman refusing to start because /testbed did not exist in the image. Creating that directory fixed it. The configured working directory has to exist inside the container, not only on the host.

The fourth was the cost number. NanoGPT's displayed per-request cost did not match the token prices I had expected. I stopped estimating from the advertised rates and summed the provider's reported usage.cost from the actual runs.

The launcher exports NANOGPT_API_KEY and sets MSWEA_CONFIGURED=true. These are small fixes, but each one took time to locate. A working model endpoint is not enough; the agent, provider adapter, and container runtime all have to agree about the interface.

The six tasks

I wrote six small Python tasks with hidden tests that neither contestant saw. The graders ran offline in the same container image.

The tasks covered a pagination off-by-one, a duration parser, an LRU cache, splitting a bill, nested environment-variable overrides, and a misleading report formatter issue. They were synthetic and intentionally bounded. That made the outcomes easy to grade, but also made them too easy to distinguish the agents.

Contestant A was mini-swe-agent with DeepSeek V4.1 Flash on NanoGPT, running non-interactively with a step limit of 250. Contestant B was my day-to-day harness with claude-opus-5-5, high thinking level, a fresh subagent, file read/edit tools, and parallel tool calls.

Both passed all 18 attempts: three runs per task. A cost $0.1918 in total; B cost $4.4948. The observed spend was about 23 times apart.

The rest of the measurements do not point in one direction. A used a median of nine model calls and took a median 64 seconds. B used six calls and took 28 seconds. Total input and output tokens were 1.93 million and 101,000 for A, versus 4.52 million and 47,000 for B. A added regression tests in 13 of 18 attempts; B did so in all 18.

The misleading formatter task also showed different approaches. DeepSeek added a local-date path; Claude moved the UTC conversion. A varied more in its effort on that task, while B was more consistent. Those are observations about these runs, not evidence that one approach is generally better.

What the numbers say, and what they do not

The cheap setup was much cheaper on this batch. It made more calls and took longer, while Claude used more input tokens and wrote regression tests more consistently. Since both agents solved every task, the benchmark cannot tell us which one is better at harder coding work.

There are several limits here. The sample is small. Claude helped write the tasks and was also one of the contestants, so the author and the experiment are the same system. This compares two agent setups, not two models in isolation. The cost figures come from different sources, and the wall-clock comparison is not like-for-like. NanoGPT's standard route is not confirmed to have zero data retention.

There is also a less obvious cost: authoring the benchmark cost $7.61 for the Claude agent, more than the $4.69 combined cost of contestant attempts. The benchmark itself was more expensive than running the comparison.

That matters because it changes what is worth optimizing. The interesting question is not whether a cheap model can win six toy tasks. It is whether the same container and harness can be reused for a harder, held-out set where both systems do not immediately score perfectly, and where benchmark construction is not quietly the largest bill.

The reusable part is the workflow: run a low-cost agent in a throwaway container, expose a real project folder without changing its host copy, keep secrets out of the container, and grade against tests the agent never saw. This run did not establish that the cheap setup is a Claude replacement. It established that I can run it safely enough to ask a better question next time.

The referral link for NanoGPT is nano-gpt.com/subscription/kUGJhgF9.

Contact the author

I write about coding agents, evaluation, and cost as metaend. More writing lives at paragraph.com/@metaend.

Written by metaend.