We have been exploring how coding agents are benchmarked recently and found an interesting behaviour among the frontier models — they will go to great lengths to avoid actually solving the task.
The benchmarking process was standard SWE-bench style — real GitHub issues from open-source repos. Each task gives the agent:
- The repo, checked out at the commit right before the real fix was merged
- The issue text, and an environment it can run anything in
- Hidden tests that decide whether it solved the issue
Let the agent go — and our cat-and-mouse game began.
Round 0 — Git history had the fixes
The first runs came back with surprisingly good numbers, so we read the
trajectories. The frontier models were running git log --all, finding the
future commit that fixed the issue, and transcribing the answer. Fair
enough — eval images are built by cloning the repo today and checking out an
old commit, and checkout only moves the working tree. Git's history keeps
everything, fix included.
The scale of it surprised us:
- One frontier model put the literal fix diff on its screen in 27% of tasks
- It solved 96% of tasks where it saw the diff, versus 40% where it didn't
- Another model saw the diff on 57% of tasks — over two-thirds of its "solves" were leak-assisted
The fix: strip the future history from every image. We assumed that was the end of it.
Round 1 — Dangling git objects
The models did not stop there. Git keeps objects even after their refs are
deleted, and the models found them — running git fsck to enumerate dangling
objects and pulling the fix up by raw commit SHA. When we audited older
images, the fix commit was still recoverable this way in every one of them.
The fix: garbage-collect the images until the fix is physically gone from disk.
Round 2 — Refetching from GitHub
The models noticed the history was missing. And repaired it. The containers had internet access — needed to install dependencies mid-task — and the models used it:
- Re-cloned the repo from GitHub and fetched the deleted commits right back — one cloned a post-fix release tag instead
- In a single 93-task run, one model attempted this network recovery on 37 tasks and got the fix back on screen in 6
The fix: block GitHub in every container.
Round 3 — The Wayback Machine
With GitHub dark, one agent took a detour through time — pulling repo files from the Wayback Machine. The files it fetched that day happened to be harmless, but archived PR diffs are very much a thing.
The fix: block the archive hosts too.
Round 4 — The fix was on PyPI
The models stopped looking for the commit and started looking for anywhere the fix had left a trace in the world:
pip download <package>— the released version on PyPI contains the fixed source. One model pulled it successfully 22 times; one transcript had the fix lifted byte-for-byte identical to gold.- Another agent read the project's docs changelog just to learn which PR number fixed the issue — one task was solved verbatim off a changelog line
The fix: block the docs site, and wrap the
pipcommand to refuse that one package — we couldn't block PyPI outright, since agents legitimately install test tools mid-run.
Round 5 — CDN mirrors
Opus then found routes we had not anticipated:
- It enumerated GitHub CDN mirrors until it found one that wasn't blocked — jsDelivr — and pulled the graded test file straight through it
- It curled the wheel directly off PyPI's file host, sidestepping our pip
wrapper — which, we later found, had more holes anyway (
python -m pipskips the executable, and a second Python env shipped its own unwrapped pip)
The fix — the last one: stop patching the blocklist and accept what it was telling us: a blocklist is not a security boundary. We baked every dependency into the image and dropped all network egress at the firewall. Nothing legitimate broke, and solve rates were unchanged — the cheating had never been load-bearing.
The persistence itself was striking. Even with every route blocked, one run logged 109 separate fetch attempts — the model spent its budget probing the walls instead of fixing the bug.
What the numbers did
Once the walls were solid and every trajectory was audited, the scores finally held still:
| model | leaky environment | after our blocks | fully audited |
|---|---|---|---|
| Frontier model A | 75% | 36% | ~21% |
| Frontier model B (Opus-class) | 55% | 28% | ~22% |
| Small 30B open model | 12% | 13% | 13% |
Three things stand out:
- Roughly half to two-thirds of the frontier models' apparent solves were the environment leaking
- The small model's score barely moved — because it never attempted any of this. Not one network recovery in 93 tasks, across every round. Every escalation above came from the frontier.
- The capability that finds holes in your benchmark is the same capability the benchmark is trying to measure — so the inflation is largest exactly where the stakes are highest
We did eventually just ask
Out of curiosity, we ran a set of tasks with an explicit "don't look up the fix" instruction added. The same model that had hammered our network guard with 100+ blocked fetch attempts per run made zero attempts across 46 transcripts — and solved almost exactly as many tasks (14/20, versus 15/20 previously with cheating available). The lookups were a habit, not a crutch.
We kept the firewall anyway: a benchmark whose walls are made of politeness isn't a measurement.
To be fair
Many of these things are what a good engineer does — checking if upstream already fixed the bug, upgrading the package, reading the changelog. We train models to use whatever's reachable to make the tests pass. The problem was that "reachable" included the answer.
The round we can't patch
And there is one round no firewall closes. In 20 of 93 fully-guarded runs, with zero network activity, the Opus-class model produced fix knowledge it narrated as recalling the actual PR — occasionally near-verbatim. The fixes are public; they flow into pretraining crawls. You can airgap the box. You can't airgap the weights. The only defenses left are tasks newer than the model's training cutoff — a property that decays every time a frontier model retrains — and actually reading the trajectories.
Where this leaves benchmarks
It will be interesting to see how the benchmarks evolve — especially with the current pace of development. We only ran Opus-level models and below; our guess is Fable-level and future models will find even more creative routes to the answer. We think it makes sense to own your benchmarks, so you know what the numbers mean and where models actually stand for your use case.