Coding with AI has moved through a few rapid stages of evolution:
- 2023-2024: AI-powered tab-complete
- 2025-2026: iterative coding agents / agents as pets
- 2026 onward: ?
We can now produce code faster than ever, but we're faced with... a whole lot more code to review! Worst of all, a whole lot of it is slop.
I've started to think of programming of late as holding AI on a leash. Pull too hard and you may as well write the code yourself. Let go entirely and you're drowning in sloppy code that you don't meaningfully understand.
How do you define slop?
I recently came across a definition for AI slop that I liked on Simon Willison’s blog:
Slop is something that takes more human effort to consume than it took to produce.
Slop is what you get when a system optimizes for "looks done" or passing superficial tests, but the underlying artifact is messier than you intended.
It's not as though slop doesn't accumulate in human-generated code, but humans perform code reviews to find code that will be hard to maintain. In the simplest sense: the reviewer tries to understand what the code is doing and how. If the reviewer can't, the code is hard to maintain and should be simplified.
Human review keeps slop low and code manageable. It charges the cost of consuming code up front, before that code becomes someone else's maintenance problem.
But as much as I love simplifying code, human review doesn't scale to LLM volumes. At this point, my coding agents produce code much faster than I can review it.
And while unit tests help loosen the leash, deterministic checks can't catch every subjective dimension, including design, quality standards, and complexity.
But is verifying against subjective constraints fundamentally impossible? Or can we make the system pay the cost of consuming its own work?
We think the answer may actually be simple: add an adversarial coding agent into the mix.
The unreasonable effectiveness of adversarial self-play
Self-play is hardly a new idea — it's how AI got past imitating humans in the first place. AlphaGo learned from human games, then improved by playing against itself; AlphaGo Zero dropped human data entirely and beat its champion-beating predecessor 100–0.
In our implementation, the harness takes a goal specification as input and runs two coding agents in a loop: one generates code; the other reviews and tests it. Both are aligned to the same goal spec but play adversarial roles. The loop ends when the verifier concedes or the review budget runs out.
The verifier can receive written rubrics in the form of skills that tell it what to grade against. The sharper the rubric, the better the outputs.
Despite its simplicity, there are a few reasons that this is effective:
-
First, using naive self-evaluation fails. A single agent with shared context doesn't do as good a job as an independent verifier. Why is this? Per Anthropic's own analysis, tuning a standalone verifier to be skeptical is far more tractable than making a generator self-critical. Self-evaluation also runs the risk of getting stuck in a local maximum, whereas self-play can elicit more dramatic turns.
-
Generation is a fundamentally hard and unforgiving search problem. Verification is simpler. OpenAI studied this asymmetry in their work on prover-verifier games, where they explicitly used weaker models as verifiers for math proofs. Their results also improved "legibility" of LLM outputs, not unlike our claims on code maintainability!
-
Lastly, adversarial evaluation introduces back-pressure into the generation loop. Over long runs, that makes a Telos loop more stable than a Ralph Wiggum loop, which falls into the same failure modes of self-criticism. Stable loops let you spend more test-time compute productively.
Measuring results in the real world with SlopCodeBench
We've defined slop qualitatively so far. Earlier this year, the folks at SprocketLab released a benchmark that measures it, which we ran our harness against.
SlopCodeBench has coding agents implement various checkpoints of a natural language spec. By making agents extend their own prior work, this benchmark measures metrics like "verbosity" and "code erosion" using its own heuristics.
By introducing the notion of spec checkpoints, the benchmark's design focuses on iterative evaluation (vs one-shot, SWE-bench style), which provides a good proxy for slop mechanics as coding agents attempt to extend real codebases.
In our evals, we selected a representative subset of five problems to evaluate against and ran two trials per arm, using gpt-5.5-high for the presented results. The system was allowed to do up to 5 passes on the loop. We gave the harness a minimal "slop rubric" describing relatively generic code quality principles.
Here are some of the results we saw:
The loop delivered simpler code
In the benchmark, "erosion" measures the share of code mass sitting in high-complexity functions, which serves as a proxy for how expensive code
is to consume. Telos came in below the one-shot baseline on four of five
problems — ~12% lower on average, up to −32% on cfgpipe.
Code erosion is but one proxy for slop. In our human review of the results, we saw that adversarial pressure forced coding agents to improve other markers of slop like overdone comments, dead code across checkpoints, and vacuous unit tests.
Verification went beyond the test suite
In a few instances, the verifier refused checkpoints that scored >95% on the benchmark's own hidden tests, backing each refusal with multiple functional bugs that the tests missed.
The loop also persisted 46 executable probes into the final codebase, with at least ten of them enforcing spec clauses the benchmark's hidden tests don't cover. Spec coverage grew while the agent worked, even with the hidden tests already saturated.
As a result, strict pass rate matched or beat the baseline on four of five problems, improving by as much as +15 points (cfgpipe).
Keeping a software contract true over time
The same loop, run over a long enough horizon, stops being a review tool and becomes a maintenance mechanism.
The benchmark also re-runs every earlier checkpoint's tests as requirements compound — and this is where one-shot and Telos diverge hardest: By checkpoint three, the one-shot agent on file_backup passed only 16% of the earlier checkpoints' tests; Telos passed 100% at every checkpoint. Across the other problems, the loop's regression pass rate typically stayed above 90%.
Without a durable specification, one-shot agents risk drifting away from the broader goal. The adversarial loop gives the system a way to pull itself back toward its desired state.
Do you still need verification as models get better?
Coding models continue to get better by the month, but there are still some important problems that our harness design solves:
- Smarter models reward hack more, especially over long horizons.
- For the same problem, coding models exhibit variance in one-shot performance across multiple runs.
- Teams have standards or practices that need to be followed 100% of the time.
We believe our adversarial loop solves some of the problems above, enabling "goal alignment" over long horizons. This arguably becomes more important as models continue to get smarter and more autonomous!
Goals as an abstraction
Today you can run coding agents to execute your long-horizon goals, but primarily as a workflow in your day-to-day process. Workflows like spec-driven development are useful but require discipline to adhere to them.
An abstraction, on the other hand, hides away low-level details and creates more leverage for the end user, but usually requires real guarantees to be met.
In our work, we're most excited by the possibility of creating a stable abstraction (the goal spec) over a non-deterministic process. There's a large difference between the autonomy of a system that is "correct 92% of the time" and one that is "correct 99% of the time." One is guaranteed to blow up in your face at some point, and the other can be trusted.
This abstraction lets us treat the spec as a durable artifact and code as the disposable implementation, much like a traditional compiler.
Real software is a living thing. Requirements change, edge cases surface, and each iteration creates another chance for the implementation to drift. The hard part isn't generating code once; it's managing software as its specification evolves.
That's what we're building Telos toward: software that manages itself, starting with the codebase.
Try it out
We've open sourced the Telos harness for folks to try out. It builds upon the excellent Pi harness.
Install with:
curl -fsSL https://usetelos.ai/install.sh | sh
Write a spec, optionally with Claude or Codex:
---
version: 0.1.0
name: hello-service
skills:
- my-design-guide
- my-deployment-process*
---
# Goal
Build a small hello world HTTP service with `/healthz`, tests, and local run instructions.
Note: starred (*) skills are adversarially verified against, enforcing your standards and rubrics.
Kick off a bounded run:
telos run SPEC.md --until 5
Or run it as a persistent process:
telos apply SPEC.md
If run as a persistent process, you can edit the spec after interacting with the output, and rerun telos apply to reconcile in the background. No chat-boxes or one-off prompts, just a declarative specification.
We're also building a managed cloud runtime for Telos. Try it out by signing in on the web or via the CLI (telos login).
Read more in the Telos documentation.
If this post resonated with you, we'd love to hear from you! Please reach out at [email protected].