At tuneloop.io, we are working on offering the right level of model intelligence for the right SWE task, and writing about our findings along the way. In part 1, we mapped out the problem statement and noted that building the right evaluation framework is a core foundational component. In this post, we look at how current coding agent benchmarks and eval frameworks are designed, and what may be interesting next.
Anatomy of a benchmark task
Each task in a benchmark generally has the following components:
- A prompt/instruction for the agent with requirements
- an environment for the agent to work in: a container with dependencies installed and the base state, such as a git repo checked out at a commit
- verifiers / graders that can score the agent's output, which is typically a patch, or output files specified in the requirements. Ensuring that a set of tests pass is usually the first step, along with other checks on code quality that some benchmarks include.
Within this framework, different benchmark datasets have made different design choices, primarily along the following dimensions:
- How are tasks sourced?
- How do we score the agent's output?
Sourcing Tasks
Of course, the nice thing about coding is that there's open source data available. We have well maintained open source repositories, with issues and merged pull requests. So, in theory, we could implement an automated pipeline to create benchmark tasks, which would roughly:
- sample a bunch of repositories
- take merged PRs from these repositories
- write a task description, either from the linked issue or from looking at the code changes, and
- separate out the tests from the merged PR to create a verifier.
And this is exactly where we started with SWE-Bench, the OG in this space. Since then, benchmarks have evolved to address the following data quality issues:
Tasks can be under or over specified
A task is considered well-defined if the tests are such that they verify the exact functional behavior specified in the instructions, nothing more, nothing less, and regardless of any particular implementation. This is usually not true if tests are extracted as is from a pull request or commit.
SWE-Bench Pro addresses this problem by augmenting the instruction set with two fields: requirements and interface, both of which give enough information to guide the agent towards the implementation shape expected by associated tests.
Others like DeepSWE, Terminal-Bench and FrontierCode provide tests written from scratch by experts to meet the guidelines above. The trade-off is quality vs. quantity: you get a higher quality but smaller corpus.
In any case, this means that tasks have to go through a round of LLM-assisted manual curation to ensure that they are well specified. The nice thing is that these benchmarks are heavily scrutinized by the community as model providers inspect tasks and agent trajectories in detail to debug model performance issues, which results in newer versions with updated instructions and tests. (See SWE-bench-verified, and terminal-bench-2.1)
Task Contamination / Reward Hacking
Tasks are public, or are sourced from public repositories, which means that the model may have already seen the solution in its training data. There are a few approaches to mitigate this:
- SWE-Bench Pro is sourced from copyleft licensed repositories, or from private repositories.
- DeepSWE's tasks are written from scratch, with no upstream commits.
- Benchmark dataset webpages now include sentinel strings to make it easy to detect if these tasks appear in the training corpus.
The other problem is that frontier models are increasingly creative and find ways to cheat on benchmarks. This can include searching online for solutions, or even infiltrating another company! Typical safeguards include:
- Validation of agent trajectories: TerminalBench requires that agent trajectories be submitted to be included in the leaderboard, following incidents of cheating. See leaderboard integrity update.
- Basic scrubbing of agent environments: Erasing future git commits is one common method, as is seen in DeepSWE's task docker files.
- Just asking the agent to not cheat in the instructions: Terminal Bench 4.0's task instructions include the line: "Do not cheat by using online solutions or hints specific to this task." One can hope the agent overlords obey.
Benchmark Saturation
Benchmarks need to scale with model capabilities to continue to be useful indicators of progress. Typical approaches here have been:
- Moving towards long horizon tasks: These are complex tasks hand-created by experts. Cognition's FrontierCode, Terminal Bench, DeepSWE and SWE-Marathon all follow this path.
- Diverse, long tail tasks: DeepSWE sources from 91 different repositories covering 5 languages, SWE bench pro includes a hidden dataset that's sourced from private repositories.
Verifiers
This is the second dimension in which benchmarks differ. While almost every benchmark includes pass-to-pass and fail-to-pass tests, the differences lie in how these tests are constructed. Today, most benchmarks ship with manually written/edited tests that ensure that the tasks are well specified, as discussed above in the Task Specification section.
FrontierCode, SlopCodeBench and SWE-Marathon include other verifiers that test code quality (mergeability / maintainability) and computer use to test application behavior.
Worth highlighting is FrontierCode's adaptive grading, which also helps mitigate concerns with tests being overspecified.
Where do we go from here?
Building benchmarks that reflect interactive coding agent usage
Benchmark tasks today are one-shot, and implementation focused, which is different from most day-to-day usage in interactive sessions. We explore codebases, build plans, write and review code, iteratively. I think SlopCodeBench's notion of tasks made of checkpoints that build on top of each other is a good paradigm here, and so is Tau-Bench's user simulator. We need datasets built from agent traces, with verifiers that can judge intermediate outputs, in addition to the signal from the final generated code change.
Private Benchmarks
The other interesting question is on getting signal from the vast majority of private, enterprise code repositories. Companies like Uber, Ramp and Databricks have built their own benchmarks - I expect this trend to continue, and perhaps we will have some sort of aggregate number that gets reported from these private benchmarks as new models come online.
We are evaluating these directions, using a combination of public datasets and internal data, with the goal of being able to evaluate models and routing configurations on enterprise workloads. Next in this series: our first experiments and what we learn from them. If you're interested in us running these experiments on your data, reach out at founders@tuneloop.io.