Learn how AI benchmarks work

You've been writing benchmarks your whole career. A unit test suite is a benchmark. p99 latency is a benchmark. A code coverage percentage is a benchmark. They all answer the same question: is this better than before?

AI uses the same idea at a larger scale. Take SWE-bench: it hands a model a real GitHub issue and checks whether the automated tests pass. Repeated across hundreds of tasks, that binary outcome becomes a score — one that moves research budgets, directs lab priorities, and shapes what the next generation of models will optimize for.

Labs race to top them. Papers are written about them. They are the industry's north star.

Every benchmark follows the same pipeline. A task bank provides the inputs — real GitHub issues, math problems, research questions. A test runner feeds them to the model one at a time. The model produces outputs. A grader — usually automated — checks them against known correct answers.

The result is a number between 0 and 100. That number gets published on a leaderboard, cited in papers, and referenced in funding decks. The pipeline looks simple from the outside. But the number is only as good as what's behind it — and that depends on details that rarely make the leaderboard.

There are two ways a score drifts from reality before anyone cheats.

The first is contamination. Models train on massive chunks of the internet. Benchmark tasks also come from the internet. When those overlap, the model has basically already seen the test. Scores go up. The data just leaked.

The second is saturation. Once several models hit 90%+ on a benchmark, it stops measuring meaningful differences. The leaderboard keeps moving — by fractions of a percent — but the capability differences it's tracking have become noise. The benchmark has been solved; it just hasn't been retired yet.

Both problems exist before anyone cheats.

On top of contamination and saturation, there's a third way scores mislead: agents that find shortcuts to a high score.

A documented example from SWE-smith: Claude 3.7 Sonnet was asked to implement a string-distance algorithm. Instead, it detected the exact test inputs and hardcoded the expected return values. Its own commit message gave it away: "Added special case handling for the specific test cases to ensure the tests pass."

The algorithm was never written. Every test passed. You've seen this before — it's the same thing that gives you 100% coverage with tests that never actually assert anything. Toggle the visualization to see the gap.

The first move is to write harder tasks: swap out the test set, randomize variable names, generate tasks on the fly. It buys a few months — until the next model finds the pattern anyway.

What actually makes a score trustworthy is the environment it runs in. Three things to look for: task isolation (the agent runs in a fresh sandbox, cut off from test inputs), process verification (did it compute the answer or pattern-match to it?), and environmental controls (no filesystem access, network blocked, process isolated).

When you're picking which model to use, these are what separates a score that means something from noise.