benchmaxxingnoun
Benchmaxxing is the practice of aggressively fine-tuning, prompt-engineering, or overfitting a language model specifically to top public benchmark leaderboards (such as MMLU, GSM8k, or SWE-bench). This often produces artificially inflated evaluation scores that look impressive on paper but fail to reflect genuine, generalizable real-world capability.
Definition of benchmaxxing according to Gemini 3.8 Flash.1Why Gemini? That will become clear soon.
Over the past two years or so, anytime an impressive new model with great benchmark numbers is released, the online discussion on reddit or HN eventually brings up the dreaded “b” word: benchmaxxed.2Why (and how) labs benchmax is a whole other story, and one I deliberately stay out of. My best guess is a mix of (i) researchers hoping that by training on tasks similar to the ones in the benchmarks, the models will perform better on similar everyday tasks, and (ii) researchers wanting to show leadership that their models keep getting better.
In this post, I use statistical analysis to test this accusation. The main premise is simple: models can’t be maxxed toward a benchmark that didn’t exist at the time of their release. In statistics speak, benchmarks released after a model are a natural control group. So, all we need to check is whether a model does significantly better on older benchmarks than on newer ones (after accounting for how capable the model is and how hard each benchmark is).
I build on the Epoch Capabilities Index (ECI); it’s a statistical model that estimates each LLM’s capability collectively from its scores on multiple benchmarks. I add a single parameter to it: a benchmaxxing coefficient, which captures whether a model overperforms on benchmarks released before it, relative to benchmarks released after it.
Just here for the results? Skip the equations and go straight to who’s benchmaxxing.
Capability minus difficulty
The idea behind ECI is quite straightforward. Each model has a capability score, and each benchmark has a difficulty score. In symbols:
The score of model m on benchmark b equals sigma of alpha b times the quantity C m minus D b.
Hover over any symbol to see what it means.
Tap any symbol to see what it means.
The S-shaped function σ turns the gap between capability and difficulty into a score between 0 and 1.
This way, a model with a capability score significantly higher than a benchmark’s difficulty aces it, and vice versa. The slope α sets how abruptly one turns into the other; the idea being that some benchmarks saturate more abruptly than others. This statistical model is fit to every available model–benchmark pairing, and this results in a capability score for each model and a difficulty for every benchmark, all on a shared scale.
One extra term
To measure benchmaxxing, I just add a single term: the benchmaxxing coefficient, β, which switches on only for benchmarks that came out before the model did.
The score of model m on benchmark b equals sigma of alpha b times the quantity C m minus D b plus beta of the model's lab times an indicator that benchmark b was released before model m.
Hover over any symbol to see what it means. The shaded part is the new term.
Tap any symbol to see what it means. The shaded part is the new term.
Because β gets added to the model’s capability score, it is measured in the same units. Basically, β is a capability bonus that a model seems to have, but only on benchmarks that predate it.
One more note: Epoch’s dataset is too sparse to estimate β separately for every model, so the coefficient is fit at the organization level. That is, there is one coefficient per lab, and the coefficient is the same for all the models from that lab. I denote that by βℓ(m). This way, the coefficient represents a lab’s average tendency to benchmax.
If anyone wants to repeat the analysis, the code is here.
So, who’s benchmaxxing?
Here are the fitted coefficients, one per lab,3Shown: 11 labs whose β has a standard error below 2.5 points. Seven labs with only a handful of models are in the fit but are not shown, as their intervals are too wide. Google DeepMind also includes Gemma. with 95% confidence intervals, sorted by β.
Select a lab to see its exact estimate.
Each lab’s β, with 95% confidence intervals.
So, what does this mean?
The first take is that Google’s models have a noticeably funny behavior where they tend to do better on benchmarks that are released before the models. So, yeah...
The second is that, although the signal is there, it is not very strong: 1.41 ECI points, less than the uncertainty Epoch puts on any single model’s score. It is not like Google models consistently saturate benchmarks that have been released before them. Instead, it’s more like a small, uniform effect, which appears only after averaging tens of benchmarks for tens of models.4See the twin analysis in Appendix C, where I compare two models with similar capability scores but with different release dates. For instance, Gemini 3.6 Flash beats GPT-5.2 on three of the four benchmarks released after it, and trails it on half of the older ones. The effect only appears when a lot of numbers are pooled at once.
How much would that matter?
Take the most suspicious reading, where all of Google’s β is benchmaxxing. Then, its 35 models would lose 1.3 ECI points on average. Gemini 4.0 Argon would drop from 4th to 6th place (below Fable 5.1 and Sonnet 5.5), and 3.8 Flash would drop from 20th place to 29th.
When I apply the adjustment to the benchmarks in Google’s own launch tables, the ECI change corresponds to about 3 to 5 points per major benchmark: Gemini 3.8 Flash’s score on DeepSWE v1.1 would drop from 73.7 to 69.3. Alibaba’s and xAI’s models would fall by about an ECI point, Anthropic’s by about half a point, and OpenAI’s would barely move. For more details, see the appendices.
For anyone who leans heavily on LLMs in day-to-day work, the results may not come as a big surprise.
Stress testing the fit
“You come at the king, you best not miss” (or something like that)
I wanted to make sure that the analysis was not off. I repeated the exercise under a range of alternative specifications (Part 1), and then played referee to my own result, with fifteen objections a careful statistician could raise and the analysis that answers each (Part 2).
Sadly, Google’s pattern survives almost all of them.
BUT, they offer some other ways to read what’s happening here:
- A curve, not a step. Google’s models do best on benchmarks from about a year before their release, and worse on both older and newer ones. Instead of a performance jump at the release date, a smooth curve fitted for each lab over the benchmark age actually gives a better fit (see Part 2, Section 2.10). That would happen if models are tuned to the benchmarks of the last 1-2 years, which is a kind of benchmaxxing.
- A weakness on agentic work. The newest benchmarks mostly have the model use tools: 15 of the 20 released in 2026 are agentic or tool-using. On these, Google’s models score about 3 ECI points below other labs’ models of the same ECI, more than any other lab once the math benchmarks with tool use are counted in. That accounts for most of Google’s decline on newer benchmarks and about a third of its β lead over the other labs. That being said, with the strictly agentic benchmarks left out, Google’s pattern remains.5On the strictly agentic benchmarks, OpenAI, Alibaba and DeepSeek fall short by as much as Google.
My personal view on Google models
I heavily rely on Claude and GPT for my work. I occasionally give a coding task to the latest Gemini model, and for the last 1.5 years, I’ve almost always walked away disappointed. Don’t get me wrong, Google had some gems that I really enjoyed working with. The first post on this blog is basically a love letter to Gemini 2.5 Pro6To be clear: it was the Gemini 2.5 Pro (March) version, not the 2.5 Pro that later came out in May and took away all the good things that model had.. Gemma 2 was the first local model that I actually ended up using regularly, and Gemma 4 (31B) was the first local model that had a “big model” smell. I truly think that Google can do great, but their performance in the last year has been pretty lackluster.
These findings explain some of that. Here is to wishing that Gemini 4.0 Argon can actually deliver.
AI disclaimer: I heavily relied on Opus 5.5 and Fable 5.1 for editing this post. The final drafts of the appendices and the additional parts are all Opus 5.5. The main post is 99% me.
Appendix A: How far would the scores fall?
If all of Google’s β were benchmaxxing, its models’ ECI would fall by 1.3 points on average.
A coefficient is abstract, so here is what it means for the index itself. Take the most suspicious reading: all of a lab’s β is benchmaxxing, and none of it is anything else. Then a model’s fair capability is the one it gets when β is part of the fit, and what matters is how far that moves the lab’s ECI scores, on average over its models. Since a part of β may well be something other than benchmaxxing, read these falls as the most it could be worth.7The most at the estimated β, that is; the intervals leave room for more and for less.
For Google’s 35 models, the average fall is 1.3 ECI points. It is not the same for every model. A model scored only on benchmarks that predate it loses the whole of β, or a bit more near the top of the scale;8All scores stay on Epoch’s scale, which fixes Claude 3.5 Sonnet at 130 and GPT-5 at 150, so a fall is measured against those two models. They move too: OpenAI’s β is negative and Anthropic’s is close to zero, so allowing for β lifts GPT-5 by about two thirds of a point against its benchmarks and leaves Claude 3.5 Sonnet about where it was. Since the scale holds them at 150 and 130, the strongest models lose a little extra, which is why the newest Gemini Flash models fall by 2.0 rather than 1.41. a model that was also scored on benchmarks released after it loses less, because those scores pin down its capability. Gemini 3.7 Flash and Gemini 3.8 Flash, the two newest Gemini Flash models in the data, are scored almost only on older benchmarks and lose 2.0 points each. Among the 278 models in my fit, that moves them from 18th and 20th place to 27th and 29th. At the other end, Gemini 2.0 Flash (December 2024), scored on two benchmarks on each side of its release, loses 0.2. Alibaba’s and xAI’s models would fall by about a point, Anthropic’s by about half a point, and OpenAI’s barely move.
Appendix B: The launch tables, without β
Google’s launch scores for Gemini 3.7 Flash and 3.8 Flash, without β: about 3 to 5 points lower per benchmark.
The same arithmetic applies to the numbers labs print in their release posts. Take a score Google reported for a Gemini model on a benchmark that was public before the model came out. If all of Google’s β were benchmaxxing, then without it the model would sit β capability points lower on that benchmark’s S-curve. I take Google’s number, move it down the curve by that much, using the benchmark’s own slope α from the fit, and read off the score. Here are the two most recent Gemini Flash launches in Epoch’s data, Gemini 3.7 Flash and Gemini 3.8 Flash, with every benchmark from Google’s tables that is also part of the ECI, in the same version and with the same metric.9That rule leaves out most of each table. Terminal-Bench 2.1 (the ECI uses 2.0, and 28 of its 89 tasks were fixed in 2.1), HLE-Verified (a revised set of questions), OSWorld 2.0 (Google reports a partial score; Epoch counts only fully completed tasks) and GDPval-AA (an Elo rating; the ECI uses a win rate) are near misses. The rest, including SWE-Bench Pro, Terminal-Bench 3.0 and 4.0, CharXiv Reasoning and LVBench, are not in the ECI at all.
| Benchmark | Public since | Google reported | Without β | Epoch’s record |
|---|---|---|---|---|
| FrontierCode 1.1 (Main) | June 2026 | 43.6 | 40.7 (−2.9) | 43.6 |
| DeepSWE v1.1 | May 2026 | 65.3 | 60.3 (−5.0) | 65.5 |
| Humanity’s Last Exam* | January 2025 | 45.7 | 42.1 (−3.6) | none |
| Benchmark | Public since | Google reported | Without β | Epoch’s record |
|---|---|---|---|---|
| DeepSWE v1.1 | May 2026 | 73.7 | 69.3 (−4.4) | 73.8 |
| Humanity’s Last Exam* | January 2025 | 45.4 | 41.8 (−3.6) | 41.7 |
Three to five points per benchmark is more than the 2.0 points each model loses on the ECI, and that is not a contradiction: benchmark scores and the ECI are on different scales. These benchmarks are steep, and the scores sit in the middle of their S-curves, where a small shift in capability moves the score the most. Epoch’s records agree with Google’s numbers on FrontierCode and DeepSWE to within 0.2 points. On Humanity’s Last Exam, Epoch’s own run of Gemini 3.8 Flash scored 41.7, close to my 41.8, but since Google does not say how it ran the benchmark, I would not read much into that.
Appendix C: Twins
Equally capable models from different labs, benchmark by benchmark: the same direction as β, but within the noise.
The coefficient comes out of a statistical model, and a model brings assumptions with it. Here is a check that needs almost none. Take two models from different labs with the same ECI, released six to nine months apart. The same capability means the same expected score on every benchmark, so which twin does better on a given benchmark should have nothing to do with when that benchmark came out. If one twin was tuned toward the benchmarks that existed when it was built, it should beat the other on those, and not on the benchmarks that appeared after its release.
Here is one such pair. Gemini 3.6 Flash (ECI 154.2, released July 2026) and GPT-5.2 (ECI 153.5, released December 2025) came out seven months apart, and Epoch scored both on the same 14 benchmarks. Ten of those were public when Gemini 3.6 Flash was released; four appeared later.
Public when Gemini 3.6 Flash was releasedReleased after Gemini 3.6 Flash
Hover over or tap a benchmark to see both scores.
On the ten benchmarks that predate it, Gemini 3.6 Flash scores 2.4 points higher than its twin on average; on the four that came after it, 1.2 points lower. That is the pattern benchmaxxing predicts. But look at the spread: single benchmarks range from 21 points in GPT-5.2’s favor to 29 in Gemini’s, and the largest gap is on SimpleQA Verified, a benchmark built by researchers at Google DeepMind and Google Research.10Haas et al., SimpleQA Verified (2025). A lab doing well on a benchmark its own researchers built is a story of its own, and the coefficient cannot tell it apart from benchmaxxing. Dropping SimpleQA Verified, DeepSWE and ARC-AGI-2 together, the three benchmarks that matter most to Google’s coefficient, shrinks it to about 1 ECI point (Part 2, Section 2.6). One pair is an anecdote.
So I did it for every pair. Among the models with a published ECI, 83 pairs consist of a Google model and a model from another lab within 2 ECI points of it, released 6 to 9 months apart. Another 257 such pairs contain no Google model, and they are the control. For each pair I take the advantage of the focal model (the Google model; in control pairs, either twin) over its twin on benchmarks that predate the focal model, minus its advantage on benchmarks that came out after it. That needs benchmarks on both sides, which leaves 51 of the Google pairs.
| Focal model | Pairs | Older minus newer (points) |
|---|---|---|
| Google model, the later twin | 21 | +3.7 ± 3.6 |
| Control: the later twin | 107 | +0.9 ± 1.2 |
| Google beyond the control | +2.7 ± 3.8 | |
| Google model, the earlier twin | 30 | +1.4 ± 1.5 |
| Control: the earlier twin | 178 | −0.8 ± 0.8 |
| Google beyond the control | +2.2 ± 1.7 |
Two things stand out. First, the control pairs hint that the later twin does better on older benchmarks even when Google is not in the pair: at the same ECI, newer models may lean toward older benchmarks, whoever made them. That is why the comparison needs a control. Second, Google’s models go the way the coefficient says in both orientations, by 2.7 and 2.2 points beyond the control. But the standard errors are 3.8 and 1.7, so neither is more than 1.3 standard errors from zero. The twins point the same way as β; on their own, they cannot tell it from noise.
The long versions
This post is the short version. Two companion posts hold everything else. Both were written by Opus 5.5, from the analyses behind this post.