harnesseval · September 2026 · the developer edition
Vibes vs. Evidence: What actually delivers AI code review quality
We measured which AI code review setups find real bugs, what they cost, and how long they take. Here is what to use and why.
What we measured. Eight models each reviewed the same pull requests three ways (as a single engineered prompt, and
running the open-source harnesses Compound Engineering and metareview) at three effort levels. Every review was
scored against 147 verified bugs on the six highest-severity pull requests, inside a campaign of 2,416 review runs
across a 50-PR public benchmark. How we kept score ↓
Disclosure. I wrote metareview (MRV),
one of the harnesses tested here. It is open source (MIT) and free. Every harness was judged by the same frozen instruments,
and the data and code to re-run the analysis are public.
Full disclosure and safeguards →
Don’t take our word for it. Pit any model, harness and effort level against each other across all 66
complete configurations: bugs found, noise, dollars and seconds.
A harness finds more real bugs than a single prompt
A single prompt (“one-shot”) gives the model the diff and a carefully written review prompt, one time. Read the exact prompt (REPORT §2.2). A harness makes the same
model work the review in passes, with tools and specialised reviewers. We tested the open-source harnesses
Compound Engineering (“CE” in the charts) and metareview (“MRV”). With the same model and effort level, the harness found more verified bugs in
39 of 42 comparisons, a mean gain of 13.5 percentage points of recall—the share of verified bugs a review found.
Verified bugs found, one-shot versus harness
Each row is a model; marks show the mean across the three effort levels, out of 147 verified bugs. Fable is omitted because its harness runs were not completed.
Every model moves right. The best harness configuration found 88 bugs; the best one-shot
configuration found 54. The three losing comparisons were all Sonnet 5 running Compound Engineering.Show this chart as a table
Verified bugs found by AI code review, one-shot prompt vs harness, same model. The harness won 39 of 42 comparisons; best harness 88 bugs vs best one-shot 54 (of 147). Data and method:
What the extra findings cost
A harness spent a median of 10× the tokens, about 5× the
dollars and 3.5× the time per review of the same model run one-shot. At low effort the median harness review took
2.8 minutes, against 54 seconds for one-shot.
There is also more to check: the median harness configuration produced 5.5 unsupported
findings (“hallucinations”) across the six reviews against 0 for one-shot, alongside 27.5 accepted useful suggestions
against 6.5. More useful findings, but more to check.
The best value: an open-weight model running a harness
Choose an open-weight model to keep the cost of a harness down. A harness uses more tokens, so the model’s token prices matter. The measured comparison: GLM-5.3 running metareview at low effort
scored 0.567 at $0.22 a review and found 72 bugs. Opus 5 running the same harness at the same effort scored 0.567 at
$2.97 and found 74. That is 1/13 the cost, with the same score at the precision shown.
Scores and uncertainty → REPORT §3.1
Why Opus 5 and GPT-5.6 Sol? They were the strongest Anthropic and OpenAI models when running a harness, and
each is priced a tier below its vendor’s flagship (price table → REPORT §3.4.1).
GPT-5.6 Sol peaked at 0.535 (GPT-5.6 Sol running Compound Engineering at high effort, $0.73 a review), a lower score than the open-weight pick at a higher cost.
Of the 30 closed-model harness configurations we completed, only Opus 5 running Compound Engineering at medium effort outscored it, and none was cheaper.
All 12 open-weight harness configurations outscored every closed-model one-shot prompt (0.478 or better, against a best of 0.427).
Four of the top five scores overall are open-weight models.
What a dollar per review buys
Every complete configuration. Further left is cheaper; higher is better. The line is the efficiency frontier: the best score available at each price. Faded points cost more and score lower than a point on the line. The score (F2′) rewards verified bugs most, credits useful advice and penalises unsupported claims.
Highest absolute score is in the upper right. The best value is in the upper left.
Opus 5 running Compound Engineering at medium effort scored highest with no adjustment for price: 88 bugs and 0.641, at $6.12 a review.
GLM-5.3-Flash running metareview at low effort found 71 bugs for $0.024. It is the cheapest configuration to find 70 or more, at roughly
1/260 of the price. GLM-5.3 running metareview at low effort, a different open-weight model, found 72 for $0.22 with fewer unsupported
findings (5 against 19). The line marks the best score available at each price: every configuration on it is an
open-weight model except the most expensive (Opus 5).
What a dollar per AI code review buys. Top score: Opus 5 · Compound Engineering · medium, 88 verified bugs at $6.12. Tightest budget: GLM-5.3-Flash · metareview · low, 71 at $0.024. Open-weight models own the value end:
Side by side: quality, price and time
Finding counts are totals across six pull requests; cost and time are averages per review.
The score is built from the bug, suggestion and unsupported columns. Verified bugs count most. A useful suggestion (an “accepted advisory” in the full report) is a finding that is
true and worth a maintainer’s attention but is not a correctness bug or a style nit; it earns credit. An unsupported finding is a false claim, a
style-only comment or vague speculation; it costs points. Bugs are proven by tests. Suggestions and unsupported findings are judged by a model, and a
finding the judge was unsure about counts as neither. How the score works → REPORT §2.6
Per token and per review are different costs
Per token, GLM-5.3-Flash running metareview cost 1.7% of what
Fable 5.1 cost as a one-shot prompt. Per review it cost 2.6%, because a harness review uses more tokens than a single prompt. Both are real; they are not the same number.
Use low effort for fast, inexpensive reviews
Within each model and harness,
low effort was the fastest setting in 14 of 14 cases and the cheapest per verified bug in 10 of 14.
Running metareview at low effort, GLM-5.3 took 95 seconds a review, GLM-5.3-Flash took 79 seconds and Opus 5 took 100 seconds.
Turn the effort up and the open-weight models get slow. GLM-5.3 running metareview took about 38 minutes a review at medium
effort and 49 minutes at high, against 3.1 minutes and 4.0 minutes for Opus 5 running the same harness. If a review
has to come back while the author still has the change in mind, stay at low effort.
We compared high and medium effort for the same model and the same way of running it, on the same pull requests.
When a confidence interval includes zero, this sample cannot tell high and medium apart. That happened in
17 of 22 comparisons. Across all comparisons, high effort cost 0.96× to 4.2× as much.
How to read the bars
Each bar is a confidence interval: a range of effects consistent with these results.
The range comes from how much the result changed from one pull request to the next. A dot to the right of zero is not a clear win if its bar extends across zero.
For example, Sonnet 5 running metareview shows a gain of +0.09, but its interval goes from −0.07 to +0.26.
High effort may have helped, done nothing, or hurt; this sample cannot tell which. Each configuration ran once, so the bars do not include
variation from running the same setup again. That is an additional source of uncertainty.
Quality change from medium to high effort
Each row is a model and a way of running it. The dot is the change in F1, a score that balances bugs found against false alarms. This comparison uses the original 42 benchmark labels, rather than the expanded set of 147 verified bugs. The bar is the confidence interval. The cost multiple is on the right. Effort comparison and limitations → REPORT §3.5.3
Grey rows: no statistically significant quality difference. High effort significantly improved F1 in four comparisons and significantly reduced it in one.Show this chart as a table
Does “high” reasoning effort find more bugs in AI code review? In 17 of 22 paired comparisons the gain over medium was not measurable, at 0.96× to 4.2× the cost. Chart and data:
High effort cost more in 20 of 22 comparisons and significantly improved F1 in four.
The largest F1 gain was Opus 5 as a one-shot prompt (+0.24, interval +0.08 to +0.39).
Even there, a harness at low effort found more verified bugs: Opus 5 running either harness found
65 and 74, compared with 47 for the same model using a single prompt at high effort.
Unsupported findings fell more often than they rose. We did not formally test high against low effort.
In raw counts, high effort found an average of 5 more bugs than low, out of 147.
Unsupported findings went down in 13 of 22 configurations and went up in 4.
More pull requests and repeated runs could clarify the effects this sample cannot distinguish.
The recommendation stays the same: use a harness, at low or medium effort. Make high effort earn its extra cost on your own workloads.
Open questions → REPORT §6
The public benchmark’s human reviewers labelled 42 bugs in these pull requests. The models
kept reporting more. So we audited them: every one of the 105 additional bugs has a test that
fails on the pull request and passes on the fix. That gives 147 verified bugs to score against.
42bugs the benchmark labelled
147verified bugs after our audit
88the most any single configuration found
140found by all 66 configurations combined; seven were found by none
Why six pull requests, when the campaign ran 50
The full campaign made 2,416 healthy review runs over all 50 benchmark pull requests, in five codebases.
It is expensive to prove every bug with a test. So we ran the complete grid only on the six highest-severity pull requests:
every model, with every harness, at every effort level. We chose these pull requests with a rule that we set before the test.
Every comparison on this page uses this set.
Then we compared these pull requests with the full benchmark, using the benchmark’s own labels across the 33 configurations that also ran
all 50 pull requests.
What held up: each configuration found about the same share of the labelled bugs on the
six as on all 50 (a mean gap of +0.006), and the model rankings stayed the same or nearly the same in four of
five groupings.
What did not: harnesses looked better on the six than they do across the whole benchmark. On the other, lower-severity
pull requests they raised more unsupported findings, which pulled their overall quality score (F1, which balances bugs caught against false
alarms) down by about 0.08 on a zero-to-one scale. GLM-5.3 running metareview at low effort, for example,
scored 0.73 on the six and 0.57 on all 50.
So read this page for which setup beats which, not for the exact scores, and expect a harness to be somewhat noisier on
lower-severity pull requests than these numbers suggest.
The full check → REPORT §2.4.2
See for yourself: compare any model, harness and effort level
All 66 complete configurations, on the axes you care about. Filter, hover for the numbers, and
click any point or table row to pin it for a side-by-side comparison. The link to your comparison is shareable.
Pinned for comparison
Leaderboard
Follows your filters. Click a column to sort, a row to pin.
This is not a contest between the harnesses. Read it for harness against one-shot, open-weight against closed, and what each costs:
the top 19 configurations are all harness runs, and the dot shows how each review was run.
Compare eight models × three ways of working × three effort levels for AI code review: verified bugs found, noise, dollars and seconds per review. Interactive, with all the data:
What to do with this
Next steps for you and your team
Run a harness, not a single prompt.If your AI review is a single prompt over the diff, this is the biggest lever we
measured: more verified bugs in 39 of 42 like-for-like comparisons.
Use an open-weight model at low effort for value.Both harnesses are open source and free. Compare them on your own pull requests.
Running metareview, GLM-5.3 at low effort found 72 bugs at $0.22 and about 95 seconds a review;
GLM-5.3-Flash is the budget option at $0.024 (71 bugs), with more unsupported findings to check (19 against 5).
Running Compound Engineering, GLM-5.3-Flash at low effort was the cheapest and fastest harness configuration we measured:
63 bugs at $0.019 in 49 seconds, with 11 unsupported findings. Compound Engineering also holds the top score overall
(Opus 5 at medium effort: 88 bugs at $6.12 a review).
Vet the provider before you send it your code.A review sends your source code to whoever hosts the model, and hosts’ terms differ widely.
Work through the provider checklist ↓
Default to low or medium effort.Make high effort earn its place on your own pull requests before you pay for it everywhere.
Measure the time you spend checking findings.We measured bugs found, noise, dollars and seconds. We did not measure developer time saved,
and that is the number that decides whether any of this is worth it for your team.
A checklist when choosing an open-weight model provider
Open weights mean many companies can host the same model, and you can host it yourself. The model is the same; the terms are not.
A code review sends your source code to whoever runs the inference, so ask these questions of any host before a pilot. This is separate from the
recommendation above: it applies whichever open-weight model and harness you choose.
Jurisdiction. Where does inference run, where is the company incorporated, and which law governs the contract? Your code should stay inside
your legal jurisdiction, from the first request to the last log line.
Data retention. Look for zero data retention (ZDR): prompts and outputs processed in memory and never written to disk. Ask what is kept for
abuse monitoring, debugging and backups, and for how long.
No training on your data. A contractual commitment, not a line on a policy page, and one that binds the host’s own suppliers.
Provenance. Which exact weights, version and numeric precision are you served, and who else touches the request on the way? A compressed or
modified variant may not review like the model measured here, which was itself a reduced-precision build (see the limits ↓).
Usage rights. Terms that let you use the output however you want, including commercially. Read the model’s own licence as well as the host’s.
Cost. A harness spent a median of 10× the tokens of a single prompt, so the pricing model matters more than the headline rate: per token or
flat rate, cached-input pricing, rate limits and overage. The costs on this page are list prices.
Speed. Throughput decides whether a review comes back while the author still has the change in mind. Here, GLM-5.3 running metareview took
95 seconds a review at low effort and about 38 minutes at medium.
Assurance. An independent security audit, a data-processing agreement and named sub-processors. We did not audit any provider’s data practices,
so get the answers in writing.
A way out. Open weights let you change hosts, or run the model yourself, without changing the reviewer. Keep that option open.
Pass it on
Know a team choosing an AI code reviewer? Hand them the evidence
Share a finding with your team or a wider audience. Each post starts with the takeaway, then gives the evidence, limits, and source links.
Copy it into LinkedIn or a team channel, edit it in your own voice, and attach the matching chart.
Find more bugs with the model you already use
Your AI code reviewer may not need a smarter model. It may need a better workflow. The same models found 1.6× as many verified bugs when a harness ran the review.
That was the median gain from adding a harness across 42 comparisons in an AI code review study. A harness runs the model through several review passes, with tools and specialist reviewers, instead of asking it to review the code in a single prompt.
→ More verified bugs in 39 of 42 comparisons with the same model and effort level
→ Best harness setup: 88 bugs found. Best single-prompt setup: 54
→ GLM-5.3 with metareview at low effort found 2.1× as many bugs as its single-prompt setup
The gain has a price: median token use was about 10× higher, cost 5× higher, and review time 3.5× longer. There were also more unsupported findings to check. A higher bug count does not automatically mean less work for a developer.
Use a harness. Then measure how much time its findings save on your own code.
The source: Dave Sifry’s open harnesseval study of AI code review. It compared eight models, a single review prompt, and the free harnesses Compound Engineering and metareview at three effort levels. The main comparison used six selected pull requests from two codebases, with 147 verified bugs and a single run per setup. Disclosure: Sifry wrote metareview. The data and code are public.
Finding and chart: https://dsifry.github.io/harnesseval/#f-harness
Technical report, methods and data: https://dsifry.github.io/harnesseval/REPORT.html
What would change if your reviewer found more bugs with the model you already use?
A comparable score for a fraction of the cost
Why pay $2.97 for an AI code review when $0.22 matched its quality score? GLM-5.3 matched Opus 5 using the same harness, at 1/13 the cost.
GLM-5.3 running metareview at low effort matched Opus 5 running the same harness and effort in this study. The score rewards verified bugs and useful advice, and penalises unsupported findings.
→ GLM-5.3: $0.22 per review, 72 bugs found across the test set
→ Opus 5: $2.97 per review, 74 bugs found
→ Both rounded to the same quality score: 0.567
The cheaper model is open-weight: its weights are available for others to host, subject to its license. The result is a reason to choose an open-weight setup for value, not a claim that all models perform equally.
Speed matters too. The open-weight setup took 95 seconds per review at low effort, but 38 minutes at medium and 49 minutes at high. Keep effort low for a fast review. Check the host’s data policies before sending code.
These are list-price estimates. Close scores do not establish equal performance on other workloads; run your own evals.
The source: Dave Sifry’s open harnesseval study of AI code review. It compared eight models, a single review prompt, and the free harnesses Compound Engineering and metareview at three effort levels. The main comparison used six selected pull requests from two codebases, with 147 verified bugs and a single run per setup. Disclosure: Sifry wrote metareview. The data and code are public.
Finding and chart: https://dsifry.github.io/harnesseval/#f-cost
Technical report, methods and data: https://dsifry.github.io/harnesseval/REPORT.html
What would your team do with that difference in review cost?
Make high effort earn its bigger bill
Do you set your AI to High, Max or Ultra and wonder why the results aren’t satisfactory?
In 17 of 22 AI code review comparisons, the study could not establish a quality difference between high and medium effort. High cost more in 20 of 22.
→ High effort significantly improved the quality score in four comparisons
→ It significantly reduced the score in one
→ Costs ranged from 0.96× to 4.2× the medium-effort price
The score here is F1: it balances bugs found against false alarms. This effort comparison used the original 42 benchmark labels, rather than the expanded verified-bug set. An uncertain result does not mean the settings are identical. It means this sample cannot tell them apart.
Default to low or medium effort. Make high effort earn its extra cost on your own code. High versus low was not formally tested, and each setup ran only once, so this is not proof that extra reasoning never helps.
The source: Dave Sifry’s open harnesseval study of AI code review. It compared eight models, a single review prompt, and the free harnesses Compound Engineering and metareview at three effort levels. The main comparison used six selected pull requests from two codebases, with 147 verified bugs and a single run per setup. Disclosure: Sifry wrote metareview. The data and code are public.
Finding and chart: https://dsifry.github.io/harnesseval/#f-effort
Technical report, methods and data: https://dsifry.github.io/harnesseval/REPORT.html
Has your team measured what high effort actually buys?
See what changes when the same model gets a harness
The model gets all the attention. The review workflow deserves more. Harnesses beat single prompts in 39 of 42 comparisons—with the same model and effort.
This chart compares a single review prompt with a harness: several review passes, tools, and specialist reviewers. Each row keeps the model the same. The marks average its results across effort levels.
→ The harness found more verified bugs in 39 of 42 same-model, same-effort comparisons
→ The average gain was 13.5 percentage points in recall—the share of verified bugs found
→ The best harness setup found 88 of 147 bugs; the best single-prompt setup found 54
It was not a win in every comparison. The three losses were all Sonnet 5 running Compound Engineering. Harnesses also used more tokens and produced more unsupported findings to check.
The single prompt was carefully written, with a reviewer role, bug categories, severity levels, and instructions to report only confident findings. Its exact text is in the technical report.
The practical takeaway: the way a model reviews code deserves as much attention as the model you choose.
The source: Dave Sifry’s open harnesseval study of AI code review. It compared eight models, a single review prompt, and the free harnesses Compound Engineering and metareview at three effort levels. The main comparison used six selected pull requests from two codebases, with 147 verified bugs and a single run per setup. Disclosure: Sifry wrote metareview. The data and code are public.
Finding and chart: https://dsifry.github.io/harnesseval/#c-harness
Technical report, methods and data: https://dsifry.github.io/harnesseval/REPORT.html
Have you compared a harness with your current review prompt?
Find the review setup that fits your budget
AI code review for $0.024 a pull request sounds too cheap to be useful. That setup found 71 of 147 verified bugs across the test set. Here’s what paying more actually bought.
That was GLM-5.3-Flash running metareview at low effort in this AI code review study. The bug count is a total across the selected pull requests; the price is the average per review.
This chart puts quality against price:
→ Highest score: Opus 5 · Compound Engineering · medium, 88 bugs, $6.12 per review
→ Lowest cost to find at least 70 bugs: GLM-5.3-Flash · metareview · low, $0.024 per review
→ Fewer unsupported findings than that budget pick: GLM-5.3 · metareview · low, $0.22 per review (5 against 19)
Left is cheaper. Up is a higher score. The line connects the best score available at each price. All but its most expensive point are open-weight models, whose weights are available for others to host under their licenses.
The score rewards bugs and useful advice, and penalises unsupported findings. The leading scores have overlapping uncertainty ranges. Use the chart to choose a setup, then measure its results and the time spent checking findings on your own code.
The source: Dave Sifry’s open harnesseval study of AI code review. It compared eight models, a single review prompt, and the free harnesses Compound Engineering and metareview at three effort levels. The main comparison used six selected pull requests from two codebases, with 147 verified bugs and a single run per setup. Disclosure: Sifry wrote metareview. The data and code are public.
Finding and chart: https://dsifry.github.io/harnesseval/#c-money
Technical report, methods and data: https://dsifry.github.io/harnesseval/REPORT.html
Which matters most for your team: more bugs found, fewer false alarms, or a lower bill?
See where high effort helped—and where it did not
“Think harder” is easy advice when someone else pays the bill. High effort cost more in 20 of 22 comparisons, but significantly improved review quality in just four.
This AI code review chart compares high and medium effort for the same model and review method. It shows the change in F1, a score that balances bugs found against false alarms, using the original 42 benchmark labels.
→ Green: a statistically significant improvement
→ Red: a statistically significant decline
→ Grey: the sample cannot establish a difference
The bar shows uncertainty across pull requests. If it includes zero, high effort may have helped, done nothing, or hurt. Grey does not mean “no effect,” and it does not tell you the cost.
Cost is shown separately on the right. High effort cost more in 20 of 22 comparisons, with ratios ranging from 0.96× to 4.2× across all comparisons. Each setup ran once; the bars do not include variation from repeating the same run.
Default to low or medium effort. Use your own evaluation to decide when a higher bill buys a result worth paying for.
The source: Dave Sifry’s open harnesseval study of AI code review. It compared eight models, a single review prompt, and the free harnesses Compound Engineering and metareview at three effort levels. The main comparison used six selected pull requests from two codebases, with 147 verified bugs and a single run per setup. Disclosure: Sifry wrote metareview. The data and code are public.
Finding and chart: https://dsifry.github.io/harnesseval/#c-effort
Technical report, methods and data: https://dsifry.github.io/harnesseval/REPORT.html
What evidence would make you turn high effort on?
Check your own model choices against the data
Is your AI code reviewer worth what you pay? Compare 66 setups side by side—bugs caught, unsupported claims, price, and speed. See where your setup stands.
This interactive explorer lets you compare models, review methods, and effort levels on the same pull requests. A harness runs several review passes with tools and specialist reviewers; the alternative is a single review prompt.
→ Filter to the models you care about
→ Compare bugs found, quality scores, useful advice, and unsupported findings
→ Put those results against dollars, seconds, or tokens per review
→ Pin setups side by side and share the comparison
The study’s main findings: harnesses found more bugs in 39 of 42 comparisons; GLM-5.3 running metareview matched Opus 5 at 1/13 the cost; high effort often cost more without a clear improvement over medium.
Every number comes from the same data as the technical report. Bug counts are totals across the test set; costs and times are averages per review. Small score differences are not reliable rankings, and the study did not measure developer time saved.
Choose a setup from the evidence, then run your own evals on your own workloads.
The source: Dave Sifry’s open harnesseval study of AI code review. It compared eight models, a single review prompt, and the free harnesses Compound Engineering and metareview at three effort levels. The main comparison used six selected pull requests from two codebases, with 147 verified bugs and a single run per setup. Disclosure: Sifry wrote metareview. The data and code are public.
Finding and chart: https://dsifry.github.io/harnesseval/#c-explore
Technical report, methods and data: https://dsifry.github.io/harnesseval/REPORT.html
Where does your current reviewer land?
There is much more behind this page
What the full report adds
The whole benchmark. Results for all 50 pull requests on the benchmark’s own labels, and a check of whether using only the selected six PRs changes the conclusions.
§3.6
Compound Engineering against metareview, head to head, with confidence intervals for the difference between the harnesses, for every model and effort level.
§3.3
Where the money and time go: input and output token use, cache pricing, and time per review for each setup.
§3.4.3 ·
§3.4.4
How findings were judged, how much the judges agree, and how changing the scoring weights changes the results.
§2.3 ·
§2.6
Every number, with its interval, and the commands to regenerate it.
§5
Read this before you quote us
What this does not prove
The headline numbers come from six pull requests. They are the highest-severity ones in a 50-PR benchmark, from
two of its five codebases. The check against the full benchmark covers recall and rankings on the benchmark’s own labels;
the 147 verified bugs exist only for these six. This is evidence for running a pilot, not proof about your repository.
§2.4
A single run per configuration. The intervals reflect variation across pull requests. They do not include variation from running the same model again.
§4
Models did much of the judging. Verified bugs rest on executed tests, but “useful advice” and “unsupported” are model
judgments, not human ones, and a GLM-family judge may favour GLM output.
§2.6
“GLM-5.3” here is a quantized, hosted build. Quantization stores model weights at reduced numeric precision. The GLM-5.3 runs were served by a gateway from
GLM-5.3-Vision-NVFP4. Its model card describes it as a reduced-precision (NVFP4)
quantization of Z.ai’s GLM-5.3 text model, unmodified, with a vision tower added. Every request in this study was text only, so the vision part was
never used. A full-precision build, or another host’s deployment, may score differently. We did not verify the gateway’s serving configuration.
§2.2
Some runs have known gaps. The GLM harness runs predate a gateway fix, and Fable’s harness grid was not completed for budget,
so Fable is excluded from the harness comparisons. §2.3.4
Costs are list prices (retrieved 2026-09-16), not invoices, and exclude failed gateway calls.
§3.4.1
The author wrote one of the harnesses. See the disclosure at the top.
§2.3.5
The short version
The findings, each one a decision you can make
“Verified” means proven: 42 bugs labelled by the benchmark’s human reviewers
plus 105 more, each with a test that fails on the pull request and passes on the fix. How we kept score ↑
Finding · harness
1.6×
Same model, add a harness: 1.6× verified bugs found by AI code review
A harness runs the model as a team of specialist reviewers instead of a single prompt. Compound Engineering and
metareview beat the same model’s one-shot prompt in 39 of 42 comparisons. GLM-5.3 running metareview at
low effort gained 2.1×.
The price: about 10× the tokens, and more unsupported findings to check.
AI code review: same model, add a harness (Compound Engineering or metareview) and it found 1.6× the verified bugs, beating one-shot prompting in 39 of 42 same-model comparisons at about 10× the tokens. Data and method:
Finding · cost
1/13the cost
Best value: open-weight GLM-5.3 running a harness matched Opus 5
GLM-5.3 running metareview at low effort scored 0.567 at $0.22 a review.
Opus 5 running the same harness scored 0.567 at $2.97. GPT-5.6 Sol peaked at 0.535,
at $0.73 a review.
The single best score is still Opus 5 running Compound Engineering at medium effort (0.641), at $6.12 a review.
Intervals overlap: read “comparable”, not “better”. Fable’s harness runs were not completed.
AI code review: open-weight GLM-5.3 running metareview at low effort matched Opus 5 in the same harness (0.567 vs 0.567) at 1/13 the cost ($0.22 vs $2.97 per PR) and outscored GPT-5.6 Sol:
Finding · effort
17of 22
Default to low or medium: high effort rarely bought a measurable gain
In 17 of 22 head-to-heads, high effort showed no statistically significant quality gain over
medium, at 0.96× to 4.2× the cost. It helped in four and hurt in one.
Measured on the benchmark’s original 42 labelled bugs. High versus low was not formally tested.
Paying for “high” reasoning effort in AI code review: in 17 of 22 head-to-heads it showed no measurable quality gain over medium, at 0.96× to 4.2× the cost. Helped in four, hurt in one. Data:
Special thanks
Acknowledgments
Thank you to the team at LunaRoute, who supported this work with an account and free usage
while their service was in alpha. The open-weight GLM reviews on this page ran through their gateway, and that help was much
appreciated. §3.4.1 I paid for the Anthropic and OpenAI usage myself.
Disclosure. I have no relationship with LunaRoute other than as a user, and I used the service free of
charge as part of the alpha.
Thank you to Ian Langworth (@statico on X and
GitHub) for reading through the copy and suggesting improvements to the wording and clarity.