Haiku vs Opus on a well-built MCP server

We ran the same 13 tasks against two MCP servers that share one issue-tracker backend. On the well-built server, Claude Haiku 5.5 solved 99.0% of its runs and Claude Opus 5.5 solved 100%, one run apart out of 104. On the second server, which copies flaws from real APIs, Opus still solved 99.0% and Haiku fell to 64.4%.

If you're choosing a cheaper model for an agent, you want to know whether its misses come from the model or from the server it's talking to. We measured that with M3, our open-source tool for testing MCP servers and the agents that use them.

The results: a good server closes most of the gap

On 12 October 2026 we ran six models at default settings: three Claude models through Claude Code 2.1.296 and three OpenAI models through Codex 0.162.1. Each model ran 13 tasks on each of the two servers, 8 times per task. That comes to 104 runs per model on each server and 1,248 runs in all. Six of them failed inside Codex before the model finished, and we left those out.

Model	Good server	Naive server
Opus 5.5 (Claude Code)	works104/104	works103/104
Sonnet 5.5 (Claude Code)	works104/104	breaks71/104
Haiku 5.5 (Claude Code)	works103/104	breaks67/104
GPT-6 Astra (Codex)	works100/104	works103/104
GPT-6.1 Sol (Codex)	works101/102	works95/100
GPT-6 Luna (Codex)	breaks89/104	breaks52/104

The good server has 9 tools with flat arguments and enums. It accepts users and labels by name, filters on the server, returns a total_count and a cursor, and writes errors that say how to fix the call. The naive server exposes about 70 generated REST-style tools, including near-duplicates and a deprecated one. It paginates like GitHub (page and per_page, 20 per page, a bare array with no total and no "more" flag) and changes status through Jira-style workflow transitions. Priority is an integer from 0 to 4 that only a separate tool explains, users are referenced by id, and a bad call gets 400 Bad Request.

These are success rates with 95% Wilson intervals. To compare two models we bootstrapped over tasks with 2,000 resamples, and we call them equivalent when the 90% interval of the difference stays within 10 points. Haiku on the good server and Opus on the naive server both scored 99.0%, a difference of 0 with a 90% interval of [-1.9, +1.9].

Haiku and Luna have the same list price, and Haiku beat Luna by 13.5 points on the good server (95% interval [3, 28]) and by 14.4 on the naive one ([2, 29]). Even on the good server, Luna (85.6%) trails Astra on the naive server (99.0%) by 13.5 points ([-26, -3]). Put another way, the good server closes 97% of Haiku's gap to Opus and 77% of Luna's gap to Astra.

Sonnet and Sol also cost the same, and they are equivalent on the good server. On the naive server Sol beats Sonnet by 27 points ([9, 48]): Sonnet went from 100% to 68% and Sol from 99% to 95%. We don't know why. Codex's code mode, where a loop over pages is easy to write, would explain it, except that Luna runs in code mode too and still fails.

Where small models break

Most of the damage is in four tasks where the model has to page through a long list and put together what it finds. One asks for the open, unassigned regression bugs across two projects and two sprints. One asks which team has the most urgent issues among all 520. One applies a triage rule to every flaky issue in a project, and one follows a chain of lookups to reassign everything a particular person filed. We'll call these the list-and-join tasks. The other nine tasks mostly held up. On the naive server Sonnet solved 66 of their 72 runs, Haiku 63 and Luna 49, and the other three models 69 or more.

Anthropic	Good server	Naive server
Opus 5.5	works32/32	works32/32
Sonnet 5.5	works32/32	breaks5/32
Haiku 5.5	works32/32	breaks4/32

OpenAI	Good server	Naive server
GPT-6 Astra	works32/32	works31/32
GPT-6.1 Sol	works32/32	works26/31
GPT-6 Luna	breaks22/32	breaks3/32

Most of these failures look like success from the outside. We ask every model to end its answer with a status line that says whether it finished. On the naive server 129 runs failed, and 113 of them (88%) reported that they were done while giving a wrong or partial answer. Only 13 said they couldn't finish. In a typical trace the model reads a page or two, sees no total and no "more" flag, and reports what it has as the complete list. On the regression bug list, Luna made 5 list calls, when one of the two projects alone takes 4 pages at the largest page size.

Which flaw does the damage

We built five more versions of the good server, each with exactly one of the naive server's flaws: vague tool descriptions, the bloated 70-tool catalog, the Jira-style arguments, bare error messages, or the silent pagination. We ran each version 5 times per task with the mid and small models of both vendors and looked at the four list-and-join tasks.

Server	Haiku 5.5	Sonnet 5.5
Good	works32/32	works32/32
Only bad descriptions	works20/20	works20/20
Only the bloated tool catalog	works20/20	works20/20
Only Jira-style schema	works18/20	works20/20
Only bare errors	works17/20	works20/20
Only silent pagination	breaks13/20	works17/19
All five (naive)	breaks4/32	breaks5/32

Server	GPT-6.1 Sol	GPT-6 Luna
Good	works32/32	breaks22/32
Only bad descriptions	works19/20	breaks13/20
Only the bloated tool catalog	works20/20	breaks13/20
Only Jira-style schema	works20/20	breaks15/20
Only bare errors	works20/20	breaks12/20
Only silent pagination	works20/20	breaks9/20
All five (naive)	works26/31	breaks3/32

On its own, no flaw takes Haiku below 65% on these four tasks. All five together take it to 13%, where multiplying the single-flaw rates would predict about 50%, so the flaws make each other worse. Luna follows the same pattern from a lower start: 69% on the good server, 45% with only pagination naive, and 9% with all five, against about 35% predicted. Sol shrugged off every single flaw. Pagination does the most harm alone for both small models. Across all 13 tasks, Haiku drops from 99% to 86% when pagination is the only naive part of the server, and Luna from 86% to 65%.

What this means for your MCP server

• Say how many results there are and whether more exist. Pagination without a total_count or a has_more flag was the worst single flaw in our one-flaw-at-a-time runs, and it usually failed as a partial list reported as complete.
• Filter on the server. When the model has to page through 520 issues and filter them itself, each page is another chance to stop early.
• Accept names where people use names. Taking users and labels by name removed a lookup chain from several tasks.
• Use plain status and priority fields where you can. Workflow transitions and unexplained integers add calls the model can get wrong.
• Write errors that say how to fix the call. 400 Bad Request gave the model nothing to retry with.
• Keep the catalog small. With only the 70-tool catalog swapped in, Haiku and Sonnet still solved nearly everything, but each paid 20 to 30% more per success.

What it costs

Cost per successful task is an estimate. For Claude Code we used the cost it reports, and for Codex we multiplied token counts by list price. Haiku and Luna list at $0.10 in and $0.50 out per million tokens, Sonnet and Sol at $2 and $10, Opus at $4 and $20, and Astra at $10 and $50.

Per successful task, Haiku on the good server costs about a hundredth of what Opus costs on the naive server, at the same success rate. The naive server raised every model's cost per success, by 3 to 4 times for the Claude models, and that includes the flagships whose success rate didn't move. It was slower too. Median latency went from 18 to 32 seconds for Opus, 28 to 41 for Astra and 38 to 55 for Sol.

How M3 runs each cell

Trackly, the tracker behind both servers, is a fake issue tracker with 6 projects, 40 users and 520 issues. Its backend is deterministic, and a random value per run changes the answers, so a model can't remember them from an earlier run. Each cell in the tables above is an ordinary pytest test that M3 runs against the real Claude Code or Codex binary at a pinned version. Here is the whole test for the regression bug list task ("open, unassigned regression bugs in PAY or WEB in the Oct-W2 or Oct-W3 sprints, minus anything labelled wontfix-candidate"):

pytestmark = pytest.mark.m3(suite_name="model-tiers")

def test_h01(agent, variant) -> None:
    assert run_task(agent, Path(__file__).resolve().parent, variant) == "success"

M3 records every tool call on the wire. The benchmark replays those calls against the same backend, then checks the tracker's final state and the status line at the end of the answer. There is no LLM judge. A scripted agent that sees only tool responses solves every task on every server version, so each failure we count is the model's, and we read a sample of the failed runs by hand. The whole run is one command, with each harness's models separated by commas and M3 options before --:

MT_VARIANTS=good,naive m3 test --runtime=managed \
  --harness [email protected]=claude-opus-5-5,claude-sonnet-5-5,claude-haiku-5-5 \
  --harness [email protected]=gpt-6-astra,gpt-6.1-sol,gpt-6-luna \
  --trials 8 --execution-timeout 600 -n 8 -- tasks _baseline

If you log in to Claude Code with a subscription, add --credential-env claude_code:CLAUDE_CODE_OAUTH_TOKEN=CLAUDE_CODE_OAUTH_TOKEN. The tasks, both servers and the grader are in the repository under benchmarks/model-tiers.

Everything ran in one domain, an issue tracker. Each result is a model running inside one agent tool (Claude Code or Codex) at default settings, and a different agent tool could move it. The costs are estimates from reported costs and token counts. Codex shows OpenAI models their tools as TypeScript declarations rather than JSON Schema, which may account for part of the gap between vendors. The benchmark README lists the fixes we made during the pilot, and the records of the pilot and of every run are kept with the benchmark.

What surprised us

Opus caught a mistake of ours. One task asked the model to rebalance a team's work so nobody had more than three issues in progress, and with the data we had generated that couldn't be done. In three of its four pilot runs, Opus said so, explained why and left the tracker alone. We fixed the data and reran it.

On the regression bug list, Haiku read "bugs" as the bug label and said so in its answer. Our data has both an issue type and a label called bug, so that was a fair reading. We changed the grader to accept either one, as long as an answer sticks to it.

Sonnet surprised us most. In our smaller pilot it lost 17 points on the naive server, and in the full run it lost 32 and landed next to Haiku, while Sol, at the same price, lost 4. Opus didn't notice the naive server and still paid 3.4 times as much per success on it.

Test your own MCP server on small models

You can run the same comparison against your own server. Install the CLI and scaffold a project:

uv tool install sf-m3-cli
m3 init      # a skipped starter test and an .env.example
m3 setup

Write the task you care about as a pytest test, then run it across several models in one invocation, so a change on the provider's side hits every column at once:

m3 test --runtime=managed \
  --harness [email protected]=claude-opus-5-5,claude-haiku-5-5 \
  --harness [email protected]=gpt-6-astra,gpt-6-luna \
  --trials 8 -- tests/

M3 is open source under Apache 2.0 and still in alpha: github.com/sineframe/m3. Start with the documentation. If a flaw in your server breaks a model we didn't try, open an issue there.