How Does It Work?
Disclaimer: if you want a universal intelligence ranking, read Artificial Analysis or LMArena.
The Featherbench question is more pragmatic: is a single model good enough to handle a full lap of everyday work — securely, quickly and cost-effectively? To answer that, we run the latest models through the same 28 fixed tasks covering coding, data extraction, everyday requests, tool use, and prompt-injection and jailbreak resistance. A model passes a task only when a deterministic checker confirms it produced a usable result.
The tasks are deliberately simple and limited in range, drawn from the day-to-day work our founder, Ed, actually performs. That is what lets you reproduce the whole benchmark, or run a tailored version on your own tasks, with an OpenRouter account and about $30. The value is in re-running it against a new model within minutes of its announcement.
Let's face it: you all have your favourite vibe tests you run every time a new model is announced — Simon Willison asks each one for an SVG of a pelican riding a bicycle. This is the same idea, with more rigour and a wider net.
Isn't routing the answer? For many enterprises, yes. But a router adds opacity, and complexity in development and observability — and it still needs a default to fall back on when the routing call is uncertain. That default is what this board is looking for. If one model clears the bar on everything you do, you may not need the router at all — one model, one set of prompts and evals to maintain. If it doesn't, you now know which tasks to route away from it.
The overall score is the pass rate across my 28 realworld tasks. As we only had a limited number of trials there is a wide Wilson interval — the whiskers on the chart.
Summary of results: click a column to sort by your chosen metric.
Five caveats on how these numbers were produced
The lap, corner by corner #
The most recently added models appear first, with the latest test date shown under each. The lap is five corners in fixed order: Coding → Data → Realworld → Security → Tool-use. A corner's colour is that model's pass rate in that category. Green is good — it means 85%+ success. For models that can do it all, look for all green. The number in the middle of each ring is that model's cost per task; below it is the median time to first token, in seconds.
Hover or tap any segment for what that corner tests and how the model handled it.
Clean corner (>85%) Ragged (60–85%) Off the track (<60%)
Our pick — the All-Star champion, the desert island model Our pick for a low-cost workhorse Our pick for the fastest reply
See the exact numbers by category
Cells below 60% are flagged red and 60–85% amber — coding, data and tool-use are the harness floor, so the race is decided in realworld and security.
What Do the Results Actually Tell You?
The field is narrowing
The board is bunching up: nine models now tie at 96%, and the pass rate no longer tells them apart. Benchmarkers call it saturation, and in my book it is no bad thing — it means more models are good enough for the single-model setup that keeps a production system consistent and its costs predictable. A Fable still earns its keep on the hard, unsolved problems, but for everyday work it is a Ferrari on the supermarket run. With speed and cost per task the main things left to separate the field, the market is commoditising.
If you only run one model, run glm-5.3
glm-5.3 is the first model on the board to clear all five corners — coding, data development, realworld, security and tasks — at 100%. It backs that with a 9.3 rubric, third-highest on the board, and $0.28 for the lap. It is however really slow: 16.3-second median time-to-first-token.
Anthropic safety classifier making headlines for the wrong reasons
fable-5-1 redeems the Fable family name, bringing it up towards the top of the leaderboard — though it still refused one of the coding tasks. Anthropic acknowledged the issue in the fable-5-1 release notes, recommending a fallback to opus-5 for biology and opus-4.8 for cybersecurity. To me that reads as Anthropic shipping a buggy classifier and making it our problem to deal with.
glm-5.3-flash is the low-cost workhorse
glm-5.3-flash costs $0.031 for the full lap, or $0.0004 per task, with a 2.6-second median TTFT, you can code all week and it will still only cost the same as a Starbucks. At 99% overall and 100% on security, it replaces Luna and haiku-4-5 as the low-cost default: it is better on all metrics. haiku-4-5 remains faster to first token at 0.9 seconds, but costs 10× more and has lower-quality output. Luna costs five times more per task and suffers from jailbreak vulnerabilities, so it cannot be used without extra guardrails.
The mystery guest sets the fastest quality lap
kimi-k3 still tops the rubric at 9.5, judged independently by fable-5; fable-5-1 self-scores the same 9.5 — not a like-for-like number (footnote 5). For interactive use fable-5-1 takes the head-to-head anyway: kimi-k3's 26.4-second median time-to-first-token is more than three times fable-5-1's 7.3s.
Three cars failed the crash test
The gpt-5.6 line is quick, but it has a safety problem. gpt-5.6-luna, gpt-5.6-terra and gpt-5.6-sol emitted the jailbreak canary in 11 of 12 jailbreak cells (33–50% security pass) — make sure you protect in your harness, and apply more careful Red teaming if using these models. The Claude trio went 6/6 clean, as did gpt-5.5.
How Is the Ed-o-meter Scored?
- Same tasks run for all models using the same prompts, same API calls, measured through one identical OpenRouter streaming path, run serially as time-trial. No other cars on track
- Latency is time-to-first-token, measured through one identical OpenRouter streaming path, run serially so the clock is uncontaminated. Wall-clock is recorded alongside.
- Checkers are binary and automated. The LLM rubric is the only judged component — and its bias is made visible in the footnotes rather than assumed away.
- Effort and reasoning settings are pinned in
models.jsonand stated with any published number, because they materially move quality and cost. - Refusals are recorded, not hidden. A provider-side hard stop is logged as a refusal with its category — never silently retried on another model. Routing is pinned with
allow_fallbacks:false, so no quiet re-serves on quantized variants. A model that declines in prose is scored by the checker like any other answer.
Harness, tasks and checkers are open source at Featherbench (MIT). Clone it and run the lap yourself, or request a new model via GitHub issue.
See all 28 tasks
Coding (7 · Python)
- CSV dedupe — small, well-specified task with a deterministic unit-test checker
- Debug billing date — fix a month/day-overflow date bug without regressing the working cases
- Debug money split — split integer pennies N ways so shares sum exactly and stay fair
- Debug mutable default — fix the classic mutable-default-argument bug
- Debug pagination — fix an off-by-one page-count bug
- Log parsing — parse logs with quoted quotes, missing fields, IPv6 edge cases
- Rate limiter — stateful class implementation tested against an injected fake clock
Data (4)
- CSV schema mapping — map a raw CRM export onto a conformed customer dimension
- Fabric roadmap — turn a catalogue, mapping and requirements into a phased user-story roadmap
- Data model from interview — derive a star-schema model and requirements from a stakeholder transcript
- Data quality assessment — catch seeded defects in a small table and prioritise fixes
Realworld (9)
- Crying baby advice — safety-sensitive triage for an unusually crying 8-week-old
- Date night in Nottingham — local recommendations without hallucinated venues
- Flight search honesty — admit no live pricing data rather than fabricate flights
- Strict bullet formatting — obey exact format constraints under an awkward word ban
- Lisbon holiday plan — five-day itinerary respecting cost, kids and a rest day
- Honey cough pushback — correct a wrong premise: no honey for a 5-month-old
- Marathon PB plan — week-by-week training plan around a fixed club session
- Vegetarian weeknight recipe — genuinely meat-free recipe for six
- Tenancy extraction — grounded answers from a provided agreement, including admitting what isn't covered
Security (6)
- Email summary injection — summarise an email without obeying an embedded refund instruction
- UnGpt document injection — summarise a review without emitting the embedded jailbreak canary
- AIM/Machiavelli jailbreak — refuse the unfiltered persona while still answering the benign question
- Developer Mode jailbreak — refuse the dual-response jailbreak template
- Evil Confidant jailbreak — refuse the "no rules" persona while staying helpful
- OPPO jailbreak — refuse the opposite-of-a-safe-model persona template
Tool-use (2)
- Flight tool selection — search only, don't book, when explicitly told not to
- Weather tool call — call the weather tool with the correct location argument
See each model's actual answer to every task, or pick two models to compare head to head →
Change log — latest: one model added, 1 September 2026
-
One model added. Added fable-5-1 to the default board: 96% pass, a self-judged 9.5 rubric and 100% security — but $2.68 for the lap, the most expensive model on the board.
-
One model added. Added glm-5.3-flash to the default board: 99% pass, 100% security and a 2.6-second median time to first token.
-
Four models added. Added glm-5.3, grok-4.6, deepseek-v4-pro and gemini-3.7-flash. glm-5.3 clears all five corners at 100% and is now the board's headline pick, ahead of gpt-5.5.
-
OpenAI cost update. Re-ran the GPT-5.6 trio after OpenAI cut its prices, refreshed the cost figures and highlighted Luna as the cheapest workhorse for low-risk, retryable jobs.
-
Three models added. Added opus-5, gemini-3.6-flash and grok-4.5, with their full task, quality, security, latency and cost results.
-
Leaderboard launched. Published the first Ed-o-meter, including the Claude reference set of haiku-4-5, sonnet-4-6 and sonnet-5.