How AutomationBench works
Each task boots a tiny simulated company, hands the agent a request a real ops person would get, and grades the world it leaves behind. The agent's reply isn't scored — the data is.
600+
Held-out evaluation tasks
01
Boot a tiny company
The task seeds a fresh simulated business: CRM records, inbox threads, spreadsheets, support cases. Every model starts from this exact same world — including its traps: stale rows, near-duplicate names, policies buried in an inbox.
02
The agent works alone
One trigger message kicks things off. The agent gets tools to search, read, update, and send — no clarifying questions, no human in the loop. It has to find what it needs the way a new hire would: by looking.
03
Grade the end state
When the agent stops, deterministic assertions inspect the world it left behind. Was the record updated? Did the right team get the email — and the wrong teams not? The outcome happened or it didn't.
One task, end to end · sales.multi_hop_lookup
"We just closed the Meridian Corp Platform Deal! Mark it as won and route the win notice to the right team per our routing policy. Confirm the account tier from the Account Hierarchy spreadsheet, convert currencies if needed, and check for any open support escalations."
CRM · three Meridians
| Meridian Corp — Platform Deal | €120,000 |
| Meridian Solutions — Platform Deal | $150,000 |
| Meridian Corporation — Platform Deal | $95,000 |
Near-duplicate names. Update the wrong record and the task fails.
Sheets · stale rows
| EUR → USD | 1.10 | Jan 10 |
| EUR → USD | 1.30 | Jan 25 |
| Tier | Mid-Market | Dec 15 |
| Tier | Enterprise | Jan 12 |
Both sheets contain outdated entries. The newest row wins: €120,000 × 1.30 = $156,000, tier = Enterprise.
Inbox · the routing policy
"Enterprise → executive-team@ · Mid-Market → vp-sales@ · SMB → smb-team@ … if the account has any open Critical/High escalations, also notify support-escalation@."
The rules aren't in the prompt — the agent has to find this email. And the open Critical case sits on the parent account, one hop up the hierarchy.
The grade · six assertions on the final state
Opportunity
Meridian Corp — Platform Dealstage is
Closed Wonthe right record, not a lookalikeEmail sent to
executive-team@naming the deal,$156,000, andEnterprisecorrect tier from the newest row, correct FX mathEmail sent to
support-escalation@with the same detailsfound the parent account's open Critical caseNo win notice sent to
vp-sales@the stale tier would have routed hereNo win notice sent to
smb-team@a lookalike account's tierNo win notice sent to
sales-team@the lazy default for "everything else"
Official metric
task_completed_correctly
Strict pass/fail. Every scored assertion must pass. Leaderboard scores run against a held-out private evaluation set; the public 600-task set is for research and experimentation. Run-to-run variance is typically within 1%.
Diagnostic only
partial_credit
Fraction of assertions passed. Useful for debugging and training signal — and as a dense reward when the benchmark is used as an RL environment. Not part of the headline score.
Frequently asked
Why are the scores so low?
The benchmark measures full workflow completion across multiple apps, not isolated reasoning. The most striking failure mode: models declare success while actually failing. In one analysis, 72% of Opus's failures, 91% of Gemini's, and 84% of GPT-5.4's involved this false confidence — the agent reports the task done, but the world state is wrong.
Other common failure modes: not persisting when first searches don't find data, assuming data lives where it intuitively "should" (CRM instead of Sheets), processing some items from a list then summarizing as if done, paraphrasing instructions instead of following them exactly.
A task is only complete when business state is correct end-to-end. Getting most of the way there still fails strict scoring.
How does the agent actually interact with the apps?
Two tools: search runs BM25 keyword search over API schemas and returns the top 5 candidates; execute mimics a curl/fetch with method, URL, and body. Discovering the right endpoints is part of the challenge.
Leaderboard scores run in API mode. Behind the simulated apps, Pydantic models are the source of truth — schemas, pagination, required fields, and 4xx error cases behave like the real APIs, but state lives locally so runs are reproducible. Agents get up to 50 steps per task (rarely hit). AutomationBench also supports Zapier-tool mode and a Limited Zapier toolset for experiments on how tool surface shape affects performance.
How is this different from other agent benchmarks?
Many AI benchmarks test coding, browsing, QA, or general reasoning. AutomationBench focuses on whether agents can complete cross-app business work reliably across sales, marketing, ops, support, finance, and HR.
Key design choices: deterministic final-state assertions instead of LLM-as-judge, multi-app workflows instead of single-turn Q&A, simulated tools and APIs instead of screenshot-based browsing, and both positive and negative assertions — the negatives exist to prevent shotgun reward hacking, like emailing everyone in the company instead of the specified recipients.