Boundary-Bench enables agent and benchmark builders to quantify the effect of a restricted environmentA computer set up with security rules that limit what users and agents alike can do: which sites they can reach, which files they can change, and which permissions they get. on success rate and cost.
Powered by
Agent performance
NIST high policyA widely used U.S. government catalog of security controls that many organizations base their policies on. Our levels are derived from it; they are not a compliance certification.Loss caused by the policy
AgentSuccess rate
Grok 4.5Grok Build · xAI
GPT-5.6 SolCodex · OpenAI
Fable 5Claude Code · Anthropic
Gemini 3.7 FlashAntigravity · Google
Opus 5Claude Code · Anthropic
GPT-5.6 TerraCodex · OpenAI
Opus 4.8Claude Code · Anthropic
Muse Spark 1.2Muse Code · Meta
GLM-5.2Terminus 2 · Z.ai
Sonnet 5Claude Code · Anthropic
Kimi K3Terminus 2 · Moonshot AI
GPT-5.6 LunaCodex · OpenAI
MiniMax M3Terminus 2 · MiniMax
Qwen3.7 MaxTerminus 2 · Alibaba
Terminal-Bench 2.1 · reasoning effort: high · 3 trials per agent-task pair
Measure coding agents
where they actually run.
Boundary-Bench evaluates coding agents inside restricted environmentsA computer set up with security rules that limit what users and agents alike can do: which sites they can reach, which files they can change, and which permissions they get., the policy-enforced setups agents meet inside real organizations, and measures the drop in success rate and the rise in cost that this enforcement causes.
Benchmarks run agents without limits. Companies don't.
Coding-agent benchmarks run agents in a permissive environment without treating that environment as a factor in the results. Real companies restrict network access, file-system access and permissions for people, programs, and agents alike.
An agent chosen from a benchmark that runs in an unrestricted environment can behave very differently once those limits apply. Boundary-Bench makes the environment the experiment.
Same tasks, same checks, enforced restrictions.
We take an established benchmark, today Terminal-Bench 2.1 and its 89 tasks (with more benchmarks to come), and run each agent in a sandboxA disposable, isolated computer created fresh for each run, so nothing the agent does can affect anything outside it. Sandboxes are the standard way benchmarks set up environments for testing and evaluation. restricted by native Linux enforcement. The agent is never told about the limits and it simply hits them, the way any program would.
Every run is scored by the benchmark's own verifierThe benchmark's official checking program. Every standard benchmark ships one: it inspects the finished work and decides whether the coding agent succeeded.. We run the full benchmark under every policy level, open and restricted alike, and the gap between the unrestricted and restricted scores is what the security policy costs.
Want to run it yourself? The run guide walks you through setup.
Three policy levels, from open to strict.
Restrictions are derived from the security controls of NIST SP 800-53A widely used U.S. government catalog of security controls that many organizations base their policies on. Our levels are derived from it; they are not a compliance certification., the catalog many enterprise policies are built on.
ControlThe open baseline. Full internet, writable files, administrator rights.
Non-rootAn ordinary user instead of an administrator. Everything else stays open.
NIST highInternet limited to an egress allowlistA list of approved destinations, such as package registries, APIs, and the model endpoint, that the environment is allowed to reach. Anything not on the list is blocked. The list is set per task., system files read-only, no path back to administrator rights.
Prefer your own mix? Compose one in the policy builder.
Protection is never free, and the price differs by agent.
Every agent we tested loses some success and spends more money under restrictions, but not equally: some stop when they encounter a block and lose success rate, while others keep trying and spend more, still losing some success along the way. The leaderboard reports that trade-off: success and cost side by side, with and without restrictions. The full analysis lives in our research.