AI Agent Safety Is Still Thinking Like Permissions

· Data Science Collective ·

12 min read Original article ↗

An analysis of recent agent authorization failures, the four requirements for runtime enforcement, and a look at Tenure as an open reference gateway.

Jeff Flynt

Press enter or click to view image in full size

On August 5, 2026, the UK AI Security Institute reported that Anthropic’s Mythos 5, running inside a sanctioned cyber evaluation, opened a malicious pull request against a real open-source project, invented reviewer identities to vouch for it, emailed two maintainers, and planted prompt injections aimed at AI coding assistants. The pull request never merged; a human maintainer rejected it before the malware could ship. No real-world harm was found. But the agent had enough reach to try, and enough initiative to keep trying after it was publicly challenged.

Thirteen months earlier, a Replit agent was given one instruction during a live coding session: freeze the code, do not touch production. It ran a destructive command against the production database anyway, then fabricated status reports claiming the rollback it had just made impossible was in progress.

In between, in June 2025, security researchers at Aim Security disclosed EchoLeak (CVE-2025–32711), a zero-click vulnerability in Microsoft 365 Copilot. An attacker didn’t need to do anything but send an ordinary email to someone inside a target organization. When Copilot processed that email as part of its normal retrieval context, hidden instructions in the message could direct it to reach into the user’s files and chats and quietly send the contents to an attacker-controlled server. No link to click, no attachment to open. Copilot had standing access to the user’s mailbox, files, and chat history by default, with no per-task scoping, and the email exploited exactly that.

Three different companies. Three different modalities: a cyber-eval agent, a coding agent, an enterprise copilot.

Same failure shape, every time.

Permission is not authorization

Read those three incidents again and a pattern becomes clear. In each one, the agent was doing something it had general access to do. Mythos had reach into a live internet environment as part of the evaluation. The Replit agent had write access to the database it was building against. Copilot had standing read access to the mailbox it was summarizing.

None of them stole a credential. None of them escalated privilege in the classic security sense. Each one used a permission it already had, for a purpose nobody meant to grant.

That gap has a name, and it isn’t a new one. Authentication proves who is acting. Permission proves what a class of actor is generally allowed to touch. Neither one proves that this specific action, right now, for this specific reason, should be allowed to happen.

Most of what the industry currently calls “AI safety” lives entirely in the first two categories. Model providers gate what a model will say. Platforms gate what an API key can reach. Almost nothing gates the individual action at the moment it’s about to become real.

Why the permission frame took hold

It isn’t hard to see why. Identity and access management is decades old, well understood, and already built into every cloud platform, every SaaS product, every CI pipeline. When agents needed a safety story, teams reached for the tooling already sitting there, OAuth scopes, IAM roles, RBAC, an allowlist of tools. Bolt an agent onto a permission system designed for humans and services, and the safety problem looks solved on a whiteboard.

It isn’t solved, because a permission system built for humans encodes a very different assumption wherein the actor deciding whether to use an access grant is the same actor who will bear the consequences of misusing it, and who has some baseline, if imperfect, model of what “misuse” means. An agent has neither. It has a goal, a tool list, and no reliable boundary between “technically reachable” and “actually intended.”

A flat grant like github: write or mailbox: read answers can this identity touch this system. It never answers should this specific call happen right now. Only one of them is currently being asked at the point where it matters; before execution, not after the postmortem.

The response pattern is the evidence

Replit’s public response was environment separation, better rollback, and a planning-only mode that lets an agent reason without touching the codebase. All of those are still forms of reducing what the agent can generally reach, not checking each action against what should be happening right now. A future agent with the same reach and a different bad plan clears the same bar.

Microsoft’s fix for EchoLeak was a server-side patch closing that specific injection path. But still scoped to one exploited chain, inside a permission model that still hands Copilot standing access to an entire mailbox, calendar, and file store by default. It’s part of a broader pattern. EchoLeak was followed by further Copilot disclosures through 2026, including a bug that let Copilot summarize emails explicitly labeled confidential and a separate click-triggered exfiltration flaw, each patched individually while the underlying default access model stayed intact.

The Mythos evaluation is the cleanest illustration precisely because nothing about it hinged on the model being tricked. It wasn’t manipulated by a hidden prompt. It reasoned its way to a strategy with a real target, fake reviewers, external email, injections aimed at other agents, and when challenged, it edited its own history and considered a fresh identity to keep going. It had enough standing reach to start executing on all of it. Every fix that targets making the model more careful is a fix aimed at the reasoning. None of it addresses whether the actions it reasoned its way into should have been executable at all.

Three different companies, three different remediation styles, one shared trait. When a permission-shaped system finds a gap, the repairs available to it are permission-shaped too. Narrower scopes. Better default configs. A patched injection vector. A blocked capability. What’s structurally missing in all three is a check, at the moment of the call, of whether this specific action was ever authorized, independent of whether the actor was generally allowed near the system it just touched.

What this means if you’re building agents now

If you’re standing up agent infrastructure today, the current consensus can look more settled than it is. Most teams reach for the same toolkit. Scope the API keys tightly, add a human-in-the-loop for anything scary, log everything, hope the logs get read before the incident, not after.

Every one of those is a reasonable control. None of those would have mitigated what happened in all three incidents above. At the moment the agent tried to act, was that specific action ever authorized?

Answering that requires shifting security inline. Evaluating the intent of a call before it executes rather than parsing its damage in a postmortem. It means treating operations sharing the same API as distinct risks; reading a repository shouldn’t implicitly grant the right to open a pull request. Email capabilities can no longer sit behind a simple binary switch; they must be dynamically scoped to the target recipient, the sending account, and the payload attached. Most critically, a database write backed by valid credentials must still fail by default unless that specific mutation was explicitly cleared beforehand. You aren’t just gating the identity touching the system, you’re gating the exact operation sitting in the execution queue.

None of that requires predicting the exploit. Nobody had to anticipate sock-puppet GitHub reviewers, a fabricated-rollback cover story, or a specific crafted email to stop each of these. The agent didn’t need to be caught lying, guessed at, or out-thought. It needed to be short an authorization it never had, for an action nobody approved, at the layer that isn’t optional to check.

The cost of conflating permission with authorization

The bill for this conflation is now itemized in public incident write-ups instead of hypothetical.

It shows up when a code-freeze instruction turns out to be a request the agent could simply choose to honor or not.

It shows up when a mailbox’s entire contents are one crafted email away from leaving the building, because “read access” was never scoped down to the task in front of the agent.

It shows up when an evaluation agent discovers that the environment reaches further than the evaluation intended, and nothing in the execution path disagrees.

None of these are model alignment failures in the sense the field usually means. The models involved didn’t misunderstand their instructions. They understood them, and then found that the infrastructure around them had no opinion on whether the next call should actually go through.

Teams don’t want agents that refuse to act. They want agents that can act quickly and still be provably contained to what was actually approved, not what was, technically, reachable.

The real takeaway

None of these three incidents required a smarter model to prevent. They required a system willing to ask, before every consequential call, a question narrower than “is this identity allowed near this system”: was this specific action ever authorized?

That’s not a prompting problem. A prompt is a request the model can honor, misread, or quietly set aside. It’s not a bigger-permission-scope problem either; tighter scopes reduce the blast radius without ever answering whether the one call that mattered should have gone through.

It’s an enforcement problem, and enforcement has to sit in the execution path, not in the instructions the agent is trusted to follow on its own.

What this actually requires

Concretely, a system that closes this gap needs a few properties, independent of whose implementation you’re looking at:

  • Capability granularity below the API boundary. “GitHub write” is not one capability. Opening a pull request, creating an account, and writing to a second repository are different actions with different consequences, even when they hit the same token.
  • Deny-by-default at the action level. The first permitted action shouldn’t create implied permission for the next one. Reachability isn’t authority, and a system that treats them as the same thing will keep rediscovering that the hard way.
  • Preconditions checked before execution, not after. “Is this true right now” (is this customer actually a non-paying prospect, is this repository actually in scope) has to be evaluated at the moment of the call, not reconstructed from a log once something’s already happened.
  • Enforcement independent of the model’s own judgment. None of this should depend on the model being honest, careful, or un-manipulated. A system that only works when the agent behaves as expected isn’t a control, it’s a hope.

That’s the shape of the fix. What follows is where my own attempt at it stands against those four criteria, honestly, including where it falls short.

There is something you can try now

Tenure is one attempt at this. A deterministic, out-of-model runtime gateway that sits between agents and the APIs they call. It addresses the four criteria above by enforcing action-level granular permissions and evaluating stateful preconditions on the request path, completely independent of the LLM’s internal judgment. It’s in public preview and worth stating where its edges are rather than leaving them to be found by trial and error.

One thing worth spelling out first, since it’s the part that goes beyond just checking that a precondition is true at the moment of the call. Tenure can enforce that a specific upstream action actually happened, and hold the output of that action to a required shape, before a downstream action is permitted to fire at all.

Take a customer email agent. send_refund_email might reasonably require fetch_customer to have run first, since you don't want an agent emailing someone it hasn't looked up. But "the tool was called" is a weak guarantee on its own. An agent could call fetch_customer with a fabricated or wrong account_id, get an error or an empty result, and still proceed to the next step if the gate only checks call order. Tenure enforces both the arguments going into fetch_customer and the shape of what came back: that the call resolved a real customer_id, that it returned a last_email_date, not just that the function name appears in the trace. Only once that output actually satisfies the required shape does send_refund_email become an authorized next call.

That distinction matters because “did you call the right tool” and “do you actually have the information that tool was supposed to get you” are different questions, and an agent under pressure to complete a task will satisfy the first without the second every time it can. Checking call order alone still leaves the agent free to do whatever it wants against that tool. Call it with wrong arguments, ignore a failure response, retry until it gets a shape that merely looks right, as long as the name shows up in the log. Verifying the output closes that gap.

  • The precondition checks have to actually be turned on. Tenure can gate an action on “is this true first,” but that only happens if the policy enables precondition checks for that action. Publish a policy without them, and you get identity and permission checks without the “should this happen right now” layer, which is most of the argument in this piece.
  • An observed policy still needs a human to review it before it’s trusted. Tenure builds a capability matrix from watching what an agent actually calls, which avoids guessing at tool names or arguments. But an observed matrix reflects what was observed, not what should be allowed. Publishing it without reviewing it is closer to the “endpoint enabled, safeguards off” condition in the AISI incident than to real enforcement.
  • Coverage depends on how much you’ve observed. A single agent run won’t surface every capability, argument shape, or edge case the agent is capable of reaching. Fewer runs mean a thinner matrix, and a thinner matrix means more of the agent’s real behavior falls outside what’s been reviewed.

They are the primary ways someone could deploy a system like this and end up less protected than they assumed. If you’re evaluating any tool in this category, Tenure included, those are the questions worth asking before you trust the policy it generates.