I've been following the recent situation with @Railway about their agent deleting a whole production volume with backups: https://x.com/lifeof_jer/status/2048103471019434248. The post describes what happened, but here's a short summary:
The agent was working on a task in a staging environment.
It saw an error, and for some reason decided to look for the wrong API key.
It found a production API key and decided to call a destructive POST endpoint with a delete operation on a production volume.
What strikes me wrong is the response from Railway: "Oh my. That 1000% shouldn't be possible. We have evals for this."
This is a huge problem, and a very bad engineering pattern that we seem to develop when it comes to agentic development.
LLMs are a probabilistic engine. There is no such thing as 100% probability in ML world.
Heck, there is no such thing even in a plain, old computing world. And if something might happen, on a large enough scale it eventually will.
Let me give you an example.
Do you know about bit flips? You might've heard it in your CS course, but here's a refresher:
There is a constant level of background space radiation around us. It is mostly negligible, but sometimes it can cause a tiny electric spike, which is just large enough... to change a value of a bit inside your computer.
It could happen in your memory, CPU, network card, bridges.
Think of it. No matter how correct your code is, no matter how many verification layers we have in the network stack - your app can still receive an invalid value. Just because of pure physics.
This almost never happens for personal computers. But on a Meta scale it happened *constantly*. We had SEVs caused by it, internal investigation tools and large enough runbooks to handle those.
You can't just assume that something can't happen.
Everything is a probability. Every branch of code will be entered. Given enough time everything can happen, and you need to be ready for that.
Now, LLMs introduce another surface of probabilities. LLMs can create branches on the fly - like ants in a maze eventually they will reach the bounds of their container.
If you want to *really* prevent LLMs from doing a certain action, forget evals - remove the capability. Here are my own rules:
Agent Safety Runbook
This runbook follows the principle of least privilege.
Sandbox
The agent sandbox must be securely isolated from the main server. Sandbox resources must be limited if you want to prevent the agent from using too much compute.
The agent must run under a non-root user, and the sandbox must have no setuid binaries accessible.
The sandbox FS should only have the necessary files and tools for the task.
Do not share raw secrets to the agent sandbox.
You can allow the agent to create or install tools, but you must make sure that it can't access tools that you have forbidden.
Network
You need to proxy all network requests from the sandbox.
APIs, package manager repositories must be filtered and restricted to prevent supply chain attacks.
Agent
LLM calls must be proxied to control the rate limiting and token cost.
All data that the agent read can be treated as instruction. The agent tools must have hooks to guard against that and if you really want to make sure you can add a classifier.
All agent actions must be logged.
Reading Files
If you don't want LLM to read a certain file there are two layers of securing it:
Don't upload the file to LLM sandbox if possible. Replace it with a stub, if the file needs to exist.
Add a hook on all tools that prevents reading from a certain path. Make sure to cover any script execution.
Writing Files
Again, if possible don't share the path where you don't want LLM to write.
Restrict the path on a file system permission level. Run agent under a non-root user with no sudo or su access. You might want to remove all setuid binaries - I can perfectly imagine the agent using the copy.fail exploit to escalate privileges.
Reading data from API
What do we want to prevent in this case:
Reading restricted data
Overloading the remote API
All remote API calls should go through a proxy that uses a temporary, scoped API key.
The proxy must handle rate limiting.
Pushing data to API
The agent must not have access to destructive API. So same rules for scoped API key and rate limiting apply. But on top of that:
All requests must be idempotent and retryable. You have to ensure data validity on the API level, not on the client-side.
Destructive operation that is not a part of agent workflow shouldn't be present in API accessible to the agent.
Destructive operation that is part of the agent workflow should be trivially reversible if possible.
Destructive operation that is part of the agent workflow that can't be reversed must not be available for execution in the API accessible to the agent. The agent can submit the operation, but only the user can start the execution - which can be non-agentic and completely separate.
I'll keep updating this runbook, and I invite others to contribute to it - I think we should uphold the reliability standards in AI industry.