Press enter or click to view image in full size
Our storage bill was upside down.
We run hundreds of dedicated AI-agent VMs. Each agent has its own /workspace, where chats, files, cloned repositories, and credentials need to survive a restart. The normal answer is a persistent disk per machine. We did that. It worked.
Then we looked at the invoice.
We had provisioned roughly 900 GB of persistent disk across a fleet that spent most of its time asleep. Those disks cost $137.93 in a $152.93 month. In other words, storage was about 90% of the bill, and about nine times the compute cost.
That was the moment we stopped asking, “How do we make disks cheaper?” The better question was: why does an idle machine own anything durable at all?
So we deleted every persistent disk.
Today, each agent’s workspace lives in its own private, snapshot-enabled object-storage bucket. The VM is an ephemeral cache with a job to do. It boots, hydrates /workspace, runs the agent, syncs changes back, and can disappear without taking data with it.
The result is a useful lesson in object storage vs block storage: it is not a religious choice. It is an access-pattern choice. And for per-tenant workloads that sleep for long stretches then wake up for short bursts, provisioned disks are often charging you for a lifestyle your software no longer has.
Block storage charges for the capacity you reserve. Object storage charges for the bytes you keep. If your machines are mostly idle, that difference can become the architecture.
The boring diagram that changed the whole design
Press enter or click to view image in full size
The control plane orchestrates. It does not sit in the data path. Each machine talks directly to its own bucket, using credentials that cannot see anyone else’s data.
That separation mattered more than we expected.
Fly’s own documentation describes a Machine root filesystem as ephemeral and says a volume is tied to one physical server and attached to one Machine. That is a good trade when you need low-latency, durable local state. It is a bad default when the machine is mostly a sleeping shell around a few hundred megabytes of tenant data. Fly’s volume overview is unusually candid about both sides of that trade.
The application never learned it had durable storage
Our first instinct was to build syncing into the agent application. We did. It was the wrong abstraction.
An AI agent does not only write through our application code. It spawns shells, clones repos, invokes tools, runs headless browsers, and occasionally does something inventive at 2 a.m. Any application-level sync layer sees only the writes that application happens to make. That is not a durability guarantee. That is selective memory.
The durable layer now lives in a roughly 400-line bash entrypoint that runs as PID 1. On boot, it hydrates /workspace with rclone. It starts the agent as a child process. While the agent runs, it performs a lock-guarded incremental sync every 45 seconds, plus another sync after a completed task.
The entrypoint is deliberately storage-aware. The agent is deliberately not.
That means every writer gets durability for free. It also solves the bit that application code cannot solve: the final sync. When the agent exits, PID 1 is still there.
Press enter or click to view image in full size
That last box is the whole game. We give shutdown 300 seconds. The agent gets at most 120 seconds to finish in-flight work and close its browser. The final sync gets 150 seconds, including a retry. The remaining slack is intentional. It turns “we usually get the last writes” into a time-budgeted contract: RPO is zero for a clean shutdown.
A hard kill can still lose up to a 45-second interval. We say that out loud. “Zero data loss” is how you end up explaining data loss later.
A 39,000-file workspace taught us that S3 timestamps have a price
The first no-op sync of our largest workspace took two and a half minutes. That workspace had 39,000 files. A clean shutdown budget does not care that most of those files are unchanged.
The culprit was a small, expensive assumption. rclone’s normal change detection compares modification times. In S3-compatible object storage, the original file modification time often exists as object metadata, which can mean extra HEAD requests during comparison. Do that thousands of times and “nothing changed” becomes a very active sentence.
We switched to --checksum. For our setup, it compares a local MD5 to the ETag already present in bucket listings, avoiding those extra requests. No-op syncs fell to seconds. The worst cold hydrate, once tuned, came down from about ten minutes to 74 seconds.
This is not a universal rclone recipe. Multipart ETags, encryption, and provider behavior can spoil the neatness. Test against your provider and your file distribution. rclone documents --checksum as a comparison option, but the real rule is simpler: measure the request pattern, not just transfer throughput. The rclone documentation is where the flags live. Your shutdown window is where they become architecture.
One bucket per agent is a cost decision with security consequences
There is a second reason this worked so well for AI agents.
Agents are adversarial-input workloads. They consume prompts, web pages, README files, dependencies, and tool output. Eventually one of those inputs will try to persuade an agent to misuse whatever it can reach. You can call that prompt injection, compromised dependencies, or Tuesday. The label matters less than the assumption.
So every agent gets one bucket and one access key. The key has permission to list, read, and write exactly that bucket. No broad storage account credentials exist on agent machines. Administrative credentials remain in the control plane.
Now the security question becomes refreshingly concrete: what can a fully compromised agent do?
It can read or delete its own tenant’s workspace. That is serious, but it is contained. It cannot enumerate another tenant’s bucket, fetch another tenant’s repository, or overwrite another tenant’s files. Those requests return 403.
We do not treat that as a policy statement. We treat it as a test. Our isolation audit uses agent A’s live credentials against agent B’s bucket for random pairs, tries List, Get, and Put, and expects every cross-tenant action to fail while a positive control proves A can still access A.
Press enter or click to view image in full size
That is the kind of guarantee I like: one you can run as a command, rather than one you hope somebody remembered when they edited an IAM policy.
Recovery needs to survive a well-behaved sync loop
A sync loop is very good at replicating mistakes.
If an agent deletes a directory, the next sync can faithfully make that deletion durable in under 45 seconds. That is why we use snapshot-enabled buckets. Tigris snapshots provide a consistent, read-only bucket view, and its dashboard supports listing bucket contents at a chosen timestamp. Earlier states remain available even after live objects are deleted. Tigris’s snapshot documentation also recommends out-of-place recovery, which is the safer default here.
Our recovery pattern is intentionally dull: list the bucket as it existed just before the bad write, copy that state into a restored/ prefix, inspect it, then promote the files back. No panic edits in production. No attempt to rewind a live workspace while someone is still using it.
[Inline illustration placeholder]
Image-generation prompt: Minimal technical recovery diagram, a corrupted folder icon on the left, a horizontal timeline with a highlighted timestamp, an immutable object-storage bucket history in the center, and a clean restored folder on the right, dark mode, restrained cyan and amber accents, no text, 3:2.
Snapshots do not magically make history free. Tigris says snapshots themselves add no charge, while new object versions stored are billable. Current standard storage is listed at $0.02 per GB-month. Its pricing page is worth checking before you publish because prices are allowed to change, unlike old invoices, unfortunately.
The second half of the savings is deleting idle machines
Removing disks stopped the provisioned-capacity bleed. Reaping machines finished the job.
A stopped VM without a unique disk is just inventory. After seven idle days, our reaper deletes it. When the customer returns, the control plane creates a new machine from the pinned image and the entrypoint hydrates the workspace. Typical workspaces wake in seconds. The biggest one takes around a minute. That is a real tradeoff, and some workloads will reject it.
But for a registered customer who has not used their agent in weeks, the residual cost is bucket bytes. Approximately zero compute. No dead machine parked forever because it happens to be attached to something expensive.
This pattern is not for databases, write-heavy workloads, or anything that needs local-disk latency on every request. We also enforce a 6 GB workspace guard because the data must fit on the ephemeral root filesystem. And snapshot history can become a problem for high-churn, long-lived buckets without an eventual compaction plan.
Still, there is a broad lesson here.
We used to think persistent disks made our agents durable. They made our machines expensive and physically sticky. The bucket is what made the data durable. Once we separated those two concerns, the machine could finally become what cloud compute has been promising all along: temporary.
Not disposable in the careless sense. Disposable because nothing irreplaceable lives there.