· Andrew Arrow · Updated
Move "Join the waitlist" to the top of our little editor, hit Save, and everything looks fine. The button is still there. The editor says "Saved." Reload the page and the button disappears. Three blocks went in; two came back.
We planted this bug, then wrote a correct patch and a convincing fake. To check them independently, we wanted three copies of the prepared app, with dependencies already installed. That raises a question a Git checkout doesn't answer: are we copying the files, or the running application?
We tried Mainbrella, E2B, and Daytona. We build Mainbrella, so one of these is ours. All three caught the fake editor repair. A second bug made the difference between their copies matter: its prepared work lived only in a running worker's memory. No coding agent attempted these repairs.
After Save
After reload
A fix has to survive reload
You can try the broken editor without an account. Its serializer assigns positions from the array index, then filters them:
blocks
.map((block, position) => ({ ...block, position }))
.filter(block => block.position);Zero is falsy. Goodbye, first block. The storage write succeeds, so the UI announces "Saved" and keeps showing the untouched in-memory draft. JavaScript has done exactly what we asked, which is rude but fair.
The correct patch removes the filter. The misleading patch calls render() after saving. It redraws the same healthy-looking draft without reading the stored document back. A video that ends at Save can look great with either patch. Reloading makes the fix earn its keep.
Our browser test checks ordered block IDs and text before Save, after Save, and after reload. Searching for "Join the waitlist" somewhere on the page would accept help text while the document had already lost the button.
The verifier reports passed, reproduced, or unverified. Reproducing this bug requires the intact document before reload and exactly that document minus its first block afterward, with the remaining order and text unchanged. An HTTP error, timeout, skipped test, or unrelated assertion is unverified. If the prepared baseline doesn't reproduce the loss, we stop before trying patches.
| Copy | Browser checks | Document after reload |
|---|---|---|
| Unchanged | 5 failed, 1 passed | First block lost |
| Correct patch | 6 passed | All blocks preserved |
| Misleading patch | 5 failed, 1 passed | First block still lost |
The passing baseline check tests Reset: it must restore the demo without deleting an unrelated browser-storage key. The other checks cover reordered blocks, the original order, edited text, both reload paths, and six shuffled documents generated from a recorded seed. Each holdout document goes through two reloads. These are controls with known answers, not an agent benchmark.
The verifier runs outside the writable app copies. A proposed repair can't fix its score by editing the test. This is the first workload for Repo Rodeo, our harness for giving repair attempts independent environments. Before handing it to agents, we wanted to know whether it could reject a patch we already knew was wrong.
What came out of the copies
The editor runs used the same pinned application archive, patch hashes, browser checks, and seed. Each provider prepared one source and made three branches. The checkout and installed dependencies survived in all nine branches; none needed another npm ci. After the controls finished, each branch kept its own marker and expected source hashes, while the source had no branch marker. The evidence index links the newer matched runs as well as the worker controls below.
Getting to that point exposed mistakes in our setup. Mainbrella forwarded the guest Host header as container, which Vite rejected with 403 until the launcher allowed that exact host. E2B's base template selected an older Node through its login shell; our launcher needed an explicit PATH to the pinned binary. Then our browser helper mishandled authenticated redirects. A ready server and a usable browser preview were separate things to check.
We kept those failures in the setup notes. Six red tests on a forbidden page tell us nothing about the button. Once the preview worked, the more interesting question was whether a copied server was already listening.
A PID alone can't establish that: a fresh process can reuse a number. Our launcher generates a random nonce once at startup and serves it from an uncached endpoint. It never loads the nonce from disk. A branch answering with the source's nonce and PID before we launch anything has inherited that process state.
All three providers gave us listening copies, through different startup paths. Mainbrella's row below comes from a later production check of the same pinned app with a saved launch recipe. We restored three unpatched copies without rerunning the browser repair controls. E2B and Daytona use the original editor runs.
Mainbrella's original run needed our runner to start each restored server. With the saved recipe, all three copies listened and served the editor over their protected previews without a client-side launch command. Each had a fresh nonce; the source kept running. All 417 tracked file hashes matched in every copy, along with dependencies, the prepared marker, permissions, and a symlink. The original process records retain the earlier PID, nonce, and file observations.
E2B brought the source's listener, PID, and nonce into all three forks before we launched anything. Its fork documentation also says capture pauses the source and drops active connections. A process remembering its work doesn't mean the terminal watching it stays connected.
The console showed the source and three branches together: four concurrent sandboxes, with htop open in the source.
Daytona's container snapshot required a stopped source. After capture, the source and all three branches needed fresh servers. Its successful rerun preserved files, dependencies, and branch isolation and produced the same repair outcomes as the other two. This tests Daytona's container snapshots; its VM memory snapshots are a separate mode we haven't exercised.
The historical settings panel identifies the sandbox as class Container and its preview as private.
For the editor, restarting was fine. The document lives in the external browser's localStorage, outside the server container. Each browser test reconstructs it. We even stopped E2B's inherited servers before checking the patches. That experiment demonstrated process preservation, but the repair never used it.
A worker you can't restart into correctness
To make the process state matter, we added a small worker with five accepted jobs held only in its JavaScript heap. Two jobs represent free items: their unit price is zero, but they still need receipts. The worker pauses before processing the batch. That paused instance, including its queue, is what we copy.
The faulty processor filters jobs by unitCents before calculating totals. It drops both free items and returns three receipts with a completed flag. The correct patch removes the filter and returns all five receipts. A misleading patch returns an empty output list and claims completion. If we let that badge decide the result, every version would pass.
The outside verifier submits the jobs once and keeps its own ledger of IDs and expected amounts. It never sends them again to a branch, and the worker never writes them to disk. Before resuming, we check the preparation ID, queue fingerprint, pending count, PID, and nonce. Afterward, every expected receipt must appear exactly once with the right amount, and the same process must have handled the same prepared queue.
There's a deliberate constraint here. The worker imports processor.mjs when we release the checkpoint, so replacing that file doesn't require restarting the process holding the queue. We aren't hot-patching a function halfway through execution. Without an explicit reload point, a copied process could faithfully keep executing the old code.
E2B's three branches inherited the five pending jobs. The unchanged processor lost the two free items, the correct patch returned all five receipts, and the fake returned none. Mainbrella and Daytona preserved the processor files but started empty workers. Applying even the correct patch to those copies couldn't test the accepted batch: the reproduction was already gone.
The table shows one run per provider: receipts out of five expected, followed by the verifier's verdict. The selected runs use the same seed, 12346, Node 22.22.2, and identical worker, patch, runner, and verifier bytes. Each provider has its own preparation ID and source nonce.
Restarting the source supplied that last control. With a new nonce, no preparation ID, and no pending jobs, the correct processor reported completion over an empty queue. The verifier called it reproduction-lost. It couldn't distinguish the good patch from the bad ones using work that no longer existed.
We also gave each branch a different in-memory marker and checked the other branches and source. Each kept its own marker. In E2B, the source still held the original five pending jobs after all three branch repairs finished. The copied queues could be consumed independently without draining their source. The E2B, Mainbrella, and Daytona observations include the full nonces, fingerprints, input ledger, and returned receipts.
One earlier Mainbrella repetition stopped with HTTP 429 while checking the cold restart. Its retained command history had filled up. We raised that limit while keeping separate output and event budgets, then reran the same seed against production. The run completed and cleanup confirmed all four containers and the workspace absent. The run history retains the incomplete attempt too; the successful rerun doesn't erase it.
A separate automatic-launch check restored a listening worker with zero pending jobs, while its source still held five. Our underlying Cloudflare snapshots exclude memory and running processes. A startup recipe can't recover this worker's accepted jobs.
This is a constructed workload with a convenient checkpoint. A durable queue or replayable input log could rebuild the batch after restart; we deliberately left that recovery path out. Live cloning helped because the three candidates could consume independent copies of work already accepted into memory. For the editor, the external browser could recreate the document, so filesystem restores were enough.
Finding work after losing the connection
A preserved process still needs a way for a new client to find it. In the matched editor follow-up, we started an eight-second job, detached after its first output, then connected with a new client. All three providers recovered the same job identity, exit status zero, and remaining output. The mechanisms differed: Mainbrella replayed an SSE event cursor, E2B recovered command output using a byte offset, and Daytona recovered retained session output. Daytona's check detached a polling client; it didn't test a broken live stream.
Losing a capture reply leaves a different problem. The cloud may have saved a workspace while the client never learned its ID. Blindly retrying can create another copy and leave the first unclaimed.
On Mainbrella, we deliberately discarded the body of a successful capture response. Before sending it, the runner saved the exact request body and idempotency key. Retrying with both recovered the same workspace; a further replay returned it again. The retained production result records three requests producing one workspace and one capture reservation.
The lost reply came after successful HTTP headers. This establishes recovery of a completed save, not what happens if capture is interrupted earlier. We didn't run keyed replay on E2B or Daytona: those adapters don't implement it. That is a gap in this comparison, not a demonstrated failure of their services. We disabled keyed replay for all three matched runs so their shared recovery check covered the same interrupted job.
Follow-up: recover the save by name
We then tested E2B and Daytona using their supported named-snapshot recovery paths. Before submitting, the client persisted a unique snapshot name. The harness received a successful save response, retained it privately, and withheld it from the recovering client. E2B used a new SDK client to list snapshots by the saved name and source sandbox. Daytona polled the saved name until the snapshot was active. Each recovered snapshot restored into a new sandbox with the prepared file intact, and each run issued exactly one capture POST.
| Provider | Recovery | Capture count |
|---|---|---|
| Mainbrella | Replay the same request key | One capture; three POSTs |
| E2B | List by persisted name | One capture; one POST |
| Daytona | Poll persisted name | One capture; one POST |
Both follow-ups completed cleanup, confirmed the two test sandboxes and saved snapshot absent, and preserved the original account inventories. E2B used a full snapshot and Daytona a container filesystem snapshot; Mainbrella's earlier check used the prepared web app. These are separate single-run behavior checks, not a matched reliability or cost benchmark. We did not resend the competitor capture requests or assume an undocumented idempotency header. Mainbrella demonstrates keyed replay; all three demonstrate recovery of a usable save after its reply is lost.
Checking what deletion actually did
Our first Daytona snapshot exceeded the harness's three-minute deadline while the provider kept working. A later delete returned 403: our key could manage sandboxes but couldn't delete persistent snapshots. We had permission to make the mess and insufficient permission to clean it up.
A screenshot from that first attempt shows the snapshot still marked Snapshotting.
The adapter now checks both sets of permissions before provisioning, saves the requested snapshot name before submission, and polls separately with a ten-minute deadline. The rerun completed the snapshot and all repair controls, then deleted its sandboxes and snapshot while preserving thirteen pre-existing snapshots.
Yet its overall result said failed. The API confirmed deletion, but several old previews returned 401 instead of the 404 or 410 the adapter required. Meanwhile, E2B passed a special rule allowing repeated 502s after API-confirmed absence. It also got more probes. Those labels reflected different acceptance rules in our code. Using them to rank providers was a mistake.
We reran the editor with a shared cleanup protocol. Every current preview had to return 200 with its credential before deletion. After cleanup, we checked exact API resource absence and made three rounds of requests to each old URL, with and without preview headers, with a two-second pause between rounds. We also provisioned a small live control server after deleting the editor resources. Its preview had to stay at 200 during each of three further rounds, so we could check whether the gateway still served a known-live sandbox.
All three completed the expected repair outcomes and restored their original API inventories. All three live controls stayed at 200. E2B's gateway could serve a live sandbox during the old URLs' 502s, but those responses still don't prove that the old token was independently invalidated. A deleted sandbox with an unusable URL and a revoked credential are different observations. The E2B, Mainbrella, and Daytona records preserve every response and its control.
Removing headers removed the E2B and Daytona tokens. Mainbrella's credential lives in the hostname, so its no-header requests still carried the same bearer credential. Daytona separately demonstrated token rotation at a live URL after restarting the source: the old token returned 401 while its replacement returned 200. That check establishes something the deletion probes alone couldn't; the protocol notes explain the credential transport and retained source.
None of these checks establishes physical storage reclamation. Our earlier local Mainbrella run left unattached Docker snapshot images after API deletion, with collection time unknown. Production capture sizes, disk reservations, monthly accounting counters, and actual charges also need separate measurements. The accounting notes retain the numbers; API absence isn't proof that the bytes or the bill have disappeared.
Which capability the workflow needs
The newer matched editor and worker runs pin Node 22.22.2; the historical editor runs used different guest versions. Allocations still differ, and the small number of controls supports no reliability, speed, or price ranking. Repairs are supplied patches. We haven't established autonomous agents repairing arbitrary repositories.
The public evidence includes document observations, worker inputs and receipts, checksums, and source provenance. Raw browser traces and API receipts stay private because they can contain bearer preview credentials. The editor bundle cannot independently replay the full original verification decision. The retained worker source and local controls let you inspect the checkpoint and reproduce restart loss without a cloud account.
For our unattended workflow, I prefer a persisted request key that can recover a completed capture after its reply disappears. That's the Mainbrella behavior we demonstrated separately. For the prepared worker batch, I'd use E2B's live fork: it preserved the work the repair had to process. Daytona's container snapshots met the editor's filesystem requirements. The editor's correct patch had to survive reload; the worker's correct patch needed five accepted jobs left to run.