Graphcoder - Parallel AI Coding Agent

We went into Graphcoder assuming agent latency was mainly an inference problem.

That lasted until we watched real sessions run. The obvious stalls were not the model thinking. They were the gaps around each turn. A tool would finish, then its result had to get back across the user's connection before the loop could decide what to do next. On a good connection that was annoying. On hotel Wi-Fi it was the product.

OpenAI's WebSocket mode for the Responses API was the first hint that this mattered. Same inference, but OpenAI reported 40%+ lower end-to-end latency on long workflows. We treated that as the starting point.

WebSocket mode makes the transport faster. It does not remove the round trip around each tool call, and those round trips add up:

Path	Round trip per turn	20-turn task
Cloud to cloud, same region	~2ms	~40ms
Laptop on good fiber	~80ms	~1.6s
Hotel or airplane Wi-Fi	up to 800ms	up to 16s

For us, that made the next step hard to ignore: if transport alone could make workflows feel that different, the loop itself shouldn't be trapped behind the laptop's request-response path.

The log is the state

Moving the loop off the laptop buys back latency, but it also breaks the thing local agents quietly rely on: process memory.

A local agent can be simple. The model responds, the agent parses, a tool runs, the result is appended, and the model gets called again. The whole loop lives in one place, so state can hide in ordinary objects, files, buffers, and whatever happens to still be reachable on the next line of code.

Once the loop spans machines, local memory stops being state. Messages, tool calls, file edits, approvals, failures, retries, and partial progress all have to survive reconnects and restarts. Process memory becomes a cache.

Graphcoder keeps durable history first and derives state from it. Formally, that history is an append-only log:

$L = \langle e_1, e_2, \ldots, e_n \rangle$

An event carries what happened and what it depends on:

$e = (\text{id}, \text{type}, \text{payload}, \text{deps})$

A tool result is not state. It is one record in the history. So is a user message, a file edit, or a worker finishing. State is what you get after replaying the prefix you have seen:

$S_t = P(L[1..t])$

where $P$ is the projection for the view you care about.

The same history can produce different views. The UI reads it one way. The filesystem reads it another way. The agent reads it into the context it needs for the next model call. One source of truth, several projections.

A horizontal row of event cells e_1..e_n with arrows pointing up to UI state, filesystem F_t, and agent state S_t. — One log, several projections. UI state, filesystem state, and agent state are different reads of the same history.

Sharding the log

The first version of this idea is one log. That is the design you want if it holds: one append path, one replay order, one place to debug.

It does not hold for long. Once many agents are running, unrelated events start queueing behind the same sequence number. The log becomes a bottleneck.

So Graphcoder shards the log by owner:

$\text{partition}(e) \rightarrow k$

Each shard is still append-only:

$L_k = \langle e_{k,1}, e_{k,2}, \ldots \rangle$

That fixes writes. It also means reads can no longer say "replay the log." There is no single log anymore.

Instead, a projection starts from the event it cares about and follows dependencies backward:

$C(e) = \text{closure}(\{e\}, \text{deps})$

A read is valid when that slice is closed:

$\forall e \in C,\ \text{deps}(e) \subset C$

So the projection gets the history it needs, not the history that merely happened nearby. That is the reason sharding works for an event log rather than turning every read into a distributed replay.

Three parallel shard logs with one root event highlighted and dependency arrows selecting only the transitive closure needed by that projection. — Highlighted events are the dependency closure for one root event. Gray events are real history, but this read does not need them.

Running ahead of authority

Once the log is authoritative, the easy mistake is making the UI wait for it.

Without speculation, every user action takes the slow path: client to authority, authority back to client, then paint. That is correct, but it spends a round trip before showing the user something the client already knows.

If the client has seen prefix $L_c$ and submits intent $i$ , it can render:

$S' = P(L_c + \text{speculative}(i))$

Graphcoder treats local intent as a temporary tail on confirmed history. The UI stays explicit about the difference: confirmed prefix first, pending intent at the end.

Confirmed prefix L_c as a solid bar followed by a hatched speculative tail past the authority cursor, with the three retirement outcomes (harden, rebase, rollback) shown as badges. — The client projects past the authority cursor. The speculative tail eventually hardens, rebases, or rolls back.

The log is still the authority. When the authority catches up, the tail hardens into history, rebases over a different fact, or rolls back.

That optimism does not stop at the UI. Graphcoder can speculatively prefill likely model prompts before every dependency has landed. It can start fetching filesystem materialization paths before the user asks for them. In both cases, the system begins from facts it already trusts and lets authority settle the result later.

What this changes

Faster inference still matters. It just does not fix the pauses we kept seeing in Graphcoder.

The bigger win was moving the loop out of the laptop's request-response path. The laptop becomes a live view over durable remote execution, not the machine that has to drive every turn.