ChatGPT Web as an Agent Runtime

Parallax is an experimental coding harness I've been building for the past couple of months whose "brain" is the ChatGPT website.

When you type a task into the Parallax desktop app, instead of sending a request to an API, like an ordinary harness, Parallax opens up a chatgpt.com tab, types in your request, takes the commands that ChatGPT produces, runs them on your computer and sends the results back to the ChatGPT tab.

This loop repeats until your task is done. I'll walk you through the core technical details of how this works in this note, and then explain why Parallax isn't available to the general public and why you shouldn't attempt to build something similar.

But first, here's a video of Parallax in action:

Core Architecture

Parallax is two things: a desktop app and a Chrome extension. The desktop app runs actual commands on your computer, while the Chrome extension runs the ChatGPT tab but can't touch anything on your computer. The ChatGPT website itself knows nothing about Parallax and only sees someone sending messages.

The desktop app has two processes: a renderer, which displays the UI, and a main process, which runs the shell commands. The Chrome extension has a script that runs continuously in the background, called a service worker, that connects to the desktop app over a WebSocket and remembers which thread in the desktop app corresponds to which tab on the browser.

If you're not familiar with WebSockets, think of ordinary web traffic as like sending a letter: your browser sends a request (like "give me this page"), to which the server sends back a single response (the page), and the exchange is over. In this paradigm, the server can never start a conversation, and only replies.

A WebSocket is more similar to a phone call in that one side calls, the other side picks up and then the line stays open. Either side can talk at any moment, as many times as they want, until one of the parties ends the call. Each message is a small chunk of text, and for Parallax specifically each chunk is always a JSON object with a type attribute.

Parallax needs WebSockets because the extension needs to continuously stream the model's response to the desktop app many times a second, while the desktop app needs to also tell the extension to send the user's message to ChatGPT. This is a two-way ongoing conversation (full-duplex), which is exactly what WebSockets are for.

The desktop's main process opens a WebSocket server at localhost:8765, which the extension's service worker connects to at ws://localhost:8765. This lets them trade messages like this:

desktop → extension   { "type": "send_message", ... }   type this into the thread's tab
extension → desktop   { "type": "stream_update", ... }  here's the answer so far
extension → desktop   { "type": "response", ... }       the answer is finished
extension → desktop   { "type": "turn_status", ... }    which turns are still running

To prevent Chrome from shutting down the extension's background script when it's idle, the desktop app pings the extension every 10 seconds, while a timer wakes up the extension's service worker every 30 seconds to reconnect if the connection dropped.

Apart from the service worker, the extension basically boils down to two .ts files: a content script, content.ts, which reads and clicks the page, and a page script, inject.ts, that is injected into the page's own JavaScript and runs as if it were ChatGPT's own code, which lets the extension do things like natively edit the text editor and read the tab's network traffic.

To do this, the content script puts a <script> tag into the page, which loads inject.ts, the page script, into the tab's own JavaScript. Then, the content and page scripts can communicate with each other using window.postMessage, which allows the content script to type in a particular piece of text, and also receive the model's response as it is being streamed back to the website.

For most extensions, Chrome ensures that an extension's code is completely separate from a website's code. This ensures that they run in a walled-off space, so that they can see and click everything on the page but can't touch the page's own JavaScript. However, Parallax needs two things that can only be done by editing the page's own JS:

1. ChatGPT's text editor. The input bar is a ProseMirror editor, and the only way to reliably put text in it is through editing its internal JavaScript object, called EditorView, which the extension's page script finds by digging through React's internals and replacing the input bar's contents directly.
2. ChatGPT's network traffic. When OpenAI's servers send the answer to your browser, the answer arrives as a stream over the network and in order to show this response as quickly as possible on Parallax's desktop app, the extension's scripts swap in its own versions of the browser's fetch, EventSource and WebSocket for the page so Parallax's code sees each piece of the answer as OpenAI sends it to your browser and copies it out.

I cover some of these in more detail in the Driving the ChatGPT tab section.

Driving the ChatGPT Tab

For Parallax to work, it needs to be able to type into ChatGPT's input bar programmatically and consistently. Simple approaches like just pasting the text into the bar don't work because when the text is too long, the website turns it into an attachment, thereby sending a file instead of a prompt. The other approach I tried, setting the textContent DOM property, only changes the text on screen but actually leaves the internal state of the input component unchanged, meaning that ChatGPT never sees the message.

To fix this, you would need the extension to edit the input bar natively, in the same manner that ChatGPT's own code does. The input bar keeps its contents in an object called the EditorView, and every change, such as someone typing, goes through this object's dispatch function. So, if an extension can locate the EditorView component and its dispatch function, it can replace the contents itself, which ChatGPT would treat like any other edit made by a user.

This isn't so obvious because the site's bundler (something like webpack, which combines all of a site's code into a few files) keeps it in a local variable with no global name that the extension could use to just reach it. However, since ChatGPT is a React website, we can still reach this EditorView object through something called fibers. For every component that React renders, it keeps an internal record called a fiber, which holds that component's props, state, refs, and also a link to its parent's fiber.

React saves each fiber on the HTML element that the React component rendered, under a property whose name starts with __reactFiber$. So, this means that the React component that wraps the input bar keeps a pointer to the EditorView, so following a path from the input bar's element up through the fibers of its parent components leads to it. While this isn't a documented API by any means, it's all still reachable from the page's own JavaScript, which is where our extension's page script runs, so Parallax finds the reference by essentially traversing a series of fibers, using the following code snippet:

let element = node;
let fiber = null;

for (let hops = 0; hops < 12 && element; hops++) {
  const fiberKey =
    Object
    .keys(element)
    .find((k) => k.startsWith('__reactFiber$'));

  if (fiberKey) {
    fiber = element[fiberKey];
    break;
  }

  element = element.parentElement;
}

for (let depth = 0; fiber && depth < 40; depth++) {
  /**
   * Check the three places a component can keep the
   * EditorView: the object behind the fiber, the
   * component's hooks, and the props it was given.
   */
  const inObject = viewWithin(fiber.stateNode, node);
  const inHooks = viewInHooks(fiber, node);
  const inProps = viewInProps(fiber, node);

  if (inObject || inHooks || inProps) {
    return inObject || inHooks || inProps;
  }

  // Not here, so climb to the parent component
  fiber = fiber.return;
}

The search starts at the input bar's HTML element with id #prompt-textarea. Since this component was created by ProseMirror, a component library written in plain JS, rather than by React, it has no fiber of its own, and the extension checks its parent elements, up to 12 levels, for the first one that does. From that fiber it climbs up to 40 levels through parent components, checking each one's state, refs and props for an object that looks like an EditorView: something with a dispatch function, a state, and the input bar as its element.

Once it has the EditorView, the page script types the message by replacing the input bar's contents through the EditorView's dispatch function, with the following code snippet in inject.ts:

const paragraphs = String(text).split('\n').map((line) => (
  paragraph.create(null, line ? state.schema.text(line) : null)
));
const tr = state.tr.replaceWith(0, state.doc.content.size, paragraphs);
view.dispatch(tr.scrollIntoView());

That is the same path ChatGPT's own edits take, so the send button turns on as it would for a person typing, and a long message stays text instead of becoming a pasted-file attachment.

After the message is sent, the page script reads the reply. It wraps fetch, EventSource and WebSocket, tees every event stream the page opens, and rebuilds the reply from the small edits that ChatGPT streams, then passes it on to the desktop app. Messages whose content type is thoughts, reasoning_recap or similar are skipped, which separates the model's reasoning from its output.

Prompt Grammar & Hierarchy

To actually send your message to the website, Parallax wraps it in {plx:task} tags ({plx:task} YOUR MESSAGE {/plx:task}), appends its system prompt, and sends it all as one ordinary chat message. So for Parallax, the system prompt is just more text in your message and has none of the priority a system prompt actually gets through an API. This is also why I had to experiment with different system prompts to convince the model to call tools in the format I wanted instead of the one OpenAI had set for chatgpt.com.

When I asked it to look through a repository, the model would routinely open its own code interpreter, list an empty sandbox, and report that the project was empty. The system prompt therefore has to explicitly prohibit ChatGPT from using its built-in tools:

Do not use native tools, a code interpreter,
or another terminal. They cannot access the
mounted workspace. Do not claim you lack access
or ask for uploads; emit Parallax actions.

Every reply by the model is either an ACT, which is one {plx:note} followed by one to four command line instructions wrapped in tags, or an ANSWER, which is a single {plx:done}, with no prose allowed outside the {plx:*} tags. The desktop app finds the shell commands it needs to run by parsing the model's reply for these tags, which is also why they use braces instead of angle brackets ({plx:note} instead of <plx:note>), because ChatGPT's renderer reads paired tags (<plx:note></plx:note>) as HTML elements and strips them from the page.

In Fig. 4, you can type into a model reply and see which notes, commands and answers the desktop app's parser pulls out of it.

The model doesn't always follow the format either. It still writes <plx:note> with angle brackets some of the time, and often leaves a note or a final answer unclosed. So before anything else reads the reply, the parser rewrites angle-bracket tags into the brace form, closes an open note where the next tag starts, and closes an open answer at the end of the reply.

The results of tool calls are sent back as {plx:result} blocks in a single message that you never see in the desktop app, but can see on chatgpt.com. ChatGPT refuses messages over a certain size, which would leave the user's task stuck, so the results are cut to fit. Short results are kept whole and only the long ones are shortened, so if one command prints an enormous file and fifteen others print a few lines each, only the long one gets cut.

In Summary

One turn of conversation in Parallax is therefore a hodgepodge of the many different interactions between a desktop app, a Chrome extension and a website. The diagram below neatly summarizes everything that happens when you send a single message.

{plx:task}
look through this project
{/plx:task}

You are Parallax's planner in a desktop coding harness. …

You could definitely take this further. It's easy to see how common harness features such as auto-reviewing commands, a model picker and sub-agents could be added to Parallax. You can't, however, overcome the system's inherent fragility.

Parallax completely relies on ChatGPT's internal behavior, which is subject to change at a moment's notice and, obviously, has no public-facing documentation. This means that any update to the website could break Parallax. If ChatGPT moves the EditorView out of the component that is holding it today, the fiber search would climb 40 levels, find nothing and render the extension unable to type and halt the entire loop. If ChatGPT were to add new markers or tags to their prompts, our parser would stop working and the loop stops again.

Parallax isn't available for use, and I'd recommend against building something like it. I built it purely to tinker with WebSockets and streaming architectures.

OpenAI's Terms also forbid "automatically or programmatically extracting output" from its product offerings, which is exactly what Parallax does, so using Parallax would put your ChatGPT account at risk. The OpenAI API, and the harnesses built on top of it, like Codex, do Parallax's job much better through an architecture that's actually meant for it.