Running DOOM Inside a Transformer

19 min read Original article ↗
DOOM frame 151 being painted from the transformer's framebuffer writes.
Code here

In my last post, I trained a world model that attempts to play Pokémon Red. The model learned to do this through its latent space, and used the learned predictions to plan actions in the game. But the game itself still ran in a Nintendo GameBoy emulator.

That made me ponder: could we put the game inside the model? Could a transformer calculate where the player moves, whether a shot hits an enemy, and what color each rendered pixel should be?

I decided to go for the meme–and see if I can have a model that plays DOOM directly in itself. The goal: insert DOOM into a transformer-based model. The model’s weights would be the computation substrate itself, and the DOOM program and its game data would be the prompt. As the model generates tokens, the weights execute the game.

I also wrote a compiler/library so that people (or their agents) can do the same thing. It is called firebrand. It allows you to take a program and turn it into weights. It should1 be able to take arbitrary state machines and compile them into a neural network.

But before we can get to DOOM, we have to add two numbers.

Choosing the weights ourselves

A simple neural-network layer

A simple neural-network layer computes:

$$ y=\operatorname{ReLU}(Wx+b). $$

Given a vector of input numbers \(x\), a matrix of weights \(W\), and a bias \(b\) added after the multiplication. Matrix multiplication is performed with the weight matrix \(W\) to produce an output number. ReLU is an activation function, and what it does is keeps positive values and turns negative values into zero.

Training a neural network usually adjusts \(W\) and \(b\) given samples of data.

Suppose we are given some input to our neural network, 2 and 3. We can multiply each input by 1, then add the results:

$$ 2\times 1+3\times 1=5. $$

Okay… so why is this useful? Well, you could imagine that we want a little addition function inside our network. We can make one by hand-crafting the weight matrix \(W=[1,1]\) and setting the bias to zero. If we pack 2 and 3 into a vector and multiply by the weight matrix we get \(Wx\):

$$ Wx= \underbrace{\begin{bmatrix}1&1\end{bmatrix}}_{\text{weights}} \underbrace{\begin{bmatrix}2\\3\end{bmatrix}}_{\text{inputs}} =\begin{bmatrix}1\times 2+1\times 3\end{bmatrix} =\begin{bmatrix}5\end{bmatrix}. $$

Matrix multiplication here means that first 1 multiplies the 2, the second 1 multiplies the 3, and finally those products are added together. Multiplying by 1 leaves both inputs unchanged, so we’ve basically just added them. We just created a primitive addition function.

Those two ones are the weights of our network. Usually, you would train a network on samples to get useful weights but here, we already know the calculation (addition) that we want to perform, so we can choose the weights directly.

It doesn’t matter which numbers are in the input, it should work for any pair. Try it yourself.

We were able to create some simple weights for our network, but we still can’t represent a full layer yet. Without nonlinearity, our little addition is just a neat party trick. Nonlinearity is a key ingredient in allowing a neural network to learn, so we need to add an activation function into the mix.

I decided to choose the simple but ubiquitous ReLU function. It returns its input if it’s positive, and zero otherwise.

For the example above, ReLU works fine. ReLU takes 5 and returns 5. But suppose we wanted to add −7 and 2. If you were to do the math, you would see we wouldn’t get -5 like you expect, but instead get 0 because of the ReLU.

Our first row, \([1,1]\), adds the inputs. But we can add more to it. What if we add a second row, \([-1,-1]\)? That row calculates \(-(-7)-2=5\): the negative of our sum. Now the matrix gives us two numbers:

$$ \begin{bmatrix}1&1\\-1&-1\end{bmatrix} \begin{bmatrix}-7\\2\end{bmatrix} =\begin{bmatrix}-5\\5\end{bmatrix} \xrightarrow{\mathrm{ReLU}} \begin{bmatrix}0\\5\end{bmatrix}. $$

ReLU still got rid of the −5, but we are still left with a positive copy of the erased value’s magnitude in the second entry. To get the answer back, subtract that entry from the first. And we know how to do that with weights: multiply by \([1,-1]\).

$$ \begin{bmatrix}1&-1\end{bmatrix} \begin{bmatrix}0\\5\end{bmatrix} =\begin{bmatrix}1\times 0+(-1)\times 5\end{bmatrix} =\begin{bmatrix}-5\end{bmatrix}. $$

This second multiplication must happen after ReLU, so our negative answer survives. This is the structure of the feed-forward block we’re building: one matrix multiplication, then ReLU, then another matrix multiplication.

This still works for 2 and 3 in the original example. It will be an exercise left for the reader.

So what are we left with? A small network that adds two numbers–give it a vector \(x\in\mathbb{R}^2\)–just a pair of real numbers, \(x_1\) and \(x_2\)–and it returns \(x_1+x_2\).

We manually decided the weights based on second-grade arithmetic rules and some simple linear algebra, but a compiler can do that for us. When we ask it to add two values, it builds this little network and connects its output to wherever the sum is needed. We can build similar networks for other primitive operations like comparisons and for choosing between values, then connect them so one calculation uses the result of another, forming an expression graph.

Generating the next number

We can add two values now. How does that get us closer to running DOOM inside a transformer?

The core components that we need are a feed-forward network (the addition network) and attention. We also need to learn how to build an autoregressive sequence from our addition function.

A quick refresher on attention

Attention uses three weight matrices to turn each input vector into a query (what it’s looking for), a key (how it can be found), and a value (the information it can pass along).

Each query is compared with the available keys to score how well they match. Those scores determine how the values are combined into an output–usually a weighted average, though we will be using hard attention which simply copies the value with the highest score.

Let’s use Fibonacci numbers as a small example for an autoregressive sequence. Each new number is the sum of the previous two, so we can reuse the addition function we just built. This lets us work out how to read from a sequence and generate its next item with a calculation we already calculated.

Begin with 0 and 1. Add them to get 1, then add the last two numbers again to get 2:

0, 1                  → 0 + 1 = 1
0, 1, 1               → 1 + 1 = 2
0, 1, 1, 2            → 1 + 2 = 3
0, 1, 1, 2, 3         → 2 + 3 = 5

To keep the example small, we’ll keep only the last digit of each answer. When we reach 5 + 8 = 13, we’ll write 3 instead. Each digit will be represented by one token, so our model only has ten possible output tokens: 0 through 9.

The goal is the same for each step: Get the last two digits, add them together, and write the last digit of the answer.

Our addition network expects two numbers, but we’re now giving the model a whole sequence. For [1, 1, 2, 3], something has to be able to select 2 and 3, and use the addition network to add them, and finally make the answer the next input.

Inside our transformer, each position gets its own vector. The embedding puts the digit’s numerical value into that vector2, and we also supply its position, so the model has both “what number is here?” and “where is here?” information. The figure shows only the vector at the last position.

Attention brings the two numbers together at that last position. We construct one head to select the current position and another to select the preceding position, then copy their digit values into two entries of the last position’s vector. Those entries now contain 3 and 2, which are the inputs our addition network needs.

The addition network writes 5 into another entry, and an output layer selects the corresponding token. The inference engine (the ordinary program that runs the model) appends that token and repeats:

sequence → attention retrieves two numbers → addition → next token
    ↑                                                       |
    └──────────────── append and repeat ────────────────────┘

In the figure below, click through from Embed to Output to follow that calculation.

Residual stream at the last position

At the last position in [1, 1, 2, 3], we start with the vector [3, 0, 0, 0]: the first entry holds the original digit, and the others leave space for our calculations.

$$ \underbrace{\begin{bmatrix}3\\0\\0\\0\end{bmatrix}}_{\text{after embedding}} \quad\xrightarrow{\text{attention}}\quad \underbrace{\begin{bmatrix}3\\3\\2\\0\end{bmatrix}}_{\text{after attention}} $$

Attention copies the current digit and the previous digit into that space, giving us [3, 3, 2, 0], then our addition layer adds [0, 0, 0, 5] to produce [3, 3, 2, 5].

$$ \underbrace{\begin{bmatrix}3\\3\\2\\0\end{bmatrix}}_{\text{after attention}} +\underbrace{\begin{bmatrix}0\\0\\0\\5\end{bmatrix}}_{\text{addition layer's output}} =\underbrace{\begin{bmatrix}3\\3\\2\\5\end{bmatrix}}_{\text{after addition}}. $$

This vector that we pass between layers is the residual stream, and adding each layer’s output lets us store the answer while keeping the inputs.

How does attention select exactly one position?

We use hard attention, which copies the value at the highest-scoring position, choosing only from the current position and earlier ones.

To retrieve position \(q\), we give each candidate position \(j\) a score using these query and key vectors:

$$ \underbrace{\begin{bmatrix}2q & 1\end{bmatrix}}_{\text{query}} \underbrace{\begin{bmatrix}j\\-j^2\end{bmatrix}}_{\text{key}} =q^2-\underbrace{(j-q)^2}_{\substack{\text{squared}\\\text{distance}}}. $$

The \(q^2\) term is the same for every candidate, while \((j-q)^2\), the squared distance between positions, is zero at the position we want and positive everywhere else, so subtracting it makes that position win.

Remembering a value

For Fibonacci, we always know where to find our inputs: the last two positions in the sequence. But suppose we want to remember the player’s health, which might change now and later when the model performs thousands of other calculations.

We can write each change into the sequence, keeping both the name of the variable and its new value. This might look something like this:

health = 100
ammo = 10
health = 75

Now we want to read health. We need attention in the transformer to find the most recent value for that variable, wherever it happens to be in the sequence.

We already used attention to retrieve a number by its position; but this time, we choose weights that grade each entry according to which variable it belongs to and how recently it was written. You can imagine we are searching for a key in a dictionary, and the value is a queue where the front of the queue is the most recent value. Both health entries match but the later one gets the higher “score”, so attention retrieves the 75 instead of the 100.

This gives us a way to update a variable even though we can only append tokens: we leave the old value in the sequence, write the new value at the end, and make future reads select the latest matching entry. Each name-and-value pair is a record, and these records form the model’s working memory.

The next figure uses processor storage locations called registers, with names like a0 and a1, but the idea is exactly the same: select a name, then watch which record attention retrieves as new writes appear.

Attention retrieves the most recent value written to the selected register.

If a register hasn’t been written yet, the model uses its initial value from the prompt.

We can use the same idea for larger memory too. Give each location an address, record writes to that address, and retrieve the latest value when we need it.

How does attention find the latest value at an address?

Each record tells us which address it wrote to (its key), and we supply the address we want to read as our query:

$$ \text{score} =-M\underbrace{(\text{key}-\text{query})^2}_{\text{address mismatch}} +\underbrace{\text{position}}_{\text{recency}}. $$

If we want address 7, a record that wrote to address 7 has a difference of zero, so its score is just its position in the sequence and the newest matching record wins.

Records for other addresses get a penalty, and we choose \(M\) large enough that even a newer record for the wrong address can’t beat one for the address we want.

Giving the model a computer to run

So we have the ability to calculate with values and remember them between different steps, but DOOM has code involving rendering, movement, and enemies. How do we get our transformer to execute that code?

A processor runs a program by following small, simple, instructions, like “add these two registers” or “read this memory address.” We can build a transformer that follows those same instructions, using attention to retrieve the inputs and our operation networks calculate the result.

For example, suppose the next instruction says:

Attention retrieves the latest values of a1 and a2, the addition network calculates their sum, and the model generates a record containing the new value of a0. That record becomes part of the sequence, so later instructions can retrieve it using the memory mechanism we just built.

We also need to remember which instruction to execute next, so we keep its address in a value called the program counter. Usually we advance to the next instruction, but a jump changes that address, allowing the program to repeat earlier instructions or take a different path.

For this project, I chose the RISC-V instruction set, specifically RV32IM,3 and built the calculations needed to execute its instructions. There are two different compilers involved here:

DOOM's C code       → GCC         → RISC-V instructions
RISC-V interpreter  → my compiler → transformer weights

I compiled DOOM with a C compiler (my GOAT, GCC) and then we basically put it into the compiled network. The weights describe the processor; the prompt contains the program and its initial game data.

There is a simple loop to execute the program: retrieve an instruction, read its inputs, calculate the result, and write back the result. The written back result will be the input record for the next iteration!

Here is a simple example of the processor running the greatest common divisor algorithm on two numbers:

gcd60.c on the compiled RISC-V interpreter, one 10-token record per instruction. Arcs: where each operand was last written.
How are instructions and 32-bit values represented?

An ordinary instruction produces a ten-token record: three tokens for the next program counter, four for the value being written, and three for the write’s destination key.

A 32-bit value is split into four bytes. That lets the arithmetic circuits work with small values and pass carries between bytes, much as we carry a digit when adding by hand.

Some instructions require several records; division is an especially awkward one: calculating the whole quotient in one step would have made the network about 800 layers deep. Those layers would affect every instruction, even though division accounts for only about 0.1% of DOOM’s instructions. Instead, division proceeds through a 64-step sequence of smaller calculations.

Drawing DOOM

The model can now execute a program. We turn the prompt into DOOM’s compiled instructions and the game’s starting memory and assets–like the levels, textures and monster sprites.4

The major roadblock is DOOM’s rendering code, which calculates what should appear on the screen. The pixel values of what is rendered is found in a region of memory called a framebuffer. The framebuffer maps memory to the 320x200 pixels that represent a DOOM frame.

To write a pixel, an instruction calculates a value and writes it to an address (similar to the registers described above)–and the transformer generates records describing the change to the address. If the address happens to be in the framebuffer, that actually changes the pixel value on the screen (I know this kind of sounds like magic, but actually Linux has a file /dev/fb0 that you can write values into to render them on your display!). 5

DOOM's rendering instructions
    → calculate pixel values
    → write them into memory
    → display the completed frame

A pixel is just a number between 0-255. When the rendering code decides that a spot on a wall should be brown, that pixel’s address is changed. You can imagine that for our transformer, writing to this address is no different from updating “health”–it generates a record that says “this address now holds this value”.

The inference engine now reads all the addresses and their associated values, matches those values to colors in the DOOM color palette, and draws that image. This is how we get the frame being written piece by piece below!

DOOM frame 151 of the demo: the player faces an imp in a corridor, the status bar below.

Frame 151 of the demo.

Frame 151, drawn in the order the trace writes it. 4 pixels per write; highlighted: the latest writes.

I must confess that we are cheating a little here. You might ask, how do you play this? Are you putting input. Technically we could do this (though a little impractical as you will see later when we discuss speed concerns), but for this experiment I actually used the built-in recorded demo of DOOM in the game, that supplies some inputs.

Getting from the startup banner to the first frame

Before building the full transformer interpreter, I ran DOOM in a small reference emulator. It needed about 17 million instructions just to reach the first frame, then roughly 1.3 million per frame after. About 87% of the instructions were in the renderer.

My first transformer interpreter was far too small for that: 16 KiB of memory and contexts of \(2^{17}\) tokens. Reaching DOOM required addresses spread across several tokens, megabytes of memory, and a much faster inference engine.

Context Problems

Similar to the modern LLM, our model has a context window, the maximum number of tokens it can read at once. Our memory system keeps every write in the sequence, so it keeps growing as the program runs.

For the full DOOM run, the context window was about 33.6 million tokens, including the prompt and everything generated afterward. Even that wasn’t enough to hold the execution history of the whole demo, so we needed a way to continue in a fresh context.

Think back to our health example. If the sequence says health = 100 and later health = 80, we only need to carry health = 75 forward to continue playing. The same idea applies to the entire computer: we need the current contents of memory and the registers, along with the program counter telling us which instruction comes next, but nothing else.

Before running out of room, the transformer writes out those current values as tokens. The inference engine copies those tokens into a fresh prompt, along with the unchanged game assets, and generation resumes from the saved program counter.

Attention can now retrieve values from that saved state until newer writes replace them, just as it retrieved the initial values from our original prompt. We’ve discarded the old execution history while keeping the state it produced.

The timeline below shows how many instructions each frame took. Running the whole demo took about 2,948 fully rotated contexts, which was about 5.45 billion total instructions. This amounts to 5,064 frames. (And each frame matched the actual real emulated DOOM code!)

Every frame of the demo and the instructions it took. The thumbnail shows every eighth frame.

A Need…a Need for Speed!

Running the whole demo in the model took…111 CPU-hours, or about 9.2 hours of elapsed time as I had multiple cores running parallel.

The demo is only 2.4 minutes long. Embarasssssinggg.

The majority of the speed issue comes from the nature of the autoregressive loop itself: we need each token before we can calculate the next one. But suppose we could propose a possible sequence–could the transformer check it faster than it could generate it?

Think back to Fibonacci. If someone gives us [1, 1, 2, 3, 5], we can ask the model to check each proposed continuation:

Input             Model calculates     Proposed token
[1, 1]                  2                    2 ✓
[1, 1, 2]               3                    3 ✓
[1, 1, 2, 3]            5                    5 ✓

Each row already has all its inputs because they came from the proposal, so the GPU can calculate these answers in parallel. Each check still uses only the preceding tokens, just as it would during generation.

This should ring some doorbells for any inference engineering nerds. What we need is speculative decoding.6

For DOOM, the reference emulator supplies the tokens we need to generate. The transformer then calculates its own answer at every position and checks whether it agrees with the proposed token.

If every token agrees, we have exactly the sequence the transformer would have generated on its own. If one disagrees, we can accept the tokens before it, but must discard the rest of the proposal because those later checks may have used the incorrect token as input.

Speculative decoding here doesn’t make the transformer a practical way to play DOOM, since we are relying on the emulator to do the work. It does give us a fast way to confirm the weights agree with the emulator.

On my RTX 3080 Ti, it took about 1.3 GPU hours to check all 72.6 billion generated tokens.

The End…??

So this leaves us with a network that can run arbitrary programs on the little CPU burned (trained?) into it. They aren’t too fast, at least at scale, but I think there is possible interesting use for this. Could we somehow combine this model with a world model, so the world model learns the difficult to specify parts, and we can construct the explicit parts that need to execute exactly?

DOOM was a fun test of how far this could go, but its not the only thing you can do with firebrand. I would be interested to see what other programs people try to compile into a network!

Has anyone compiled programs into transformer weights before?

RASP describes transformer computations through a small programming language. Tracr compiles RASP programs into transformer weights, giving interpretability researchers models whose calculations are known. Giannou et al. also showed how a transformer with fixed weights, run in a loop, can execute programs stored in its input.

The work this project was most inspired by on most directly is this post that I had read earlier this year Percepta’s Transformer VM, which has a transformer that interprets WebAssembly.

Another interesting and similar project (even with a cool similar blog post!) is Torchwright. I think it takes a different approach to firebrand and the Percepta work, but it is cool nonetheless. I have not looked too deeply into how it works however so I could be wrong and it is more similar than I think!

Citation

If you reference this post and feel like citing it:

Monty Bichouna. Running DOOM Inside a Transformer. October 13, 2026.

@misc{bichouna2026doomtransformer,
  author = {Monty Bichouna},
  title = {Running {DOOM} Inside a Transformer},
  year = {2026},
  month = {October},
  url = {https://nostalgia.dev/posts/running-doom-inside-a-transformer/}
}