GitHub - maddiedreese/gbc-transformer

2 min read Original article ↗
IMG_4019

TinyStories-260K running locally on a stock Game Boy Color.

This is a proof-of-concept GBDK-2020 ROM that runs a quantized transformer language model on the Game Boy Color CPU. Prompt entry happens on the handheld with the D-pad/buttons and an on-screen keyboard. The ROM tokenizes the prompt, runs transformer prefill, then autoregressively generates a short continuation.

It is intentionally tiny, slow, and imperfect. The current build proves the end-to-end path on hardware; it is not a polished text generator yet.

What Works

  • Desktop FP32 reference runner for stories260K.bin.
  • Desktop row-wise INT8/Q8 packer and zero-float Q8 runner.
  • GBC ROM with the Q8 model embedded as MBC5 bank-switched cartridge data.
  • On-device BPE tokenization using the 512-token TinyStories tokenizer.
  • Integer/fixed-point transformer inference with RoPE, attention, RMSNorm, SwiGLU-style MLP, and greedy argmax decoding.
  • Cartridge SRAM KV cache so base WRAM stays below 8 KB.
  • On-screen keyboard with explicit ENTER.
  • Runtime progress markers (P for prefill, G for generation).

Current Limitations

  • Extremely slow on real hardware. The C implementation works, but the hot matmul loops need SM83 assembly to become demo-friendly.
  • Output quality is rough/gibberish. The GBC path uses heavy quantization and fixed-point approximations.
  • Context is capped at 16 tokens.
  • Greedy decoding only; no top-k or temperature sampling yet.
  • Built for one model shape: TinyStories-260K (dim=64, hidden_dim=172, layers=5, heads=8, kv_heads=4, vocab=512).

Hardware

  • Game Boy Color
  • EZ-FLASH Junior or another MBC5-compatible flashcart
  • microSD card mounted at /Volumes/bootfs for make sd-card

No soldering or internal Game Boy modification is required.

Dependencies

  • Python 3
  • C compiler for desktop tools (clang or gcc)
  • GBDK-2020 4.5.0 or compatible
  • curl and unzip for fetch helpers

By default the Makefile looks for GBDK at:

You can also point it elsewhere:

make GBDK_HOME=/path/to/gbdk gbc-rom

Fetch Model Assets

The TinyStories checkpoint and tokenizer are downloaded from Andrej Karpathy's karpathy/tinyllamas Hugging Face repository:

This creates:

  • stories260K.bin
  • tok512.bin

Expected sizes:

  • stories260K.bin: 1,056,540 bytes
  • tok512.bin: 6,227 bytes

Build

Desktop references:

make run-fp32
make run-q8
make run-gbc-sim

Game Boy ROM:

Output:

Prepare EZ-FLASH Junior SD Card

Fetch the official EZ-FLASH Junior stable kernel:

Then copy the ROM and kernel to the SD card:

The SD root will contain:

  • ezgb.dat
  • TINYSTORY.GB
  • README.txt
  • GBC-TINYSTORIES/ reference folder

In the EZ-FLASH menu, run TINYSTORY.GB from the SD card root.

Credits