Meet a tiny robot with a big attitude. Hold a button, tell her "build me a tower", and she does: blind, never seen a thing, absolutely sure she just made a masterpiece. Wrong height, wrong place, zero doubt, plenty of sass. She lives entirely on your own machine: local model, local ears, local voice, no cloud. Come argue with her.
llm-sandbox-demo-1.5x.mp4
What this experiment is about
This is not a game. It is a test of one idea: split the controls by how fast your intent is.
- Walking, looking, aiming: your hands already know how to do that. Keyboard and mouse, under 16 ms, the model never gets involved.
- "Build a bridge over there", "make it night", "give me wings": there is no key for that. That is what your voice is for. A few seconds of latency is fine here. It does not feel like lag. It feels like casting a spell.
Everything else follows from that split. The character talks back right away so the wait feels like a conversation, not a loading bar. A ghost outline appears where you aimed the instant you release the button, before the model has said a word. And the model is blind on purpose: it never sees the world, only what you point at and what you say. The gap between what you asked and what she proudly delivers is where the fun lives. We tried to fix it and the game got worse.
What she can do
- Push-to-talk. Hold right click, talk, release. A ghost appears where you aimed while she thinks.
- Answer first, build second. A short spoken reply lands in under a second. The structure lands a few seconds later.
- Build, edit, undo. Towers, houses, trees, statues, bridges. Paint it red. Make it taller. Copy this over there. Delete everything. Undo.
- Come to you, fly with you. She walks over when you call her. Ask for wings and you take off.
- Switch languages mid-sentence. English or Spanish, detected per turn. Adding a language is one config entry and two prompt files.
- Never repeat herself. The words she already said are forbidden next turn. Without that, a 26B model says "amazing" forever.
The stack
All local, all open weights, one consumer GPU (tested on an RTX 3090).
| layer | what we use | why |
|---|---|---|
| Brain | Gemma 4 26B A4B via llama.cpp | mixture of experts: 26B of knowledge at 4B speed. Strict JSON Schema output, so the world never gets garbage. |
| Ears | Whisper large-v3-turbo | fast enough that push-to-talk feels instant. |
| Voice | Supertonic 3 | very fast local TTS, pitched up to sound small and cheeky. |
| World | three.js, vanilla JS, no bundler | one index.html, open it and play. |
| Glue | one Python file | static server plus proxy to the three services, with per-request traces. |
The model does not emit cubes. It emits volumes with stretch labels ("tall", "flat", "wide"), which the client expands into blocks. Same token budget, objects ten times bigger. Two model calls run per turn against a single llama.cpp slot: a tiny one for the spoken acknowledgement, a bigger one for the build. The small one hides most of the latency of the big one.
Every number lives in config.js. Voice pitch, ghost size, token budgets, detail levels, language markers, key bindings. If you want her sillier, faster or bigger, change a number. You never have to read the logic.
Run it
You need Python 3, a browser (Firefox or Chrome), a GPU that fits a ~15 GB GGUF, and three local services. They are not part of this repo; anything that speaks the same contract works.
| service | default port | contract |
|---|---|---|
| STT | 6969 | POST /transcribe, body = raw audio bytes, returns text/plain |
| TTS | 6971 | POST /tts?voice=&fmt=&speed=&pitch=, body = text, returns audio |
| LLM | 6981 | OpenAI-compatible POST /v1/chat/completions with response_format: json_schema |
The llama-server command we run:
llama-server -m gemma-4-26B-A4B-it-UD-Q3_K_M.gguf -ngl 99 --flash-attn on \ -c 262144 --cache-type-k q4_0 --cache-type-v q4_0 -np 1 --port 6981 --reasoning off
Then:
git clone https://github.com/franciscocarloserra/llm-sandbox && cd llm-sandbox echo "your-local-bearer-token" > .token # only if your STT/TTS require one python3 server.py # http://localhost:7801
Override anything with env vars: SANDBOX_PORT, STT_URL, TTS_URL, LLM_URL, LOCAL_TOKEN.
Click the page, hold right click, talk.
| input | action |
|---|---|
| right click (hold) | push-to-talk |
| left click | pin the aimed object as "this" / drag to move it |
| middle click | replicate the aimed object (no model call) |
WASD Space Shift |
move, jump, sprint |
F |
fly / no-clip |
Z / Y (or Shift+Z) |
undo / redo |
F1 F2 F3 |
debug panels: state, chat, edits (off by default) |
Tests
node test/run.mjs # 564 unit tests, no browser, no services needed python3 test/services.py # integration against the three live services
How it was built
By AI coding agents, several in parallel, with a human directing by voice. Two files hold the project together: AGENTS.md (the rules, the map, the traps that already cost us) and INTENT.md (the why behind every decision, measured rather than assumed). The character, the rules she follows and the output schema live in lang/*/, not in code. If you want to change who she is, edit a text file.
License
MIT. audio/Level_One_Morning.mp3 is the only bundled asset; see its own license.