A dumb benchmark to see how far, fast and cheap various decision APIs can play Crossy Road
AI Slop
Fully prompted to Opus 5.5 xhigh
Videos
Best run of each model (the last 25 seconds at most). The caption shows the model, its last answer and how long that request took.
gpt-6-luna (17) |
jev-preview (14) |
jev-latest (12) |
|---|---|---|
 |
 |
 |
clef-flash (7) |
clef (6) |
|---|---|
 |
 |
Results
| Model | Games | Mean score | Median | Best | Median latency | p90 latency | Decisions/s | Tokens/decision | $/M input | $ per 1k decisions | $ per game |
|---|---|---|---|---|---|---|---|---|---|---|---|
gpt-6-luna |
10 | 7.4 | 6.5 | 17 | 229 ms | 515 ms | 3.13 | 550 | $0.100 | $0.0549 | $0.000857 |
jev-preview |
10 | 5.6 | 5.5 | 14 | 242 ms | 324 ms | 3.67 | 1126 | $0.042 | $0.0473 | $0.000686 |
jev-latest |
10 | 4.1 | 3 | 12 | 252 ms | 340 ms | 3.60 | 1126 | $0.042 | $0.0473 | $0.000610 |
clef-flash |
10 | 3.2 | 2.5 | 7 | 616 ms | 1061 ms | 1.47 | 595 | $0.038 | $0.0226 | $0.000201 |
clef |
10 | 2.4 | 1.5 | 6 | 267 ms | 464 ms | 3.54 | 599 | $0.240 | $0.1437 | $0.020520 |
| Model | Scores |
|---|---|
gpt-6-luna |
3, 7, 6, 12, 17, 8, 6, 8, 2, 5 |
jev-preview |
6, 1, 5, 6, 14, 2, 8, 2, 2, 10 |
jev-latest |
2, 2, 3, 3, 12, 1, 8, 5, 4, 1 |
clef-flash |
1, 2, 2, 3, 2, 4, 2, 7, 3, 6 |
clef |
1, 1, 2, 1, 5, 1, 3, 4, 0, 6 |
10 games per model, 50 in total, run round robin on October 23, 2026 from a MacBook Pro. Score is the number in the top left corner of Crossy Road when the run ends. The whole benchmark cost $0.23 for 1,947 decisions.
- How far:
gpt-6-lunawent furthest on average and had the best single run (17). It answeredupon all 156 of its decisions, and spamming up turned out to be the best strategy anyone found. The two Jev models triedwaitnow and then and scored a bit lower.
- How fast: The Jev models had the lowest and most consistent latency (about 250 ms median, about 330 ms p90).
gpt-6-lunaandclefhave a similar median but a longer tail.clef-flashis the slowest here at 616 ms median, even though it is the small model.
- How cheap:
clef-flashis the cheapest per decision and per game. Jev sends about twice as many tokens per image as the others but is still cheap because of its price.clefis the most expensive per token, and its cost per game is mostly one game where it answeredwait1,093 times in a row for the full 300 second limit without ever moving (score 0, $0.157 of its $0.205 total). Without that game it averages $0.0053 per game and 3.22 decisions per second. - Ten games is a small sample and every Crossy Road map is random, so the order of the middle three could easily change with more runs.
Raw data for every game is in results/runs.jsonl and the game over frame each score was read from is in results/frames.
How it works
Every model plays the real Crossy Road app on a Mac through the same loop:
- A background thread keeps grabbing the game window with
screencapture -l(about 5 fps), downscales it to 512px and JPEG/base64 encodes it, so the next request body is always ready. - The newest frame is sent to the decision API with one
choicequestion:up,left,right,downorwait, plus the last six moves. A move that did not change the screen is marked(no effect)so the model can get around trees. - As soon as the answer arrives the key is pressed and the next request goes out with whatever frame is newest at that moment. Nothing waits for a fresh screenshot.
- A referee thread watches the score in the corner. When
TOPshows up the run is over, and the final score is read from the game over frame by template matching the game's pixel font (games/crossy_road_digits.json).
Every model gets the same prompt, image and choices, and the same 10 runs, interleaved round robin. There is no game logic in the decision loop. The only Crossy Road specific code starts a run (clicks through gifts, reminders and ads, and relaunches the app when an ad has no close button) and reads the score.
| Provider | Endpoint | Model | Price |
|---|---|---|---|
| OpenAI | POST /v1/decisions |
gpt-6-luna |
$0.10 / M input tokens |
| TypeSafe | POST /v1/systemone |
jev-latest, jev-preview |
$0.042 / M input tokens |
| Cloudflare Workers AI | POST /ai/run/@cf/cloudflare/clef |
clef |
$0.24 / M input tokens |
| Cloudflare Workers AI | POST /ai/run/@cf/cloudflare/clef-flash |
clef-flash |
$0.038 / M input tokens |
None of them charge for output tokens. Cost in the table is input tokens times these prices.
Running it
Needs macOS, the Crossy Road app open in a window, uv and ffmpeg. The terminal needs Screen Recording and Accessibility permissions. It takes over the keyboard and mouse while it plays, but only presses keys while the game is the frontmost app, and only focuses the game after 20 seconds without any keyboard or mouse input.
cat > local.env <<EOF OPENAI_API_KEY=... TYPESAFE_API_KEY=... CLOUDFLARE_API_KEY=... CLOUDFLARE_ACCOUNT_ID=... EOF uv run play.py crossy_road --model clef-flash --video run.mp4 uv run bench.py --games 10 uv run report.py
bench.py tops every model up to --games runs in results/runs.jsonl, saves the game over frame to results/frames/ and the video to videos/. report.py re-reads every score from the saved frames, writes results/summary.md and makes the charts and GIFs in media/.
The run stops instead of clicking when it sees a consent prompt or the prize machine, and it never clicks anything on a screen that shows a price.
Other games
The harness is not tied to Crossy Road. A game is a Game subclass in games/ with a window title, instructions, a list of Action(name, description, key), a start() that gets it to a playable state and a read_screen() that says whether the run is over and what the score is. Then uv run play.py <game> --model <model>.
License
MIT