LLM Benchmarks

Purpose

This benchmark compares how well different LLM+harness setups can produce a competitive AoE2 build-order DSL file under similar constraints.

What It Tests

This test is technically a coding optimization problem, which I feel represents pretty well what we want our agentic coders to do. It is, however, kind of a weird test, weird enough that it's likely quite out of distribution.

• Instruction following: honor the exact benchmark prompt and required scoring lines.
• Needle in a haystack: the JSON is 1000 lines and some elements are critical.
• Context rot: several model iterated their full context. This compounds the above problem.
• Out-of-distribution coding: the DSL syntax is strict and original.
• Strategic thining: it's not 100% trivial to achieve all 4 objectives and have a good build order.
• World knowledge: some models show obvious existing knowledge of typical AoE2 build orders.

Setup

I give the model a simple prompt (see bottom of page), the JSON describing game data, a grammar help, and a mostly blank build order.

Disclaimer

For cost reasons, I haven't rerun these benchmarks too many time. I don't think they're _that_ high variance to be honest, but it's worth keeping in mind.

These results are overall not amazing. However, all models that I could run did succesfully write a DSL script.
To be fair - the LLMs have a slightly worse interface to work with, but I would expect better.
What I find interesting is that there is a very clear skill divide on display, when all these models are "good at code".
With the latest models, 5.6 seemed extremely solid - close to saturating. My one opus 4.8 run wasn't amazing however. Didn't test fable cause it eats too many tokens, will see later.
I plan to test out Kimi 3 & glm at some point.
Partly inspired by the brillant minebench.ai

Reproduce

This whole website is GPL3. Benchmark assets live here.

I ran the model with the same constraints and access, but pi has some issues with a few of them.

Prompt