June 2026. Every tok/s number is from real runs on a rented Lambda 8×NVIDIA B200 node. Same FP8 GLM-5.2 weights, same prompts, 3 runs averaged. Each local engine measured at its best config. Nothing mocked.
TileRT is Xiaomi MiMo's inference runtime. It's the thing that pushed a trillion-parameter model past 1000 tok/s on a standard 8-card node. The model it ships with is GLM-5. GLM-5.2 came out with no published speed numbers, so I rented an 8×B200 and tried to measure it on TileRT myself.
Three walls. One of them permanent. I got the model running at ~480 tok/s, quality identical to OpenRouter, about 5× faster than vLLM on the same GPUs. Here's what broke, where the 5× actually comes from, and the catch.
Wall 1: the B200 you rent isn't CUDA-13-ready
TileRT 0.1.4's image is built for CUDA 13. The Lambda B200 boots on the older driver, R570 / CUDA 12.8, managed by Lambda's own driver stack. First thing TileRT did was crash.
RuntimeError: The NVIDIA driver on your system is too old (found version 12080).
No CUDA-12 build of TileRT exists. I didn't want a compatibility shim polluting the numbers, so I upgraded the driver for real: R570 to R580. On a Lambda node that's fiddly. The fabric manager (the piece that lets the 8 GPUs talk to each other over NVLink) has to match the new driver exactly, or the GPUs can't see each other. And you have to confirm the new driver actually built for the running kernel before you reboot, or the box comes back with no GPUs at all.
It worked. Lesson stuck though: a B200 you rent today is not, out of the box, ready to run a CUDA-13 stack. Budget an hour for the driver before you measure anything.
Wall 2: TileRT doesn't support GLM-5.2
TileRT 0.1.4 supports GLM-5.1. GLM-5.2 looks almost identical on paper, 78 layers, 256 experts, MLA attention, FP8. But it adds one thing TileRT has never seen: IndexShare.
Both models use DeepSeek-style sparse attention. A small "indexer" picks the top-2048 tokens each layer attends to. In GLM-5.1, and in what TileRT expects, every layer runs its own indexer. GLM-5.2 marks every 4th layer full (it computes the index) and the 3 layers between shared (they reuse the previous full layer's pick). About 2.9× fewer attention FLOPs at max context. It's also why the conversion died on the spot:
KeyError: 'model.layers.3.self_attn.indexer.wk.weight'
Layer 3 is shared. It has no indexer weights, because in GLM-5.2 it doesn't need them. TileRT's converter assumes every layer has them.
So I made GLM-5.2 look like the uniform model TileRT wants. The remap synthesizes the missing indexer subtree on each shared layer by copying it from the full layer it shares from. 399 small tensors. The 700 GB of real weights stay untouched, only the index gets patched. TileRT's converter then ate it as a plain GLM-5, and the model ran.
The part that mattered: I checked the output against OpenRouter's GLM-5.2 on knowledge questions. Capital of Australia (Canberra), 17×24 (408), the bat-and-ball trap ($0.05), gold's symbol (Au), War and Peace (Tolstoy). Every answer matched. The remap isn't a lobotomy. Inside its window it's bit-for-bit GLM-5.2.
Wall 3: the 2048-token ceiling, and this one doesn't move
"Inside its window" is carrying the whole post.
The remap is exact only while total context stays under 2048 tokens. Reason: when the sequence is shorter than index_topk (2048), the top-2048 grabs every token. Attention is dense, and what the indexer picked stops mattering. Past 2048, sparse selection starts to matter. My synthesized indexers then compute the index from each shared layer's own hidden state, instead of reusing the full layer's pick. They grab the wrong tokens. The model falls apart into wait wait wait wait, then 0: 0: 0: 0:. You can watch it the second a chat crosses ~2048 tokens.
I tried to break it. The kernel hardcodes the window. Set index_topk to anything but 2048 and it throws idx_selects must have last dim 2048. The obvious fix is to make the shared layers reuse the full layer's real selection. That needs a hook between the indexing step and the attention step. There isn't one. TileRT runs the whole model in a single closed kernel that stays resident on the GPU: one call runs all 78 layers inside it, and from the outside you never see the individual layers. No place to reach in, and the compiler that builds that kernel isn't public. So the ceiling can't move without TileRT shipping GLM-5.2 itself. I cap output at 2000 tokens so it cuts clean instead of spewing garbage.
This is the catch nobody puts in the headline. Fast and correct, but length-capped. Fine for a coding turn. Useless for a 100k-token doc.
The benchmark
Forty questions, four domains (Python, JavaScript, algorithms, SQL), 3 runs each, averaged. Same FP8 weights for the two local engines, no-think mode, each engine at its best config. That last clause matters and I'll get to why.
| Domain | TileRT (8×B200) | vLLM (8×B200) | OpenRouter |
|---|---|---|---|
| python | 492.5 | 96.1 | — |
| javascript | 479.2 | 96.1 | — |
| algorithms | 490.0 | 96.1 | — |
| sql | 459.8 | 96.1 | — |
| overall | 480.4 tok/s | 96.1 tok/s | ~104 tok/s |
TileRT is about 5.0× faster than vLLM on the same hardware. vLLM and OpenRouter land in the same ~100 tok/s spot, which answers a question I actually cared about: OpenRouter isn't a ceiling. ~100 tok/s is just what a normal serving stack does for GLM-5.2. TileRT is the outlier, not OpenRouter.
(One best-case coding prompt hit 540 earlier. 480 is the honest cross-domain average. MTP acceptance moves with content, so I'm reporting the average.)
Every question
All 40, TileRT 3-run mean vs vLLM 3-run mean, same prompt. vLLM sits flat at 96 because its decode rate doesn't care what you ask. TileRT moves with content: more boilerplate code means higher MTP acceptance means faster.
| # | domain | prompt | TileRT | vLLM | × |
|---|---|---|---|---|---|
| 1 | python | flatten an arbitrarily nested list | 507 | 96 | 5.3× |
| 2 | python | retry decorator (3 attempts) | 482 | 96 | 5.0× |
| 3 | python | merge two sorted lists | 541 | 96 | 5.6× |
| 4 | python | simple stack class | 522 | 96 | 5.4× |
| 5 | python | generator of first n primes | 492 | 96 | 5.1× |
| 6 | python | top-3 word frequencies | 442 | 96 | 4.6× |
| 7 | python | timing context manager | 480 | 96 | 5.0× |
| 8 | python | validate an IPv4 address | 462 | 96 | 4.8× |
| 9 | python | group list of dicts by key | 508 | 96 | 5.3× |
| 10 | python | Levenshtein edit distance | 489 | 96 | 5.1× |
| 11 | javascript | debounce (leading-edge option) | 468 | 96 | 4.9× |
| 12 | javascript | deep-clone an object | 475 | 96 | 4.9× |
| 13 | javascript | fetch with AbortController timeout | 497 | 96 | 5.2× |
| 14 | javascript | flatten a nested array to depth | 503 | 96 | 5.2× |
| 15 | javascript | curry with partial application | 498 | 96 | 5.2× |
| 16 | javascript | useLocalStorage React hook | 485 | 96 | 5.0× |
| 17 | javascript | throttle to once per N ms | 472 | 96 | 4.9× |
| 18 | javascript | TS generic pick-keys | 431 | 96 | 4.5× |
| 19 | javascript | parse a query string | 481 | 96 | 5.0× |
| 20 | javascript | memoize by stringified args | 481 | 96 | 5.0× |
| 21 | algorithms | binary search, correct boundaries | 454 | 96 | 4.7× |
| 22 | algorithms | quicksort + complexity | 520 | 96 | 5.4× |
| 23 | algorithms | BFS on an adjacency list | 522 | 96 | 5.4× |
| 24 | algorithms | Dijkstra with a heap | 475 | 96 | 4.9× |
| 25 | algorithms | LRU cache, O(1) | 485 | 96 | 5.0× |
| 26 | algorithms | two-sum in O(n) | 453 | 96 | 4.7× |
| 27 | algorithms | linked-list cycle (Floyd) | 518 | 96 | 5.4× |
| 28 | algorithms | merge sort + stability | 525 | 96 | 5.5× |
| 29 | algorithms | trie insert/search | 502 | 96 | 5.2× |
| 30 | algorithms | DP coin-change | 446 | 96 | 4.6× |
| 31 | sql | second-highest salary | 455 | 96 | 4.7× |
| 32 | sql | find duplicate emails | 473 | 96 | 4.9× |
| 33 | sql | INNER vs LEFT vs FULL join | 470 | 96 | 4.9× |
| 34 | sql | window-function rank by dept | 497 | 96 | 5.2× |
| 35 | sql | 7-day moving average | 472 | 96 | 4.9× |
| 36 | sql | B-tree indexing, when it helps | 426 | 96 | 4.4× |
| 37 | sql | orders in every month of 2024 | 475 | 96 | 4.9× |
| 38 | sql | ACID with an example | 434 | 96 | 4.5× |
| 39 | sql | pivot rows into columns | 446 | 96 | 4.6× |
| 40 | sql | N+1 query problem | 450 | 96 | 4.7× |
The 3 runs per question are tight, no cherry-picking. Q3 (merge sorted lists) on TileRT: 540.0, 540.7, 541.4. Q6 (word frequencies): 442.0, 441.5, 441.8. vLLM was 96.0-96.1 on every single run of all 40.
Watch it run
Recorded live on the node. Real time, no speed-up, no audio.
Where the 5× comes from (and why "best config" matters)
The part most benchmarks skip. TileRT's 480 isn't one trick. It's two, and only one is the kernel.
The persistent megakernel: TileRT's base forward rate, speculative decoding off, is about 152 tok/s. vLLM's is 96. So the kernel itself, keeping the model resident with no per-op launch gaps, is worth roughly 1.6×.
MTP: GLM-5.2 ships a tiny "nextn" layer that drafts the next few tokens. The big model verifies a block in one pass and keeps whatever matched. TileRT accepts about 3.54 tokens per forward pass off it. That's the other ~3×, 152 up to 480.
So the headline is 1.6 × 3, not 5× of pure kernel. Worth saying out loud, because MTP is something anyone can turn on.
Which is where "best config" comes in. To make it fair I tried giving vLLM MTP too. It made vLLM slower.
| vLLM config | tok/s |
|---|---|
| no MTP | 96 (best) |
| MTP, 1 draft token | 86 |
| MTP, 3 draft tokens | 77 |
Not a bug in my setup. Documented behavior. Speculative decoding is a bet: draft a block, verify it in one big pass, eat the overhead. High acceptance, big win. Below ~0.5 acceptance the draft-plus-verify overhead beats the savings and you go net-negative. vLLM's generic version of this gets low acceptance on GLM-5.2's single draft layer. That layer was trained to guess only the very next token, so when you push it to guess two or three ahead, it's usually wrong. The rejected guesses cost more than the accepted ones save. So vLLM's best config is MTP off.
That cleans up the framing instead of muddying it. Each engine at its best is TileRT-with-MTP (480) against vLLM-without (96), because turning MTP on is a loss for vLLM. And it means part of TileRT's win is a real MTP-implementation edge: it gets 3.54 accepted tokens out of the exact same nextn layer where vLLM's generic recursion falls over. Not cheating. The co-design MiMo keeps talking about.
The thing vLLM does that TileRT can't
One number goes the other way. vLLM 0.23 supports GLM-5.2 properly, IndexShare and all. Point it at the original weights, no remap, and it serves the real model: correct attention, a million tokens of context, no 2048 ceiling, at 96 tok/s.
So the tradeoff is sharper than "TileRT fast, vLLM slow."
TileRT: ~480 tok/s, capped at ~2000 tokens, identical to GLM-5.2 only inside that window.
vLLM: 96 tok/s, the genuine unbounded model.
Short latency-sensitive turns, chat, code completion: TileRT is a different category of fast. Long context, or you can't live with a hard cap: vLLM is the boring correct answer.
Takeaways
TileRT is genuinely fast, ~5× over vLLM on the same B200s, and it's not a short-context trick (vLLM at the same length still only does 96). But ~3 of that 5× is MTP, a better-built version of something anyone can do. The pure kernel win is ~1.6×.
GLM-5.2's IndexShare is the wall. TileRT doesn't support it, the remap only holds under 2048 tokens, and the closed megakernel means I can't lift the ceiling.
OpenRouter is not a ceiling. It's a normal ~100 tok/s stack, same as vanilla vLLM.
Speculative decoding is not free throughput. It made vLLM slower here. Check acceptance before you trust it.
The fast version of GLM-5.2 on B200 is real, for the first 2000 tokens, because a closed kernel decides where the cliff is. The correct unbounded version is vLLM at 96. Pick the one that fits the job.
Raw per-question 3-run JSON for both engines, the IndexShare remap, the OpenAI-compatible server shims, and the full build-it-yourself plan are in my working notes. Every number in the tables came from those runs.