Husky: a model-specific inference engine up to 4.5× faster than Apple's MLX

Research · Inference · Underdog

Husky is a model-specific inference engine that runs Underdog's Pareto-frontier model, Woof. On a MacBook, Woof runs up to 730 tokens a second with Husky Flash on, up to 4.5× faster than MLX, and answers 5× sooner with the same weights. Built for Apple silicon, so Underdog runs on your Mac and iPhone, even without wifi.

730 tok/sdecode, Flash onup to, on a function edit; MLX 163. 1.3 to 4.5× across all sixteen tasks

5,150 tok/sprefillcold, 859-token prompt, at the caller; MLX 3,094. Inside the engine, 5,460

35 msto the first wordon a continued chat; MLX 140 ms with its cache, 280 ms rereading. 3.6 to 5.5× sooner

16 of 16tasks faster than MLXsame weights, same answers. Without Flash 1.02 to 3.9×, edits 1.8 to 3.9×

Against MLX, the runtime Underdog has shipped Woof on since day one, Husky is faster on all sixteen prompt types we measure. The edits are fast because Husky verifies eight tokens per step at 1.5× the cost of one, and most of the reply is already in the prompt. The writing lead comes from pipelining: the next step is encoded and submitted while the current one runs, so the GPU never waits on the host. Cold, on a prompt neither has seen, Husky reaches the first word 1.8 to 3× sooner; against MLX's plain generate call, which prefills the whole conversation again, 4 to 7×.

Husky: higher model data efficiency on Apple silicon than MLX

Producing a token means streaming all 2.4 GB of Woof from unified memory through the 40 GPU cores, about 6 ms on the M5 Max, and no engine reads faster than the bus. What Husky changes is how many tokens come out of each read.

The same edit, written on both dies, same clock1×

MLX0 tokens

Husky0 tokens

Measured on this edit: MLX 159 tokens per second, Husky 463, Flash on 535.Tokens kept per read of the weights: MLX 1.0, Husky 5.4 on edits, Flash on 2.7 on writing.Click a die to pause.

Figure 1. One decode step on the M5 Max, on an edit. Producing a token means streaming all 2.4 GB of Woof across the memory bus into the 40 GPU cores, about 6 ms either way. MLX gets one token per read and waits on the host between reads. Husky feeds eight candidate rows through each read, keeps the ones that match, has the next read queued before this one ends, and keeps the conversation's state in memory instead of rereading it. Being built for one model is what makes each of those possible. Click to pause.

Results

One M5 Max, one set of Woof weights, greedy decoding. Sixteen prompt types drawn from what the app asks Woof to do: emails, thread replies, rewrites, plans, notes, call summaries, questions over a document, invoices to JSON, data to a table, code from scratch and code edits. Each prompt was measured on both engines in its own quiet window, three runs each, with a canary run first and the order alternated from prompt to prompt.

Figure 2. Decode throughput across sixteen prompt types, tokens per second while writing the reply, three bars per prompt: MLX, Husky, and Husky with Flash on, the draft model described below. Husky alone is 1.02 to 1.27× faster than MLX on writing and 3.8× on the function edit, where most of the reply is already in the prompt. Flash on adds 1.1 to 1.7× on writing, 2.4× on Invoice to JSON and 3.2× on code from scratch, and on the edits it defers to prompt lookup and edges it.

Figure 3. Time to the first token when a conversation is re-sent with one more message, the way a chat works. Both engines continue from their caches: Husky from its prefix cache, MLX from mlx-lm's prompt cache extended over the previous turn, so only the new message (26 to 31 tokens) is fed. MLX's plain generate call, which prefills the whole conversation again, takes 145 to 295 ms. Cold prompts are Figure 4.

Figure 4. Cold prefill measured at the caller: prompt tokens per second of wall-clock time to the first token, on prompts neither engine has seen. Inside the engines the long-prompt rates are 4,878 (MLX) and 5,460 (Husky).

Table 1. Sixteen prompt types, medians of three runs per engine, each measured in its own quiet window with a canary check first and repeats that agree within 12%. Flash on is the same engine with the trained draft in its slot; its Faster column is against MLX. First-token times are for a conversation re-sent with one more message, the way a chat works, with both engines at their best: Husky answers from its prefix cache; MLX from mlx-lm's prompt cache extended over the previous turn, fed only the 26 to 31 new tokens. MLX's plain generate call, which prefills the whole conversation again, takes 145 to 295 ms instead; that is the number a caller sees without the cache. Cold, on a prompt neither engine has seen, Husky reaches the first word in 0.05 s on the short prompt and 0.17 s on the 859-token one; MLX takes 0.15 s and 0.30 s.

Prompt	MLX tok/s	Husky tok/s	Faster	Flash on	Faster	MLX first	Husky first	Sooner
Function edit, 368 tokens in	163	614	3.77×	730	4.48×	177 ms	32 ms	5.5×
Add a field to a JSON file, 477 tokens in	157	611	3.89×	672	4.28×	145 ms	30 ms	4.8×
Rename a SQL column, 240 tokens in	158	487	3.08×	547	3.46×	143 ms	33 ms	4.3×
Write a function, 40 tokens in	155	170	1.10×	545	3.52×	145 ms	29 ms	5.0×
Fix typos in a paragraph, 301 tokens in	159	463	2.91×	535	3.36×	141 ms	31 ms	4.5×
Invoice to JSON, 321 tokens in	164	208	1.27×	496	3.02×	144 ms	30 ms	4.8×
CSV to a table, 303 tokens in	158	287	1.82×	462	2.92×	141 ms	34 ms	4.1×
Data to a table, 247 tokens in	163	188	1.15×	282	1.73×	150 ms	31 ms	4.8×
Repeated transcript, 857 tokens in	153	164	1.07×	273	1.78×	140 ms	39 ms	3.6×
Tone rewrite, 195 tokens in	162	168	1.04×	269	1.66×	138 ms	34 ms	4.1×
Short email, 38 tokens in	151	164	1.09×	260	1.72×	148 ms	38 ms	3.9×
Project plan, 51 tokens in	158	171	1.08×	246	1.56×	139 ms	37 ms	3.8×
Meeting notes to to-dos, 249 tokens in	163	175	1.07×	234	1.44×	148 ms	36 ms	4.1×
Reply to an email thread, 280 tokens in	163	170	1.04×	217	1.33×	139 ms	36 ms	3.9×
Call summary, 657 tokens in	162	166	1.02×	211	1.30×	151 ms	34 ms	4.4×
Question over a document, 297 tokens in	159	193	1.21×	210	1.32×	137 ms	34 ms	4.0×

How it works

Husky refuses to be general, and that is where the speed comes from.

Kernels compiled for Woof's exact shapes. Every matrix in Woof has one size, so every GPU kernel is written for that size and nothing else.
Weights packed the way the kernels read them. Woof's 4-bit weights are laid out on disk in the kernels' own tile order and mapped straight into memory. No conversion at load, no copy.
An 8-bit cache the attention kernels read directly, and a memory plan worked out for your Mac when the engine starts. No config file, nothing to tune.
It remembers your conversation. Send the same chat again with one more message and Husky answers from where it left off, which is why the first word takes about 35 ms. MLX's prompt cache does the same job and still takes about 140 ms, the fixed cost of a call on that path.
It checks eight tokens at once. Every step verifies a block of eight tokens for the price of one. Today the guesses come from your own prompt, which is why a function edit runs 3.8× faster: most of the file is already there. With Flash on, a draft trained on Woof guesses for everything else.

Figure 5. Inference engines are general: one runtime, many models. Husky runs one. A general engine keeps one kernel for any shape and checks sizes as it goes. Husky knows Woof's four matrices at compile time, so each has a kernel drawn to its size, and the weights are stored on disk in the tile order that kernel reads.

Why prose is close, and how Husky pulls ahead

Producing one token means reading all 2.4 GB of Woof's weights. On this Mac that takes about 4.5 ms even for a plain streaming read, and no engine gets under it. Husky and MLX both sit near that line for plain writing: about 6.1 ms per token against 6.3. What separated them was the host. Every step used to cost almost half a millisecond of encoding and waiting between GPU commands; Husky now encodes the next step while the current one runs, so the GPU goes straight from one token to the next. That is the 5% on the transcript. The way past the line itself is to get several tokens out of one read.

That is what the eight-token check is for. The guesses come from the prompt when the reply repeats it, and otherwise from Flash, a draft trained on Woof itself, which lands about 2 tokens a step today and takes prose from 164 to 175 tokens per second to 210 to 273. Every extra token a step is another 60 or so tokens per second.

Built for edits

Much of what an assistant does to your text is not writing, it is editing: fix the typos in this paragraph, add a field to this file, turn this CSV into a table, rename this column, change one thing in this function and give it all back. In every one of those the reply repeats long runs of the prompt word for word. Husky is built for that shape.

Every decode step verifies a block of eight tokens for about one and a half times the price of one. When the last few tokens Husky wrote also appear in the prompt, it proposes the seven that followed there, and Woof keeps the ones it agrees with, typically five or six. A plain engine writes those replies one token at a time; Husky writes them five and six tokens at a time, which is where the 1.8 to 3.9× in the table comes from, with the answer identical to the letter. Underdog asks Woof for edits like these all day, so this is not a corner case, it is the common one.

Flash on

Prompt lookup only helps when the reply repeats the prompt. For everything else the eight-row step needs a guesser, and we trained one, the Flash draft: a single small layer that reads Woof's own hidden states from five of its layers and proposes the next seven tokens. It is trained by self-distillation on Woof's replies to 139,000 conversations: Underdog's own prompt shapes (edits, JSON, tables, HTML and markdown UI, tool calls, mail, code, summaries, browser steps, follow-ups) and public chat, instruction and code sets, about two hours on one B200. Nothing about the answer changes: Woof still checks every proposal and keeps only what it would have written.

With Flash on, Husky decides each step who guesses. If the last few tokens appear earlier in the prompt, lookup proposes, since it is better at copying. Otherwise the draft runs while its guesses land at least half a token a step, and rests on the one-row path when they stop. On writing the draft lands about 2 tokens a step, which turns 164 to 175 tokens per second into 210 to 273. On code from scratch it lands 5 and the reply runs at 545, three and a half times MLX; on Invoice to JSON, 496. The edits stay with lookup and gain a little from the draft between matches: 462 to 730 tokens per second, 2.9 to 4.5× MLX. This is the same weights, the same answers, and a 273 MB draft file in the package.

Two engine changes made the draft pay. A draft step is an eight-row step, and the eight-row projections had been running at 2.3× the cost of one row because the kernels walked a 12,800-wide input as one serial chain of matrix ops per simdgroup; splitting that input across the eight simdgroups brought the step to 1.5×, which is also why the edit rows above rose from 403 to 487 and from 483 to 611 in the same change. And the draft is judged the way lookup is, by what it landed, over a window long enough to see past a reply's opening, where every draft guesses worst.

MSI: model-specific inference

We call the approach MSI, model-specific inference: an engine that knows one model's every shape at compile time and spends none of its time being general. Today that means four things, all shipped in the numbers above.

Model-shaped, specialized Metal megakernels. Each projection has its own kernel for its exact matrix, with the norm that precedes it folded in, the gate and up projections walked together, the 4-bit weights unpacked four at a time straight into the multiply, and the eight-row step's long inputs split across the simdgroups. A layer's whole MLP already runs as one persistent dispatch; the whole step is the direction below.
A resident grid for the recurrence. The 24 linear-attention layers run their recurrence on a persistent grid of threadgroups that hand off through device-side counters, not through the command stream.
Weights and cache in kernel order. The package stores the weights in the tile order the kernels read, and the 8-bit key-value cache in the layout the attention kernels consume, so nothing is rearranged at load or at run time.
Pipelined submission. The next decode step is encoded and submitted while the current one runs, with the next token handed over on the GPU. The host is off the critical path, which is where the last 5% on long prose came from.

Figure 6. One decode layer as a row of dispatches, to scale in GPU time. A general engine issues one per operation with the host encoding between them. Husky folds the norm into each projection, walks gate and up together and splits the long inputs across the simdgroups, so a layer is four kernels back to back, and the next token's step is already submitted when this one ends.

The direction from here is a whole-step megakernel: one persistent dispatch that runs all 32 layers of a decode step against a static tile schedule, meeting at device-side barriers between stages. We built the first piece, a layer's whole MLP as one persistent dispatch with a grid barrier, and measured it honestly: identical output, and no faster. On this GPU, dispatches placed back to back in one encoder already run with almost no gap, so a megakernel only pays where it can overlap the next stage's weight loads with the current stage's tail. That is the version worth building, and it matters most for the eight-row verify step a trained draft will make the common case.

What's next

A better draft. Round four lands about 2 tokens a step on writing; later rounds continue from it on fresh data while the held-out score keeps improving. Every extra token a step is another 60 tokens per second on prose.
Husky in the app. Husky is live in Underdog on the Mac today, with no setup.
CUDA megakernels. The same approach, one model's shapes known at compile time, is being written for NVIDIA GPUs for our upcoming launches with NVIDIA and Microsoft.

Method

Apple M5 Max, 40-core GPU, 128 GB, macOS 26.5.1. Woof is Underdog's published 4B model at 4 bits, the same file for both engines; Husky repacks it without changing a value, and greedy output is compared token by token against MLX after every change. MLX is mlx-lm 0.31.3 on MLX 0.32.2. The sixteen-prompt suite ran on the evening of 20 September with a load average under 10 and the production app idle for every measurement; the cold prefill figures come from an earlier quiet window the same day. Runs that disagreed with their own repeats were discarded, and the receipts are kept.

Try it today

Husky is live in Underdog on the Mac. Download Underdog, connect your mail and calendar, and Woof runs on your own machine at these speeds today, with nothing sent anywhere and no setup. Get Underdog at underdog.ai.

Cite

Wen, Sigil. "Husky: a model-specific inference engine up to 4.5× faster than Apple's MLX." Underdog, 20 September 2026. https://husky.underdog.ai