Apples® to Apples®: MLX vs llama.cpp for Gemma 4 12B - Ziraph blog

This is an engineering field report, structured like a paper for clarity - not a study. One M1, 16 GB, 3 runs per condition, dogfooding my own tool. No peer review, no claim of generality. The structure is for navigating the data, not a claim of rigor it doesn't have. Where a number is an upper bound or rests on a single machine, it says so.

Abstract

A fair fight between two local AI runtimes on the same workload and the same machine - Gemma 4 12B on a 16 GB M1 - at matched precision and under a stack of controls. The contestants are the lean mlx_lm runtime and raw llama.cpp (llama-cli), both running text-only at an effective ~4.9 bits per weight, both wrapped by ziraph and driven by one ziraph campaign file.

The going-in expectation was that MLX should win - that's what the literature sells on Apple Silicon. On this M1 it doesn't win, and it doesn't lose: decode is a tie. Short prompt or long, both runtimes generate tokens at the same ~7-7.8 tok/s, because both are pinned against the same memory-bandwidth wall (93-97% of M1's measured ceiling) and matched precision means matched bytes per token. The engine cannot change a bandwidth-bound number.

What does differ is everything around decode - a per-invocation startup tax (MLX's Python load is ~9 s against llama.cpp's ~3 s, enough to lose the one-shot short query) and a 2 to 4.5× CPU-energy bill from the Python orchestration layer - not decode itself, where GPU energy per token is a tie. And the headline "MLX is faster on Apple Silicon" turns out to be M4-class and baseline-dependent: most of the MLX "decode edge" I'd seen in an earlier pass against Ollama was the daemon's own overhead, not the engine.

The one-line story is that I wanted a fair "does MLX actually beat the Ollama/GGUF path for Gemma 4 on my Mac?" measurement, and getting to fair took three dead-ends and a 24 GB re-quantize I kept trying to avoid (§3.2). What follows is how the machine forced the answer out of me, one control at a time.

The result, up front - matched quant, short prompt, three runs a side, the full ziraph compare:

ziraph compare — Variant A (llama.cpp, Q4_K_M) vs Variant B (mlx_lm, mixed_4_6)
gemma-4-12B · short prompt · M1 16 GB · aggregate across 3 runs/side (± = across-run σ)

                              Variant A         Variant B          Δ         Δ%  Sig
energy_per_token J/tok      1.16 ± 0.13       1.37 ± 0.01      +0.21   +18% (A)  ≈1σ
tokens_per_second tok/s     6.92 ± 0.45       5.38 ± 0.02      -1.54   -22% (A)  SIG ⭐
reported decode tok/s              7.77              7.64      -0.13    -2% (A)  —
non-decode overhead %              10.9              29.6      +18.7  +172% (A)  —
─── Energy ──────────────────────────────────────────────────────────────────
mean_ane J              0.0000 ± 0.0000   0.0000 ± 0.0000    +0.0000          —  —
mean_gpu J                      184 ± 7           189 ± 0     +4.723    +3% (A)  ≈2σ
mean_cpu J                       10 ± 3            42 ± 2     +31.06  +297% (A)  SIG ⭐
duration s                        24.75             31.84     +7.090   +29% (A)  —
─── Average power ───────────────────────────────────────────────────────────
mean_ane mW             0.0000 ± 0.0000   0.0000 ± 0.0000    +0.0000          —  —
mean_gpu W                  6.57 ± 0.37       6.21 ± 0.11    -0.3607    -5% (B)  ≈2σ
mean_cpu W                0.724 ± 0.104     1.142 ± 0.048    +0.4181   +58% (A)  SIG ⭐
Media Engine W                   0.0169            0.0175    +0.0006    +4% (A)  —
peak Media Engine W              0.0227            0.0205    -0.0022   -10% (B)  —
─── Bottleneck Hints ────────────────────────────────────────────────────────
  Variant A:  (no hints fired)
  Variant B:  (no hints fired)
  Match — no hints fired on either side
Bandwidth (% of measured roofline):
  A  97% of M1 ceiling (61.4 GB/s, calibrated 2026-06-05) [bus-saturation/amc-sum]
  B  97% of M1 ceiling (61.4 GB/s, calibrated 2026-06-05) [bus-saturation/amc-sum]

ziraph compare output — short prompt, three runs a side, llama.cpp (Variant A) vs mlx_lm (Variant B).

Don't read it all yet. The story lives in three rows: reported decode is a tie (7.77 vs 7.64), the wall-clock tokens_per_second goes to llama.cpp (+22%, all of it the MLX startup tax in the non-decode overhead row), and mean_cpu J is +297% - MLX's real, consistent cost. By the end you'll read every cell at a glance, including the long-output table where the totals look like an MLX win and the per-token rows say otherwise.

---

1. Motivation & hypothesis

I wanted two things the day after Gemma 4 12B landed (2026-06-03, announced as a capable workload that runs in 16GB): to run it on my old M1, and to test a hypothesis I walked in with - that an MLX build would be faster than a non-MLX one.

I expected MLX to win comfortably, because everything I'd read sells it as the fast path on Apple Silicon. The practical question most people actually have is "MLX or Ollama?", but Ollama is a daemon wrapping llama.cpp, and I didn't want to measure the daemon - I wanted to measure the engine. So the clean experiment is mlx_lm against raw llama.cpp at matched quant, both in-process, with the consumer "which Ollama tag?" question kept as a separate sidebar (§6). Answering it means measuring the real per-workload telemetry - energy per token, GPU/ANE/CPU power, memory bandwidth, wall clock - grounded in the hardware rather than in vibes, which is exactly what ziraph exists to do, so the whole exercise doubles as dogfooding.

2. Background: why decode is a bandwidth problem

The going-in expectation isn't naïve - it's the consensus, since independent 2026 benchmarks put MLX ahead of llama.cpp by 21-87%, except that band carries three qualifiers that matter: modern mlx_lm (not the heavy mlx_vlm runtime), 4-bit weights, and M3-or-newer silicon (mostly M4-class data).

The mechanism matters, because autoregressive decode is memory-bandwidth-bound rather than compute-bound: each token streams the workload's weights through the GPU once, so tokens/sec is governed by how fast you can read DRAM and not by how many FLOPs you have. The runtime "advantage" comes down to which one extracts more of the bandwidth the chip has. MLX can pull ahead because it treats unified memory as the primitive (no "copy to GPU" step) and uses lazy evaluation to fuse ops and cut intermediate allocations - but on a bandwidth-starved M1 base (68.25 GB/s spec, ~61 GB/s achievable, as §5 confirms) the ceiling saturates both runtimes and that advantage collapses toward parity. That collapse is the whole result: M1 is the regime where the headline band shrinks to zero.

There is a second axis the headline benchmarks blur, prefill-and-startup versus sustained decode: a short turn is dominated by model load and prompt processing, a long generation by decode. That split, more than any single benchmark number, is the hinge between the two regimes below - and it's the axis where the runtimes actually differ, since decode itself turns out to be a tie.

3. Apparatus & method

3.1 Hardware & tool

The machine is a MacBook Air M1, 8-core GPU, 16 GB unified memory, macOS 26.6, and every measurement comes from ziraph wrapping the inference subprocess. The distinction that makes those numbers worth anything is that ziraph measures power, energy and memory bandwidth from IOReport hardware counters while it only scrapes each runner's self-reported decode rate, so reported decode tok/s and the wall-clock tokens_per_second are two genuinely different things

• one is pure decode, the other folds in per-run model load - and the gap between them, the

non-decode overhead row, is itself a result.

3.2 Models & matched quantization

The whole comparison hinges on both sides running the same weights at the same precision, and getting there took more work than I expected.

• llama.cpp's gemma4-12b.gguf is GGUF Q4_K_M, a mixed quant where most weights are 4-bit but the important tensors (attention V, FFN down) stay at 6-bit, for an effective ~4.96 bits per weight.
• The off-the-shelf MLX builds (mlx-community/gemma-4-12B-4bit) are uniform 4-bit, ~4.5 bpw - and on Gemma 4 they are garbage: asked for the capital of France, the 4-bit build emitted <image|> sixty times, a dead model. Re-converting from full precision with the current mlx-vlm converter fixed it.

3.2.1 Re-quantize it at home

Comparing Q4_K_M against uniform-4bit would measure the quantization scheme as much as the runtime (the trap some public benchmarks fall into), so the fix is a single re-quantize from the ungated BF16 base with a matching recipe:

# base (≈24 GB), ungated, Apache-licensed BF16 — same lineage as the GGUF
hf download unsloth/gemma-4-12b-it --local-dir models/gemma-4-12b-it-base

# re-quantize with the current converter + a Q4_K_M-matched scheme (mixed 4/6-bit)
mlx_vlm.convert --hf-path models/gemma-4-12b-it-base -q \
  --quant-predicate mixed_4_6 --mlx-path models/gemma-4-12B-mixed46

The result is 4.94 bpw (against Q4_K_M's ~4.96, within ~0.02 bpw - a genuine match), coherent output, both sides now running the same source weights at the same effective precision in the same mixed scheme. The only thing left varying is the runtime.

The 30-second sanity check - one short prompt, read the output - saved a 20-minute benchmark on a workload that was never going to produce a real number. Do it before you trust any convert.

3.3 Runtime choice: lean against lean

Gemma 4 12B is a gemma4_unified build (vision + audio + text), and the obvious way to run it is mlx_vlm, the multimodal runtime - except that stacks a torch/arrow/pandas import tax and two entirely unused towers on top of a text-only job. The fair fight is the lean mlx_lm runtime on a text-only build: patch the model_type tag (gemma4_unified → gemma4) and strip the stray vision_embedder.* weights, and mlx_lm loads it directly (Appendix B). Because the decode rate matches mlx_vlm on the same weights - the runtime cannot change a bandwidth-bound number - what the switch removes is purely startup, which is what makes it the right A-side: text-vs-text, lean-vs-lean against llama.cpp's lean GGUF path.

3.4 The control stack

Matching the quant was necessary and nowhere near sufficient, because a fair comparison turns out to be a stack of controls, each invisible until the machine shows it to you:

Confound	What it does	Control
Quant mismatch	measures precision, not runtime	re-quantize to mixed_4_6 (above)
Multimodal tax	mlx_vlm loads two unused towers	lean mlx_lm on a text-only checkpoint
Thinking default	gemma4 runs chain-of-thought by default → different workload	--reasoning off + enable_thinking:false → answer-only both
GPU compositing	external display + WindowServer inflate GPU energy	sleep the 5K display; Terminal.app not Ghostty
Cold start	first MLX run JIT-compiles kernels, cold page cache	2 untimed warmups per side
Interleaving	16 GB can't co-reside both → sequential	ziraph campaign … schedule=interleaved orders the runs; no shared thermal, flagged under Limitations

Using raw llama-cli instead of Ollama drops one more confound the live version of this post had to carry: GPU attribution. ollama run is a thin client that hands the work to a separate ollama serve daemon, so ziraph can't pin the GPU work to the wrapped process and flags "unattributed GPU load." llama-cli does its Metal work in-process, like mlx_lm, so both sides attribute cleanly and the asterisk is gone. (It comes back in the Ollama sidebar, §6, where it belongs.)

3.5 Protocol

Everything ran on a quiet machine - no concurrent builds, the 5K display asleep - because contention pollutes the GPU attribution. One ziraph campaign file per regime drives it:

• Runs: 2 untimed warmups + 3 measured runs per side, both regimes.
• Order: interleaved A/B with a 5-second cooldown between every run.
• Mode: answer-only on both sides.
• Prompts: short ≈170 tokens ("Explain attention mechanisms in one paragraph."); long up to 2048 tokens (a full Transformer-architecture tutorial).
• Attestation: variant, quant, model, runtime and methodology are baked into every trace by the campaign [[variant]] block.

The commands are in Appendix A.

4. Results

4.1 Short prompt - one-shot goes to llama.cpp, but only on startup

The full short-prompt comparison is the table up top. Three rows decide it:

• reported decode: 7.77 vs 7.64 - a tie (llama.cpp +2%, inside the run-to-run noise). The engines generate tokens at the same rate.
• Wall-clock tokens_per_second: 6.92 vs 5.38 - llama.cpp +22%. That gap is not decode; it's the non-decode overhead row: 10.9% for llama.cpp against 29.6% for MLX. Almost a third of every MLX invocation is startup - loading 6.9 GB plus mlx_lm's Python import. On a 170-token answer that startup dominates the wall clock.
• mean_cpu J: 10 vs 42 - MLX +297%. The Python orchestration layer. It barely moves the energy per token (1.16 vs 1.37 J/tok, since GPU dominates), but it's the one axis where the two engines genuinely separate, and it shows up in every condition.

mean_ane is ~0 on both sides because this is GPU work start to finish - exactly what LLM decode on Apple Silicon should look like, and a quiet confirmation ziraph isn't phantom-reading the ANE.

4.2 Long generation - the totals lie, the per-token rows don't

Same controls, up to 2048 tokens, three runs a side:

ziraph compare — Variant A (llama.cpp, Q4_K_M) vs Variant B (mlx_lm, mixed_4_6)
gemma-4-12B · long prompt · M1 16 GB · aggregate across 3 runs/side (± = across-run σ)

                              Variant A         Variant B          Δ         Δ%  Sig
energy_per_token J/tok      1.01 ± 0.14       1.00 ± 0.05      -0.01    -1% (B)  —
tokens_per_second tok/s     7.00 ± 0.08       6.83 ± 0.06      -0.17    -2% (A)  ≈1σ
reported decode tok/s              7.20              7.16      -0.04    -1% (A)  —
non-decode overhead %               2.8               4.6       +1.8   +64% (A)  —
─── Energy ──────────────────────────────────────────────────────────────────
mean_ane J              0.0000 ± 0.0000   0.0000 ± 0.0000    +0.0000          —  —
mean_gpu J                   1744 ± 109         1585 ± 24     -159.1    -9% (B)  SIG ⭐
mean_cpu J                       51 ± 3           109 ± 2     +58.24  +115% (A)  SIG ⭐
duration s                        274.6             257.7     -16.89    -6% (B)  —
─── Average power ───────────────────────────────────────────────────────────
mean_ane mW             0.0000 ± 0.0000   0.0000 ± 0.0000    +0.0000          —  —
mean_gpu W                  6.27 ± 0.21       6.18 ± 0.10    -0.0951    -2% (B)  ≈1σ
mean_cpu W              0.2296 ± 0.0221   0.4008 ± 0.0172    +0.1712   +75% (A)  SIG ⭐
Media Engine W                   0.0273            0.0282    +0.0009    +3% (A)  —
peak Media Engine W              0.0340            0.0311    -0.0029    -8% (B)  —
─── Bottleneck Hints ────────────────────────────────────────────────────────
  Variant A:  (no hints fired)
  Variant B:  (no hints fired)
  Match — no hints fired on either side
Bandwidth (% of measured roofline):
  A  94% of M1 ceiling (61.4 GB/s, calibrated 2026-06-05) [bus-saturation/amc-sum]
  B  93% of M1 ceiling (61.4 GB/s, calibrated 2026-06-05) [bus-saturation/amc-sum]

ziraph compare output — long generation (~2,048 tokens), three runs a side, llama.cpp (Variant A) vs mlx_lm (Variant B).

Read quickly, the mean_gpu J (-9%) and duration (-6%) rows look like an MLX win. They're an artifact. MLX hit end-of-sequence naturally at 1,765 tokens; llama.cpp ran on to its -n 2048 cap (its long outputs are visibly cut mid-formula). Different output lengths, not different speed - and the direction is the giveaway: generating ~14% fewer tokens should buy MLX ~14% less GPU energy, yet it saved only 9%, so per token llama.cpp is, if anything, the leaner one. The rows that normalize for length - the per-token ones - tell the truth: energy_per_token is a tie (1.01 vs 1.00), reported decode is a tie (7.20 vs 7.16), and wall-clock tokens_per_second is a tie the other way (7.00 vs 6.83, llama.cpp +2%).

This is the entire reason you compare per token and not per wall-clock - and the entire reason a profiler that surfaces the token count earns its keep. The mean_cpu J gap survives normalization (MLX +115%, ~2.5× per token) because it's real; the GPU "win" doesn't.

4.3 The bandwidth wall

The decode race is so close because both runtimes are pinned against the same memory wall, which ziraph reads directly off the IOReport counters - 93-97% of M1's measured 61.4 GB/s ceiling in every condition above. That saturation is the signature of a bandwidth-bound workload and the direct explanation for why the runtime gap is a couple of percent rather than the headline 21-87%: at matched precision both engines read the same bytes per token, and the wall, not the runtime, sets the pace.

One caveat for readers on other hardware: this live read is chip-specific. On M1 it's the real IOReport counter (the 93-97% above). On the chips where Apple no longer exposes a per-GPU bandwidth counter - M5 most notably - ziraph reports a derived estimate instead (flagged as such, and checked against the calibrated roofline), which reaches the same bandwidth-bound verdict by a different route.

4.4 A tie, when the headlines say MLX wins

The consensus sells MLX as 21-87% faster than llama.cpp on Apple Silicon, so a tie deserves scrutiny - and it survives, because the headline and this result are measured in different regimes, with the bandwidth wall between them.

That 21-87% band is overwhelmingly higher-tier silicon with bandwidth to spare - the vllm-mlx data is M4 Max - and broader roundups say it plainly: "on base-tier chips (M1, M2, M3) performance is roughly comparable; as you move to Pro, Max, and Ultra chips with more GPU cores and memory bandwidth, MLX increasingly pulls ahead" (local-llm.net, itself clocking MLX +26% on an M4 Max). Those same sources note the gap closing toward parity on the largest models "because memory bandwidth becomes the bottleneck." M1 base reaches that bottleneck early: at ~61 GB/s it has 4-6× less bandwidth than an M-Max or Ultra, so a 12B / ~4.9-bpw decode saturates the bus - the 93-97% roofline §4.3 measured - at a size that still leaves headroom on the bigger chips.

Two things narrow it further. Matched quant: the usual MLX-vs-GGUF benchmark pits MLX's native 4-bit (~4.5 bpw) against GGUF Q4_K_M (~4.96 bpw), handing MLX ~8% fewer bytes per token - which on a bandwidth-bound chip is ~8% of a "decode win" that owes nothing to the engine. §3.2 removes it. And the direction isn't even fixed: at long context MLX decode has been measured ~50% slower than llama.cpp (open; M3 Ultra, 146K) as its caching degrades - the same weak spot §7 flags. So this isn't "the literature is wrong." Its win is real on the chips and quants it was measured on, and it evaporates on a bandwidth-starved M1 at matched precision - the regime most M1 owners are actually in.

5. Validation: checking the tool that did the measuring

A field report that asks you to trust its numbers owes you a check on the instrument, so I validated the one the whole comparison rests on - memory bandwidth - against a path that shares no lineage with ziraph at all.

5.1 The roofline closes

If decode is bandwidth-bound, the measured bandwidth and the measured tok/s have to be consistent with a known ceiling. M1's spec is 68.25 GB/s; ziraph's own calibrate measures 61.4 GB/s on this M1, about 90% of spec, in line with published STREAM and memcpy results. The gemma runs pinning 93-97% of that ceiling on both runtimes are precisely the signature of a bandwidth-bound workload, and explain why the runtime gap is single digits.

5.2 ziraph self-validation on a dense model

The clean test uses a dense, uniformly-quantized workload (llama-2-7B Q4_0), where bytes-per-token is unambiguous, so model_bytes × tok/s becomes an independent bandwidth estimate that never touches IOReport:

Check	ziraph (IOReport)	Independent	Δ
decode bandwidth	57.8 GB/s (peak 58.2)	56.1 GB/s (3.83 GB × 14.67 tok/s)	+3%
GPU peak power	6.53 W	6.65 W (powermetrics)	−2%
decode rate	14.67 tok/s	14.00 (llama-bench) / 14.15 (ref #4167)	+5% / +4%

The bandwidth number is the keystone - ziraph's IOReport plateau lands +3% off a first-principles estimate computed with no shared code path - and the power agrees with Apple's own powermetrics to −2%. The harness is in Appendix A.

5.3 What I could not validate

Energy per token is the weakest leg: no external M1 reference for joules-per-token exists, and ziraph's GPU joules are a system-wide upper bound - on Apple Silicon the GPU energy is measured chip-wide, so the absolute figure folds in WindowServer and the terminal. On a quiet machine that bound is tight, and it's common-mode - both sides ran under the same compositor, so it largely cancels in the Δ. The signal I trust is the GPU power delta and the decode rate, both pointing the same way; the energy-per-token direction is solid, the magnitude wants more runs before I'd quote it hard.

6. Sidebar: the real-world Ollama tags (short and long)

The clean experiment above isolates the engine. The question a normal person actually asks is narrower: Ollama ships two tags - gemma4:12b and gemma4:12b-mlx - which do I pull? So I ran them against each other, out of the box, no re-quantize. One caveat up front, and it's load- bearing: this is not a matched comparison. gemma4:12b is GGUF Q4_K_M (~4.96 bpw, ~11.9B params); gemma4:12b-mlx is nvfp4 (~4.5 bpw, ~13.0B params - the full unified architecture is in the weights, though Ollama lists this tag as text-only). It blends three variables at once - engine, quant, and model scope (text-only vs full multimodal) - so read it as a consumer data point, not an engine verdict. I ran both regimes - short (startup-dominated) and long (decode-dominated, MLX's more favorable one) - so the two tables below bracket the tag question from both ends.

With all of that stipulated, on the short query the default GGUF tag came out ahead of the MLX tag on every axis:

ziraph compare — Variant A (Ollama gemma4:12b, Q4_K_M) vs Variant B (gemma4:12b-mlx, nvfp4)
gemma-4-12B · SHORT prompt · M1 16 GB · 3 runs/side · ⚠ not matched (quant + scope differ)

                             Variant A       Variant B         Δ        Δ%  Sig
energy_per_token J/tok   1.331 ± 0.032   1.436 ± 0.094   +0.1053   +8% (A)  ≈2σ
tokens_per_second tok/s     5.54 ± 0.47     5.27 ± 0.32   -0.2734   -5% (A)  ≈1σ
reported decode tok/s            7.063           6.750   -0.3133   -4% (A)  —
non-decode overhead %            21.52           21.93   +0.4067   +2% (A)  —
─── Energy ───────────────────────────────────────────────────────────────
mean_ane J             0.0000 ± 0.0000 0.0000 ± 0.0000   +0.0000         —  —
mean_gpu J †                   184 ± 9        193 ± 15    +9.154   +5% (A)  ≈1σ
mean_cpu J †                    22 ± 6         45 ± 16    +23.74 +110% (A)  SIG ⭐
duration s                       26.22           29.67    +3.447  +13% (A)  —
─── Average power ────────────────────────────────────────────────────────
mean_ane mW            0.0000 ± 0.0000 0.0000 ± 0.0000   +0.0000         —  —
mean_gpu W                 6.76 ± 0.50   6.233 ± 0.088   -0.5248   -8% (B)  ≈2σ
mean_cpu W               0.986 ± 0.135     1.64 ± 0.29   +0.6498  +66% (A)  SIG ⭐
Media Engine W                  0.0212          0.0262   +0.0050  +24% (A)  —
peak Media Engine W             0.0275          0.0316   +0.0041  +15% (A)  —
──────────────────────────────────────────────────────────────────────────
† system-wide (upper bound) — not process-attributed; both sides run via the ollama daemon
─── Bottleneck Hints ─────────────────────────────────────────────────────
  Variant A:  Unattributed GPU load during run
  Variant B:  Unattributed GPU load during run
  Match — both fired: Unattributed GPU load during run
Bandwidth (% of measured roofline):
  A  91% of M1 ceiling (61.4 GB/s, calibrated 2026-06-05) [bus-saturation/amc-sum]
  B  92% of M1 ceiling (61.4 GB/s, calibrated 2026-06-05) [bus-saturation/amc-sum]

ziraph compare output — short prompt, three runs a side, Ollama gemma4:12b (Variant A) vs gemma4:12b-mlx (Variant B). Not a matched comparison — quant and model scope both differ.

Even the small decode difference on the short query is scope and noise, not bytes: the two tags differ in quant (Q4_K_M vs nvfp4) and in what they pack - the MLX tag is the full gemma4_unified build (13.0B on disk: the text decoder plus vision and audio towers), the GGUF is text-only (11.9B). Those towers never move during text decode, so they cost decode speed nothing; and nvfp4's slightly lower precision means, if anything, fewer bytes per token, not more. What's unambiguous is the CPU cost - the MLX tag draws 2× the CPU energy (mean_cpu J +110%, SIG) - which echoes the matched experiment. The honest takeaway for the tag question is narrow: out of the box, on a short query, the default gemma4:12b was the snappier, leaner pick.

That left one promise open - the long half, the decode-dominated regime where I expected the MLX tag to close the gap. It does. Same two tags, same machine, up to 2,048 tokens:

ziraph compare — Variant A (Ollama gemma4:12b, Q4_K_M) vs Variant B (gemma4:12b-mlx, nvfp4)
gemma-4-12B · LONG prompt · M1 16 GB · 3 runs/side · ⚠ not matched (quant + scope differ)

                              Variant A         Variant B          Δ         Δ%  Sig
energy_per_token J/tok    0.987 ± 0.020     0.993 ± 0.016     +0.006    +1% (A)  ≈1σ
tokens_per_second tok/s   6.517 ± 0.016     6.506 ± 0.023     -0.011    -0% (A)  ≈1σ
reported decode tok/s             6.663             6.647     -0.017    -0% (A)  —
non-decode overhead %             2.193             2.121     -0.072    -3% (B)  —
tokens_total tok                   5602              5144       -458    -8% (A)  —
─── Energy ──────────────────────────────────────────────────────────────────
mean_ane J              0.0000 ± 0.0000   0.0000 ± 0.0000    +0.0000          —  —
mean_gpu J †                  1815 ± 87         1665 ± 65     -149.9    -8% (B)  SIG ⭐
mean_cpu J †                     71 ± 4            80 ± 1     +8.632   +12% (A)  SIG ⭐
duration s                        286.5             263.5     -23.01    -8% (B)  —
─── Average power ───────────────────────────────────────────────────────────
mean_ane mW             0.0000 ± 0.0000   0.0000 ± 0.0000    +0.0000          —  —
mean_gpu W                  6.31 ± 0.13     6.289 ± 0.067     -0.017    -0% (B)  ≈1σ
mean_cpu W                0.266 ± 0.006     0.322 ± 0.004     +0.055   +21% (A)  SIG ⭐
Media Engine W                   0.0262            0.0281    +0.0020    +8% (A)  —
peak Media Engine W              0.0302            0.0307    +0.0005    +2% (A)  —
─────────────────────────────────────────────────────────────────────────────
† system-wide (upper bound) — not process-attributed; both sides run via the ollama daemon
─── Bottleneck Hints ────────────────────────────────────────────────────────
  Variant A:  Unattributed GPU load during run
  Variant B:  Unattributed GPU load during run
  Match — both fired: Unattributed GPU load during run
Bandwidth (% of measured roofline):
  A  86% of M1 ceiling (61.4 GB/s, calibrated 2026-06-05) [bus-saturation/amc-sum]
  B  96% of M1 ceiling (61.4 GB/s, calibrated 2026-06-05) [bus-saturation/amc-sum]

ziraph compare output - long prompt (~1,800 tokens), three runs a side, Ollama gemma4:12b (Variant A) vs gemma4:12b-mlx (Variant B). Not a matched comparison - quant and model scope both differ.

On sustained decode the two tags converge: reported decode is a dead tie (6.66 vs 6.65 tok/s) and energy_per_token is a tie too (0.99 vs 0.99 J/tok). The MLX tag's lower GPU-energy total (-8%) is once again a length artifact - it generated 8% fewer tokens (5,144 vs 5,602) - while the one real penalty it carried on the short query survives: CPU energy (mean_cpu J +12%, with +21% mean CPU power). So the consumer arc completes cleanly: gemma4:12b is the snappier pick on a short query, the two are interchangeable on sustained decode, and the MLX tag costs more CPU throughout.

7. Limitations

n = 3 runs per condition, one M1, one macOS version; the campaign is sequential (16 GB can't co-reside both workloads), so the two blocks don't share a thermal history. GPU power and energy are a system-wide upper bound rather than process-attributed (§5.3). llama-cli's conversation mode reports a decode rate but not an exact token count, so its per-token figures use the reported rate over the decode window (which pins the long run to its observed ~2048-token cap and the short run to ~170, matched to MLX's length); the MLX side reports an exact count directly. The thinking-parity control assumes --reasoning off and enable_thinking:false produce equivalent answer-only workloads, checked by output shape.

Above all, this is a single-turn measurement: one prompt, one generation, so prompt/KV caching never engages and every run is cold. That makes the decode tie a claim about single-turn throughput only - it says nothing about agentic, multi-turn use, where prompt-cache reuse dominates and MLX has a documented weak spot. mlx-lm's prefix-cache reuse has been reported broken for hybrid-architecture models - sliding-window and SSM/Mamba designs, a class that explicitly includes the Gemma family (alongside Qwen 3.5 and GPT-OSS), per #980 - so a multi-turn agent may pay a full prompt recompute every turn where llama.cpp's more elaborate caching reuses it. I ran single-turn and didn't exercise that path (nor test whether the latest mlx-lm has narrowed the gap), but if anything it tilts the practical picture further from MLX, not toward it. None of this is peer reviewed - it is one developer, one Mac, measuring carefully and showing the work.

8. Honest considerations - what's new, what isn't

Let me be straight about what this is worth, because the easiest critique is the right one to answer: didn't you just re-confirm what everyone knew? On speed, mostly yes - decode is bandwidth-bound, base-tier chips tie, MLX needs headroom to pull ahead, and §4.4 cites others who said it. What this adds is the corrected version: the usual "MLX is faster" taken apart into the ~8% that's quant mismatch, the daemon overhead, and the output-length artifact, with the tie pinned to a measured 95% roofline instead of a hand-wave. A myth-correction and a method, not a discovery - and I'd rather say so than oversell it.

Both corrections are the kind of thing honest measurement exists to catch. The "MLX wins long" first read was just output length - MLX stopped 283 tokens short of llama.cpp's -n cap. And the "MLX decode edge" I'd reported in an earlier pass was measured against Ollama; against raw llama.cpp it vanishes - 7.20 vs 7.16, dead level. The headline "MLX is faster on Apple Silicon" is real, but it's M3/M4-class and measured against the right baseline; on a bandwidth-starved M1 at matched quant it's a tie, and the regime, not the runtime, decides the wall clock. Practically, that makes the choice regime-shaped, not winner-shaped: llama.cpp for snappy, one-shot, interactive use; for long batch they're equivalent, so you'd pick on ecosystem, not speed.

The one number nobody else publishes is the energy split. Every Apple Silicon benchmark reports tokens per second; none report joules. Matched, the two engines tie on speed and on GPU energy per token - but MLX spends 2 to 4.5× the CPU energy doing it, the cost of a Python orchestration layer over a C++ one. Small in absolute watts, but real, repeatable, and the only axis the engines actually separate on. If there is a thread here for an MLX engineer to pull, it is that one - the per-token CPU path, not the GPU kernels.

And the durable idea isn't a verdict at all, it's the roofline. Telling you that on your chip, your workload sits at 95% of the bandwidth wall answers the question the leaderboards can't: whether switching runtimes could help you at all (here, no - buy bandwidth or quantize smaller), and, for the people building these engines, that base-chip decode is a ~5%-headroom game where the real wins live in prefill, caching, startup, and energy instead. (The bottleneck-hint engine fired blank on this run - both sides came back "no hints" - the honest outcome on a clean, well-saturated workload: here the roofline number did the work, not the hints.)

Where ziraph falls short of actually helping the engine teams: it tells you that MLX burns more CPU, not where it goes - no split across tokenizer, sampling, and the decode loop, no per-operation attribution. A flag, not a fix. The honest state of things is that this profiler is good at catching what a careless comparison gets wrong, and at putting an energy number on the result that nobody else does - and the genuinely useful sequel is the version where the tool can also say which line is spending the joules.

And it's still only the single-turn story. For multi-turn agentic use, prompt-cache reuse - where MLX has documented gaps for the sliding-window models Gemma belongs to (§7) - matters more than raw decode, and on current evidence would widen the gap in llama.cpp's favour, not MLX's. That's the next measurement, not this one.

Appendices

A. Reproducing it

Each regime is one declarative ziraph campaign file - two [[variant]] blocks, 2 untimed warmups + 3 measured runs each, schedule = "interleaved". The short-prompt campaign, verbatim (paths are relative to the run directory):

# gemma-4 12B — MLX vs llama.cpp, matched quant, SHORT prompt (one-shot regime).
# Both sides are lean in-process subprocesses, so the only variable is the engine:
#   llama.cpp = GGML/Metal backend (llama-cli) on a Q4_K_M GGUF  (~4.96 bpw)
#   MLX       = mlx_lm.generate on a mixed_4_6 build             (~4.94 bpw)
# Both load, run, and free memory on exit (no daemon), so ziraph attributes GPU work to
# each process directly and only one model is resident at a time — schedule="interleaved"
# is thermally fair and safe on 16 GB. Thinking is off on both sides (answer-only).
#
#   ziraph campaign campaign-llamacpp-vs-mlx-short.toml

schema           = "ziraph-campaign/1"
name             = "gemma4-12b-llamacpp-vs-mlx-short"
out_dir          = "campaign-out/llamacpp-vs-mlx-short"
runs_per_variant = 3
schedule         = "interleaved"

[methodology]
name = "default"   # 2 untimed warmup + 3 measured runs per variant

[output]
aggregate      = true
compare        = true
compare_format = "text"

# ---- llama.cpp: GGML/Metal backend on Q4_K_M ----
[[variant]]
label   = "llamacpp-gguf"
command = ["llama-cli", "-m", "models/gemma4-12b.gguf", "-p", "Explain attention mechanisms in one paragraph.", "-n", "512", "-st", "--jinja", "--reasoning", "off"]
model_name      = "gemma4-12b.gguf"
quantization    = "Q4_K_M"
runtime_version = "llama.cpp-b9330"

# ---- MLX: mlx_lm on mixed_4_6 ----
[[variant]]
label   = "mlx-lm-mixed46"
command = ["mlx_lm.generate", "--model", "models/gemma-4-12B-text-mlxlm", "--prompt", "Explain attention mechanisms in one paragraph.", "--max-tokens", "512", "--chat-template-config", "{\"enable_thinking\": false}"]
model_name      = "gemma-4-12B-text-mlxlm"
quantization    = "mixed_4_6"
runtime_version = "mlx-lm"

The long-prompt campaign is that same file with two args changed per variant - swap the prompt and raise the cap to 2048 (plus the name/out_dir suffix to -long):

  name = "gemma4-12b-llamacpp-vs-mlx-long"    # out_dir → campaign-out/llamacpp-vs-mlx-long

  # llama.cpp variant — only these two args change:
- "-p", "Explain attention mechanisms in one paragraph.",     "-n", "512",
+ "-p", "Write a thorough … Transformer-architecture tutorial …",     "-n", "2048",

  # mlx_lm variant — only these two args change:
- "--prompt", "Explain attention mechanisms in one paragraph.",     "--max-tokens", "512",
+ "--prompt", "Write a thorough … Transformer-architecture tutorial …",     "--max-tokens", "2048",

The instrument check (§5.2) wraps a dense llama-2-7B Q4_0 run and compares ziraph's measured bandwidth against model_bytes × tok/s. Never -no-cnv with llama-cli - it ignores -n and runs away.

B. The lean mlx_lm text-only path

Gemma 4 12B ships as gemma4_unified (vision + audio + text). mlx_lm already supports the text backbone, so present it a text-only checkpoint: patch the model_type tag to gemma4 and drop the unused vision_embedder.* weights. That sheds mlx_vlm's torch/arrow/pandas startup tax and the two unused towers. The decode rate is unchanged (bandwidth-bound on the same weights); what the switch buys is purely the startup.

C. Per-run consistency

The per-run spread behind the §4 aggregates - reported decode rate, the metric the runners report directly:

Condition	reported decode tok/s (per run)
llama.cpp, short (×3)	7.8 / 7.8 / 7.7
mlx_lm, short (×3)	7.647 / 7.650 / 7.608
llama.cpp, long (×3)	7.2 / 7.2 / 7.2
mlx_lm, long (×3)	7.119 / 7.171 / 7.175

The runs are tight - which is why the §4 "tie" is a real tie, not a wide-error shrug.

D. The profiler caught its own tool

Reading these traces surfaced three bugs in ziraph itself, all since fixed. The roofline line was written to stderr and dropped when a comparison was saved to a file; the "unattributed GPU load" hint fired on every GPU run instead of only when ziraph genuinely couldn't attribute the work; and calibrate read over 100% of its ceiling - impossible, so the ceiling was wrong - because its Metal bench timed each pass with a wall-clock window that folded CPU submission overhead into a ~1 ms GPU pass, under-reading by ~12% (51 vs the true 61.4 GB/s). Reading the GPU's own start/end timestamps fixed it; the M1 ceiling is now ~90% of spec, and these runs read a believable 93-97%. Flagged by the tool, on the tool - about as on-brand as it gets.

E. Sources

• Ollama MLX backend + tag scheme: https://ollama.com/blog/mlx
• Confounded public GGUF-vs-MLX gemma4 benchmarks (mismatched quants): https://theagenttimes.com/articles/gguf-outpaces-mlx-for-gemma-4-on-apple-silicon-developers-re-ca3911a0
• Off-the-shelf gemma-4 4-bit garbage + the mlx-vlm converter fix: https://github.com/Blaizzy/mlx-vlm/pull/893
• The 21-87% throughput band (M4-class data): vllm-mlx, "Native LLM and MLLM Inference at Scale on Apple Silicon," arXiv 2601.19139
• MLX prompt-cache gaps (multi-turn / agentic): mlx-lm #980 (prefix-cache reuse broken for hybrid-architecture / sliding-window / SSM models - Gemma, Qwen 3.5, GPT-OSS)
• MLX mixed-quant recipes: https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/convert.py
• Ungated BF16 base: https://huggingface.co/unsloth/gemma-4-12b-it
• MLX-leads-sub-14B + mechanism: Groundy; prefill weakness: Kunar
• M1 STREAM / achievable bandwidth: arXiv 2502.05317; llama.cpp Discussion #4167 (7B-Q4 @ 14.15 tok/s, M1 8-core)