Benchmarking Qwen3.8 27B quantizations: 4-bit holds up, 1-bit collapses

> Second, besides noise (bars are Wilson 95% confidence intervals, very conservative for run-to-run noise), there is little difference down to 4-bit; only the 2-bit scores a bit lower.

Confidence intervals have nothing to do with run-to-run variation. They have little to do with anything people usually ascribe to them (https://link.springer.com/article/10.3758/s13423-015-0947-8 ), but even less with run-to-run variation (https://link.springer.com/article/10.1007/s10654-016-0149-3 misconception 22).

I would be very interested in a similar benchmark for *KV cache* quantizations.

I use Qwen3.8 27B Q4_K_M for coding sometimes and therefore need a relatively long context. I settled on q8_0 because it is the only way to fit the model + 100k tokens into 24GB VRAM, but still wonder what am I loosing in quality, and what other options are there.

I also heard that KV cache quantization matters more with longer contexts. It may be interesting to benchmark this too: what the quality looks like on different combinations of model quantization × KV cache quantization × context size.

This confirms a theory I have to explain the minimal loss in quality when using lower quants (I use IQ3_XXS with an 8-bit KV cache) and the XHIGH (default) thinking level.

It's well-known that while quantization affects the sampling probability distribution (given the same context, which next token is the most probable), Qwen 3.8 27b seems to offset that by just thinking more and as a result eventually finishing the task (benchmark or otherwise).

So as long as the thinking (albeit longer) is sound, this leads to the same success rate (as shown in the article) but potentially at the cost of more tokens and hence more time.

I think it'll be further useful to chart each quantization's used tokens as well, in addition to the success rate.

Thanks for doing and sharing the research!

There's a real hole here at Q3. A critical breakpoint here is sub 16-GB cards, which covers the 5080, 5070 Ti, 5060ti, and several other cards from this generation and the last. It would be instructive to see where the quality knee is.

hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"?

Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc),

But on the other hand, the result that I'm assuming the author means to communicate ("on these evals, generation quality seems fairly good") sounds worthwhile to share?

Because I really struggle with this question at the moment. Am I allowed to draw an adverse inference that "if the writeup presents irrelevant text side by side with the data, then this may be a sign that the author does not understand the task that they are attempting to write up"?

Measuring quality e2e definitely makes sense.

But I think there is a bit more to this:

> Measuring token prediction differences (KL-divergence, top-1 predictions) is easy, but it does not tell us whether the model gets worse at solving tasks.

A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext - maybe because of tradition, to make numbers more comparable? It's not measuring how well the model follows bf16 on agentic tasks.

I've been trying to check KLD recently for some quants of Qwen 3.8 27B, and the numbers are dramatically different, depending on which dataset you use. KLD computed on agentic traces is much higher, and top-1 % is way lower than if you compute it on chats or wiki text.

You look at a published number, and see "oh, nice, top1 is 99% - quant is different just in 1 token out of 100", but chances are it's computed on wiki, and on agentic / coding it can be 10 tokens out of 100.

Common intuition is that on agentic tasks errors compound, and that's why it degrades more than metrics show - but maybe the metrics themselves are also wrong, too optimistic.

Still investigating it though :)

I always wonder is it safe to run one of these models on a personal pc, or do you guys recommend something like docker, sorry a bit new to all of this.

If you want to do a 1-bit model you have to QAT at pre-training with way more data than chinchilla to compensate for the cliffs (like 50x). Quantization on an existing pre-trained model will almost always collapse at 1-bit

I have a very interesting self-made coding benchmark, very intricate and technical, but 100% a real world problem I had to solve. I’m not going to further elaborate, since I don’t want future models to train on the solution.

To my own surprise, Q6_K_XL (from unsloth) comes up with a solution, anything Q5 doesn’t. To further surprise me, so far only the XL Q6 variant managed to solve it.

The problem, at least as stated, seems to be right on the edge of what the Q6 quantization can do.

Unfortunately even a successful run is rather long, so I don’t have a whole lot of data.

But the whole thing sure made me doubt the common idea that you wouldn’t perceive a difference until crossing past 4 bits quantization.

> As you may see, the scores are around the random guessing level, with the smallest model being below that threshold.

Err… can someone explain to me what is meant here? Surely the model wouldn’t consistently “guess wrong” compared to randomly, as that would be better. I guess some things like general coherency (i.e. is it even readable or gibberish) factor into that score?

I’m more curious how each 4 bit quant compares. It seems like NVFP4 outperforms Q4_K_M in terms of speed and top 1 but is only good for expensive Nvidia cards

I use a 2-bit quant from Unsloth on my MBP M5 32GB of RAM. It run slower than molasses at 2 too/s kind of thing. Not sure it is usable at that rate for anything.

Note that these quants are not quantized uniformly, so 4-bit isn't actually a "true" 4-bit here, so these observations won't necessarily hold up to other quants which might be done differently.

1-bit, 2-bit, penny and dime.

Could this be used to estimate how many fingers LLM have?

It's strange that the author has completely ignored the 3 bit quants which allow someone with a 16GB GPU to have 100-120k and still get full performance. You can't run any of the 4-bit quants on a 16GB gpu with enough context to be useful for all but the most basic tasks.

General purpose agents can need up to 30k just to reply with "1+1=2" because their prompting is so overloaded. 60-70k is decently usable, still not great for anything complex. A long running task in a general purpose agent can easily hit 100k.

What the vast majority of people care about is performance around what desktop consumer GPUs can run. 8GB, 10, 12, and 16GB of VRAM. What do models that will run at full performance, do?

Also important to know is how Qwen3.8-27B stacks up against qwen3.6-35B-A3B, which due to being MoE, will run on a 16GB card with plenty of speed 90% of the time, at higher quant - so you get more parameters and better quant. But 3.8 is supposed to be "better", so...?

Isko ek karna hai

Those benchmarks are very interesting.

But is there any model that actually works in a decent way at quantization of 1?

Qwen3.8 27B seems like it was clearly supposed to be a high-end consumer open-weights model, but the t/s is so low for me on my old M1 Max 64GB that I hope others are getting use out of it.

Unfortunately, the calculus has changed and it seems cheaper to me to just use MiMo V2.5 for pennies or DeepSeek V4 Flash instead of using Qwen anymore unless I need a local model specifically for doing reverse engineering work that gets otherwise rejected.

the 4-bit matching bf16 on terminal-bench is a useful data

I've been running Unsloth IQ3_S on my 5060ti with mmproj offloaded, getting 600-1000 prefill and 30-50 tg with this config:

       /data/llm/llama.cpp/build/bin/llama-server
        --threads 4
        --threads-batch 8
        --batch-size 4096
        --ubatch-size 256
        --port 9999
        --temp "1.0"
        --top-p "0.95"
        --top-k "20"
        --min-p "0.0"
        --presence-penalty "0.0"
        --reasoning auto
        --reasoning-preserve
        --reasoning-budget 4096
        --gpu-layers-draft all
        --spec-type draft-mtp,ngram-map-k4v,ngram-mod
        --spec-draft-n-max 3
        --spec-draft-p-min 0.75
        --spec-ngram-mod-n-match 24
        --spec-ngram-mod-n-min 4
        --spec-ngram-mod-n-max 16
        --spec-ngram-map-k4v-size-n 8
        --spec-ngram-map-k4v-size-m 16
        --spec-ngram-map-k4v-min-hits 1
        --n-gpu-layers all
        --ctx-size 131072
        --repeat-penalty 1.0
        --jinja
        --metrics
        --model /data/llm/models/unsloth/Qwen3.8-27B-UD-IQ3_S.gguf
        --chat-template-file /data/llm/models/qwen3.6-chat-template.jinja
        --fit off
        --flash-attn on
        --cors-origins localhost
        --mmproj /data/llm/models/unsloth/Qwen3.8/mmproj-BF16.gguf
        --no-mmproj-offload
        --parallel 1
        --kv-unified
        --cache-type-k q4_0
        --cache-type-v q4_0
        --cache-type-k-draft q4_0
        --cache-type-v-draft q4_0

local llm don't make sense currently consumer compute is not upto mark it may take atleast 7 more years to be usable