Quantization hurts knowledge nonlinearly - Qwen3.6 27B case study

Piotr Migdał3 August 2026

In previous blog posts, I was both raving about Qwen3.6 27B and investigating how its quantization affects pelican-making capabilities and benchmark scores. SVGs with pelicans on a bike were not affected by anything at 4 bits or more, and even for 3-bit and some 2-bit models the quality difference was small, visible only when averaged over many samples. Likewise, differences in benchmark results were surprisingly small.

So, why is it that everyday experience says smaller models are “worse”? While intelligence might not be that space-heavy, factual knowledge is incompressible - you cannot put the whole internet in a few GBs, no matter how hard you try. Incompressible Knowledge Probes (IKP) is a 1,400-question trivia benchmark by Bojie Li. As we see, there is a strong linear correlation between the IKP accuracy and the logarithm of the number of parameters:

If you are curious about the questions spanning 7 tiers of obscurity, here are a few examples:

• T1: “What strait separates Sicily from mainland Italy?” - Strait of Messina
• T2: “What is the second largest moon of Saturn?” - Rhea
• T3: “Who composed the opera ‘Prince Igor’?” - Alexander Borodin
• T4: “What is the highest mountain in Oceania?” - Puncak Jaya
• T5: “In what year was Norwegian Olympic Museum founded?” - 1997
• T6: “In computer science, what is the research subfield of Myung Hoon Sunwoo?” - computer architecture
• T7: “In what year was Pont 1 (sot de les Mines) in El Brull, Spain opened?” - 1901

For reference, the score of the original model is the following:

Let’s see how the score is affected by 55 quantizations: ones on Hugging Face by Unsloth and Bartowski, as well as ones by llama.cpp, with llama-quantize. We used llama.cpp to run these, locally on my MacBook M5, and on Modal. First, let’s decompose it by tier:

So it all changes with model size, and the most obscure are lost first.

Let’s chart IKP accuracy as a function of the logarithm of model size.

Quantizations above 20 GB (5-bit or more) show the same quality as the original. Among 3-bit models there is a drop, which gets even stronger in the 2-bit range.

However, unlike in the main chart, the dependency is not linear. I had to double-check that it is not a problem with using different variables or scaling. So, let’s overlay it on the full chart.

Here it gets interesting. We get an arc over the chart. While Qwen3.6 27B already has a tagline “punches above its weight”, its slimmed-down 4-bit variants make this statement even… punchier (please excuse me, I just couldn’t resist).

Size matters, but divergence even more

This might look puzzling. Across models, IKP accuracy grows linearly with the logarithm of size. Within quantizations of a single model, the curve rises steeply, then flattens. When compressing a model (and quantization is a lossy compression), what matters is not the size itself, but the statistical distribution of generated tokens. So, another chart might be more revealing, Kullback–Leibler divergence from the original, floating-point BF16 model:

Here we have a beautiful linear correlation.

My interpretation is the following - when compressing a model a bit, we lose nothing, just getting rid of some needless noise. So we save space, at no expense.

But at some point the distribution starts breaking down, and quality falls more rapidly than it would with just training a smaller model. Because if we train a smaller model, we still optimize it for the training data. Quantization is blind. Even ones by Unsloth and Bartowski optimize for closeness to the original distribution, not fine-tuning data.

Consequences

This has two consequences. First, quantization is the way to go - you just need to know how far you can push it, so that there is no noticeable loss of quality. Usually you want to use the biggest model that fits your GPU RAM and leaves enough space for context. Unless you need your GPU RAM (or shared RAM, if it is Apple Silicon) for something else, there is no benefit in using cheaper models, usually there is no speed benefit.

While in general “the bigger the better”, at some point there are diminishing returns. In the case of Qwen3.6 27B, an 8-bit quantization (e.g. Q8_0 from Unsloth) is a safe bet. If you are using a good 4-bit one (e.g. Q4_K_M from Bartowski), you are likely to be fine. As we already saw, the pattern of pelican generation is different from that of factual knowledge. It may happen that for your particular problem there is a different tripping point.

Second, not only intelligence matters. A smart model without adequate knowledge will fall short.

If you use your model to code in Python, you may not care if it even knows the easiest Tier 1 facts like the one about the Strait of Messina. At the same time, there are millions of pieces of niche trivia about the Python package ecosystem - each package with its API, patterns, and version-dependent quirks. All do matter. Most cannot be deduced at all. So either you have a model that is good at searching documentation, code examples and bug reports (and you give it enough time), or it will fail.

In models you use, did you notice a tripping point of model quantization?

Stay tuned for future posts and releases