Gemma 3 1B at 114 tokens per second on a laptop CPU

Research 7 October 2026 9 minute read

How a ternary conversion, a 1.6-bit kernel and a cheaper output layer made a small model 3.8x faster than llama.cpp on my laptop, and what it cost in quality.

I took Google’s Gemma 3 1B-it, retrained its decoder to use only three weight values (-1, 0 and +1), and wrote a CPU inference engine for it in C. On my 6-core laptop it generates 114 tokens per second for one user. The same model in llama.cpp’s 4-bit format runs at 30 tokens per second on the same machine.

Sounds too good? There is a catch, and it is quality: the converted model is about 14% worse on a standard benchmark, and you can feel it. More on that below.

Everything is open, go play with it:

• Model: huggingface.co/kunalsin9h/gemma-3-1b-it-ternary
• Engine and CLI: github.com/kunalsin9h/citric

On a CPU, speed is bytes

To produce one token, a model reads every one of its weights from memory. With one user there is very little arithmetic per byte, so the speed limit is simple:

tokens per second ≈ memory bandwidth / bytes read per token

My laptop (AMD Ryzen 7 7445HS, Zen 4, 6 cores) has a single memory channel. I measured 26 GB/s, and it does not go up with more threads. Gemma 3 1B in 4-bit reads about 500 MB per token, so it cannot go much faster than 50 tokens per second on this machine, whatever the code does.

So the plan was simple: read fewer bytes per token. Make the weights smaller, and stop reading weights I don’t need.

Step 1: a kernel for 1.6-bit weights

A ternary weight needs log2(3) = 1.58 bits. Most engines store it in 2 bits. I pack five weights into one byte (3^5 = 243 fits in 256 values), which is 20% smaller.

The usual way to compute with packed ternary weights on a CPU is a lookup table over groups of inputs. I measured that against plain int8 multiply-add with AVX-512 VNNI (vpdpbusd), and with the data in cache the two ran at about the same speed. So the gain from ternary weights comes from fewer bytes, and the arithmetic stays the same cost. My kernel unpacks the weights back to int8 and uses vpdpbusd.

Now the trick that makes the unpacking cheap. vpdpbusd multiplies groups of 4 bytes, but the 4 weights in a group do not have to sit next to each other. I put “trit number i” of 4 neighbouring bytes into one group, and reorder the input vector once per token to match. Then each unpacked plane is directly an operand: two byte-permute instructions and a blend, no transposes, no tables to build. The arithmetic is exact integer math, so results do not change with the thread count or batch size.

Before touching Gemma, I tested the kernel on a model trained ternary from the start, BitNet b1.58 large (729M parameters), against llama.cpp’s ternary format:

Generation is 1.6x faster, mostly from fewer bytes (161 MB per token vs 206 MB). Prompt processing is 11x faster because my engine runs all prompt tokens through each weight matrix in one pass, while llama.cpp’s ternary path handles them one at a time.

Step 2: making a ternary Gemma

There are very few good ternary models out there, so I made one. I converted the decoder of Gemma 3 1B-it: all the linear layers in its 26 blocks, about 700 million weights. The embedding table (262,144 words x 1,152) stayed as it was.

The recipe, on one rented H100 for about 6 hours and $27 (cheaper than a dinner out):

• Each linear layer becomes ternary, with one scale per matrix. The inputs to each layer are rounded to int8. Gradients pass through the rounding unchanged (the straight-through estimator).
• The rounding turns on gradually over the first 500 steps instead of all at once.
• The original model is the teacher. The loss is the KL divergence to its full output distribution over all 262k words, plus a little ordinary cross-entropy.
• Data: 230 million tokens of FineWeb-Edu, with 20% chat conversations (SmolTalk) to keep the chat behaviour.

After 230M tokens the model reaches a WikiText-2 perplexity of 31.4, against 27.6 for the original (lower is better). The curve was still going down when my budget ran out.

One lesson cost me a wrong conclusion first. During training I checked perplexity on one 82-token paragraph, and on that paragraph the final model looked better than the original. On WikiText-2’s 292,000 tokens it is 14% worse. Never judge a model on a few hundred tokens.

With the ternary decoder, the decoder weights for one token take 5.0 ms instead of 14.5 ms. But the total only went from 44 to 74 tokens per second. Where did the rest go?

Step 3: the output layer became the bottleneck

After the decoder shrank, the output layer ate more than half of every step. It scores all 262,144 words of Gemma’s vocabulary against the final hidden state, and it is still a 4-bit copy of the embedding table: 6.4 ms.

But to pick the next word you only need the few words with the highest scores. So the engine finds candidates first and scores only those exactly. I call it the sketch head:

1. Take the 32 dimensions of the hidden state with the largest values and score them exactly, for all words.
2. Score the other dimensions with a 1-bit copy of the table: only the sign of each value. One XOR and one bit count compare the signs of 64 dimensions for 8 words at once. The dimensions are grouped by size, and each group is weighted.
3. Keep the ~1,000 words with the best approximate scores, and compute their scores exactly. All other words get a score of minus infinity.

The approximate part alone is not reliable: the 1-bit sketch puts the correct word among the candidates only 86 to 99% of the time. The exact part over the strongest dimensions fixes that. In 758 checked positions (English text, code, Hindi), the sketch head picked the same word as an exact search over all 262k words every time. The output layer now takes 2.1 ms, and the model runs at 114 to 116 tokens per second.

I am not the first one to search for the best words instead of scoring all of them. A recent ICML paper, Accelerating LLM Inference via Vector Index Based Output Embeddings, uses a graph index (HNSW) for the same job on CPUs, also with Gemma 3. My method is a different, brute-force variant: it reads every word, but only 1 bit per value, and it picked the same word as the exact search on every position I tested.

Many users at once

With many users, the engine reads each weight byte once for all of them, and the limit moves from memory to arithmetic. The ternary model still wins, but by less (676 vs 589 tokens per second with 32 users), because unpacking the 1.6-bit weights costs a little arithmetic and arithmetic is now the limit. The engine runs prompt processing, batches and speculative decoding through one code path: every step is a set of “lanes”, where a lane is one token of one sequence at one position.

A side finding: Gemma 4 E4B in 4-bit

I also ran Gemma 4 E4B-it (effectively 4B parameters) without any training. In plain 4-bit it lost a lot of quality. To find out why, I wrote a full-precision reference in NumPy that loads one layer at a time (loading the 16 GB checkpoint the normal way crashed my laptop), and tested each kind of matrix in 4-bit on its own:

Two matrix types cause almost all of the damage: down_proj and the projection of the per-layer embeddings. They are 28% of the bytes. Gemma 4 also needs one activation scale per 32 values instead of one per token, because its residual stream has a few very large dimensions. With those two matrix types in 8-bit and everything else in 4-bit, perplexity was 15.96 against 17.65 for full precision on my test paragraph, the same within noise.

What did not work

I measured these too, so you don’t have to:

Limits

Watch out, read these before you use the model:

• Quality. 14% worse perplexity than the original. In chat it answers facts and simple arithmetic correctly, but it is wordier, it ignores some format requests, and on long agent-style prompts it can get stuck repeating itself.
• Long context. The engine supports Gemma’s 32K context, but I trained on 2K-token sequences. In my tests the model found a fact in a 9K-token prompt, but it lost the thread at 23K.
• Hardware. The engine needs x86 with AVX-512 VNNI (AMD Zen 4 or newer, Intel Ice Lake or newer). There is no ARM version yet.
• One machine. All numbers come from one laptop with one memory channel, on AC power. Runs vary by a few percent. The llama.cpp comparison uses llama.cpp’s standard 4-bit formats of the original model, not the same number of bits.

Try it

So lets go..

git clone https://github.com/kunalsin9h/citric && cd citric
make gemma3t
hf download kunalsin9h/gemma-3-1b-it-ternary model-trit5.safetensors tokenizer.json \
    --local-dir models/gemma-3-1b-it-ternary          # 744 MB
./citric "Explain a CPU cache in two sentences."
./citric --bench                                       # measure it on your machine

You need Linux, gcc, uv for the tokenizer, and access to Gemma on Hugging Face (accept the Gemma license once). If you run the bench, I would love to see your numbers.

What is next

The decoder and the output layer now both sit close to the memory limit of my laptop: about 186 MB per token, which caps one user at roughly 139 tokens per second. Going faster means reading fewer bytes, and that is a training problem:

• Trained sparsity. I wrote a kernel that skips groups of 4 zero inputs. It is 1.6-1.8x faster when 80% of the groups are zero, but the current model has almost none. Training with a ReLU-type activation and a sparsity loss could change that.
• A better conversion. A stronger teacher (Gemma 3 4B or 12B use the same tokenizer), longer training sequences, and agent-style data should fix most of the quality problems above.

Model: kunalsin9h/gemma-3-1b-it-ternary · Engine: kunalsin9h/citric · The model is a derivative of Gemma and is subject to the Gemma Terms of Use.

Tags:AIresearchtech