BitBop trains ternary language models — every weight is in {-1, 0, +1} — without keeping
full-precision latent weights in the ternary body. Instead of a shadow float copy per weight
(the STE recipe), the only per-weight training state is a small bf16 flip momentum (the tied
token embedding and output head stay float with Adam — a small fraction of the parameters). That
makes the optimizer state ~5× smaller, so larger models fit in the same GPU memory, and the shipped
weights are int2-packed (2 bits/weight stored; ~1.3 bits of entropy at ~63% sparsity, below BitNet's 1.58).
This repository is the reference implementation and the reproduction kit for the technical report. It is a proof of concept, trained on a single RTX 3060 (6 GB) and an Apple M4 — not a production model.
A 325M-class model trains within a single 6 GB GPU only with BitBop; the STE and float baselines spill or do not complete (see §R2 of the report).
What we claim (and what we don't)
| Claim | Scope | |
|---|---|---|
| Quality | BitBop beats STE — the fair, same-family paired comparison (CI-backed, up to 91M) — and is competitive with an untuned same-recipe float at a matched training budget. On BabyLM a 125M ternary model reaches parity with a same-data GPT-2‑124M (float): 68.0 vs 67.9 BLiMP, same harness — but single-seed, and the 0.16-pt margin is inside the measured seed spread (0.11), so a tie, not a win. | budget-matched, ≤125M; not asymptotic |
| Memory | ~2.3 bytes/param of optimizer state (ternary body) vs 6 (Adam8bit) / 12 (AdamW). A 325M-class model trains in 6 GB — only BitBop completes a full step there (the 5× and the 6 GB fit are measured on the full model, float embed/head included). | training-state memory |
| Inference | int2-packed weights, 2 bpw stored (~1.3 bpw entropy at ~63% sparsity), ~16× smaller than fp32. Peak RAM/speed unchanged without a dedicated int2 kernel. | storage only |
| Speed | Not a claim. There is no ad-hoc kernel yet, so no wall-clock speedup. Ternary weights do remove the multiplies (adds/subtracts only) — the potential is there, the kernel is future work. | — |
We are not claiming state of the art, a speedup, or anything above 325M-class.
On the baselines. All three variants (bop / ste / float) share a single recipe — thresholds, schedule and batch were tuned for BitBop; lr
1e-3, no warmup, no gradient clipping — and the float and STE baselines are not separately tuned. The comparison isolates the training algorithm at fixed hyperparameters; it is not evidence that BitBop beats a well-tuned float, which would likely be stronger. The robust, statistically-backed result is BitBop > STE (same ternary family, same recipe).
Results (all same-harness, our validated BLiMP harness)
BLiMP macro-accuracy (67 phenomena × 1000 minimal pairs):
| Model | Params | Training data | BLiMP |
|---|---|---|---|
| BitBop (ternary) | 125M | BabyLM strict-small (~10M words) | 68.0 |
| BitBop (ternary) | 50M | BabyLM strict-small | 66.8 |
| BitBop (ternary) | 25M | BabyLM strict-small | 66.0 |
| STE ternary (baseline) | 25M | BabyLM strict-small | 63.2 |
| GPT-2‑124M (float) | 124M | BabyLM 10M (same data) | 67.9 |
| BabyLlama‑58M (distilled) | 58M | BabyLM 10M | 68.9 |
| GPT-2‑124M (float, data-rich) | 124M | WebText (≫ data) | 82.5 |
The data-rich GPT-2 (82.5) is the data ceiling we do not reach. The point is that a ternary model at the same data budget reaches parity within noise with a float GPT-2 of the same size (see the tie caveat above).
On TinyStories (internal, budget-matched), BitBop-91M reaches 1.49 validation nats vs STE 1.58 and float 1.59 — sample efficiency at a matched, under-trained budget (~1 token/param, far below compute-optimal), not an asymptotic result. The STE gap is the fair one (same recipe, same family); the float number uses BitBop's untuned recipe and should not be read as beating float. See the technical report.
What's new — BitBop II (July 2026)
The follow-up report (BitBop II, PDF) closes the three questions the first report left open — more than three weight states, a controlled canary memorization study, and parameters-vs-bits at matched memory.
Everything below is measured at
≤25M params, two small corpora (TinyStories / BabyLM strict-small), seeds {0,1}, with
pre-registered designs and the falsified predictions reported as such.
| Finding | One line | Scope |
|---|---|---|
| N-level frontier | Generalising the flip rule to 5/9/15 weight levels traces a concave quality-bits frontier that reproduces across three scales — but it was measured under a fixed momentum rate, a regime that specifically handicaps ternary via late-training weight churn. | toy → 25M |
| Horizon-free control (T4) | A validation-plateau ratchet on the momentum rate replaces the hand-tuned schedule; it rescues ternary (−0.185 nats at 25M, causal, 2 seeds) and barely moves 9-level. Its value is timing: early flips preserved, late churn suppressed 18×. A constant-high momentum control fails (early flips starved 12.8×). | 25M |
| Levels lose their case | At matched storage, under the controller, the big ternary model beats the 15-level one by 0.12 nats (2 seeds) — reversing our own earlier fixed-momentum finding. What levels were buying was churn robustness, which the controller provides for free. | toy, matched bytes (≈3× FLOPs caveat) |
| The float baseline overfits small data; ternary does not | At 17 epochs on BabyLM-small the same-recipe float overfits (min 2.94 → 3.00); every ternary variant is monotone and ends below the float's best checkpoint. A true-STE control attributes this to reduced fit capacity, not the optimiser. Regime-bound: with more data this could reverse. | 25M, 2 seeds |
| Latent-free ≥ STE, again | The flip optimiser matches STE+AdamW (nominally 0.02–0.03 nats better, within the measured dtype bound) at ≈5× less per-weight training state. BLiMP vs float: parity (~.66 under one harness). | 25M, 2 seeds |
| Controlled canary memorization | With Carlini-style canaries, ternary flip training exposes far less than float (5.3–6.7 vs 13.0–13.3 bits at 64×); the low exposure survives the controller (≈0% of the gap at 16×, 21–31% at 64×), and a designed worst-case falsifies "high momentum crystallises": exposure tracks fit, not the momentum rate. | toy, 2 seeds |
One practitioner's warning worth repeating: STE breaks silently under autocast — the
straight-through pattern w+(q-w).detach() stops training the quantised path in bf16 autocast and
degenerates to float training. Our first STE control was invalid for exactly this reason; the
diagnosis and the fp32 rule are in the report. The latent-free optimiser is immune (no latent to bypass).
Quickstart (~15 min on CPU, a few minutes on GPU)
Reproduce the headline BabyLM comparison from published weights:
pip install -r requirements.txt bash scripts/prepare_data.sh # downloads BLiMP bash scripts/reproduce_babylm.sh # downloads BitBop-125M + baselines, runs BLiMP
This downloads Feyd89/BitBop-125M-babylm from the Hugging Face Hub, evaluates it and the
public baselines (GPT-2, BabyLlama) with the same harness, and prints the table above.
How it works
In five lines:
- Weights are ternary, stored 2-bit packed (4 trits/byte), unpacked just-in-time for the matmul.
- The only per-weight state is
m, an EMA of the negative gradient (bf16). - A trit flips when
mcrosses a threshold set from the per-row RMS ofm— easy to fall into 0 (tau_in), harder to leave it (tau_out), which keeps the weights sparse. beta(the EMA rate) follows a one-decade decay schedule;mresets on flipped entries.- No latent float weights are ever stored. Details and the full recipe in the technical report.
Train it yourself
# full reproduction of the 25M BabyLM point from scratch (~3h on a 6 GB GPU) bash scripts/train_babylm_25m.sh # or drive the trainer directly python train.py --config configs/91m_bop.json --device cuda
Configs for every reported run are in configs/ (bop / ste / float, 91M TinyStories
and the 25M/50M/125M BabyLM ladder).
Weights
On the Hugging Face Hub (MIT):
Feyd89/BitBop-91M— TinyStories, the crossover headlineFeyd89/BitBop-125M-babylm— BabyLM, BLiMP 68.0
STE and float baselines are reproducible from configs/ (weights not shipped).
Layout
bitbop/kernel.py int2 packing + BopTernaryLinear/Int2Lin + ScheduledBop (the flip optimizer)
bitbop/model.py LMComposite (SWA body + full-attention layers, NoPE, tied interface)
bitbop/data.py BPE tokenizer + packed .bin loader
train.py one driver for bop | ste | float
eval_blimp.py BLiMP harness (+ HF branch for external baselines)
configs/ one JSON per reported run
scripts/ data prep + reproduction
docs/BitBop-technical-report.pdf reference technical report (PDF)
Citing / data
BitBop is MIT-licensed. It builds on public datasets and benchmarks — please cite their authors:
TinyStories, the BabyLM Challenge (2024), and BLiMP. BabyLM data is not redistributed here;
scripts/prepare_data.sh points you to the source.
Limitations
Budget-scoped quality claims only; ≤325M-class; single corpus per experiment; one author with an AI-assisted workflow. No speed claim without a dedicated kernel. See the reports for the full "tried and didn't work" table and the ongoing directions; BitBop II adds its own limitations section (data-limited 17-epoch regime, per-budget crossovers, matched bytes ≠ matched FLOPs).