GitHub - synchopate/llama-modchip: 🐋 Run 284B DeepSeek-V4-Flash (<59GiB RAM, 44.4 tok/s pp, 9.1-14.7 tok/s) & 30B MoEs (25-111 tok/s) on a pure 64GB CPU at 256K ctx (0 GPUs). SwiGLU-H2, Stockfish-PVS L3 & 13ms Claude Code restore.

Overview: Breaking the Sub-4-Bit CPU Quantization Barrier

Until today, running frontier Mixture-of-Experts (MoE) models on consumer or workstation CPUs forced an agonizing trade-off: either pay the heavy DRAM bandwidth penalty of 4.50-bpw scalar quantization (Q4_0 / Q4_K_M), or drop to standard 2–3 bit scalar quants (Q2_K / Q3_K) that suffer from slow multi-shift bit unpacking and degrade JSON tool-calling schemas.

llama-modchip v2.5.0 eliminates this compromise. By introducing LUT-VQ3-64 (GGML_TYPE_VQ3_0 = 43, 24 bytes / 64 weights = 3.00 bpw)—a 3D Lloyd-Max Vector Quantizer decoded directly inside 6 AVX-512 ZMM registers via single-cycle VPERMI2B lookups—and pairing it with Fractal-64 (GGML_TYPE_Q2_0 = 40, 30 bytes / 64 weights = 3.75 bpw) in a domain-aware hybrid topology:

1. 100/100 E2E Quality Parity with BF16 & Q4_0 at -33.3% MoE Footprint: On real uncompressed BF16 MoE weights (blk.1.ffn_up_exps.weight), Fractal-64 (3.75 bpw) achieves 17.75 dB SQNR (99.15% Cosine Similarity, beating Q3_K's 16.30 dB by +1.45 dB), while LUT-VQ3-64 (3.00 bpw, 2.333 index bits/weight) achieves 13.57 dB SQNR (97.78% Cosine Similarity, beating Q2_K's 10.15 dB by +3.42 dB). Pairing LUT-VQ3-64 (3.00 bpw) on expansion/gate projections with Fractal-64 (3.75 bpw) on residual output projections preserves 100% E2E quality (100.0 / 100.0) while cutting up_exps DRAM traffic by 33.3% vs. Q4_0.
2. 284B DeepSeek-V4-Flash in < 59 GiB RAM (+16.9% Faster Pure Decode): DeepSeek-V4-Flash-0731-VQ3 (284.33B total, 13.17B active, all 6 routed experts per layer) shrinks to 56.07 GiB on disk (58.31 GiB VmRSS), boosting pure N=1 decode to 9.05 tok/s (+16.9% over Fractal-64's 7.74 tok/s), prefill to 41.11 tok/s (+8.8%), and agentic tool bursts to 19.4–25.3 tok/s.
3. 30B Nemotron-3.5-A3B in 16.04 GiB RAM (100/100 Quality Gate): Nemotron-3.5-30B-A3B-VQ3-64 (13.08 GiB on disk) saves -5.67 GiB RAM vs. Q4_0 and -1.23 GiB vs. Fractal-64, runs at 24.18 tok/s E2E (+23.7% vs. Q4_0), lifts Stockfish-PVS speculative acceptance to 70.19% (+11.1% vs. Fractal-64), and scores a perfect 100.0 / 100.0 on benchmarks/eval_quality_and_tools.py.
4. Instant 10.4 ms Claude Code Slot-0 Restore + HARP-Cache + RouteCrack + Stockfish-PVS: Pre-warms and restores the full 40-tool Claude Code prefix (2,776 tokens) from /dev/shm in 10.4 ms (0.16s E2E TTFT, f_keep = 1.000) on both 30B and 284B models.

📊 Hard Empirical Evidence: Precision & Speed vs. BF16 and Q4_0

1. Ground-Truth Signal Fidelity on Uncompressed BF16 MoE Weights (libggml-base.so)

Measured directly via quantize_row_*_ref $\to$ dequantize_row_* on the uncompressed 59.0 GiB BF16 master checkpoint (blk.1.ffn_up_exps.weight, $N = 4,194,304$ weights, heavy-tailed kurtosis $\kappa = 3.2–4.8$):

2. End-to-End Model Benchmarks (benchmarks/eval_quality_and_tools.py + llama-bench)

Measured on 16-Core AMD Zen 5 (CT 100, --cpu-range 0-15 --cpu-strict 1, Transparent HugePages enabled; full report in benchmarks/QUALITY_GATE_VQ3_64.md).

A. NVIDIA-Nemotron-3.5-Lightning-30B-A3B

B. DeepSeek-V4-Flash-0731 284B (284.33B Total / 13.17B Active, Top-6 Routing)

🧠 How Domain-Aware LUT-VQ3-64 + Fractal-64 Hybrid Retains 100% E2E Quality

1. Why 3D Sphere Packing (LUT-VQ3-64) Beats Scalar 2–3 Bit Grids (+3.42 dB vs. Q2_K)

Scalar uniform quantizers quantize each coordinate $w_i \in [-M, +M]$ independently across the 3D Cartesian bounding cube $[-M, +M]^3$ of volume $8M^3$. However, heavy-tailed MoE weights (kurtosis $\kappa = 3.2–4.8$) have an isotropic radial/ellipsoidal joint density $p(\mathbf{w}) \propto \exp(-|\mathbf{w}|_p^\alpha / \sigma^\alpha)$ concentrated inside the inscribed $L_2$ sphere ($\text{Vol} = \frac{4}{3}\pi M^3 \approx 4.189 M^3$, leaving $47.64%$ of the cube empty) or $L_1$ cross-polytope ($\text{Vol} = \frac{4}{3}M^3 \approx 1.333 M^3$, leaving $83.33%$ of the cube empty).

LUT-VQ3-64 discards the Cartesian grid and packs 128 3D Lloyd-Max centroids ($7\text{ bits} / 3\text{ weights} = 2.333\text{ bpw}$, antipodally symmetric $\mathbf{c}_{127-k} = -\mathbf{c}_k$) directly onto the populated spherical/cross-polytope shell, paired with:

• 2 bytes: FP16 super-block scale $d$.
• 2 bytes: Four 4-bit unsigned sub-scales ($s_b = 4 + u_b \in {4..19}$) for the four 16-weight sub-blocks.
• 20 bytes: 21 7-bit 3D indices (147 bits) + 1 tail 4-bit scalar index (4 bits) = 151 bits inside 160 bits (24 bytes / 64 weights = 3.00 bpw).

2. Single-Cycle AVX-512 VBMI VPERMI2B Register Lookup + Domain-Aware Split

The entire 128-entry 3D codebook occupies $128 \times 3 = 384\text{ bytes}$, fitting permanently inside 6 AVX-512 ZMM registers (zmm26..zmm31). The AVX-512 VBMI VPERMI2B (_mm512_permutex2var_epi8) instruction resolves 64 centroids per instruction in 1 clock cycle directly in registers with zero L1/L2 cache reads. By encoding expansion/gate projections (ffn_up_exps, ffn_gate_shexp, attn_q_b, attn_output_a) in LUT-VQ3-64 (3.00 bpw, 97.78% CosSim) and preserving residual-stream output projections (ffn_down_exps, attn_output_b, attn_kv) in Fractal-64 (3.75 bpw, 17.75 dB SQNR, 99.15% CosSim), the model achieves 100% E2E quality parity (100/100) with BF16 and Q4_0 while cutting memory traffic by 20%–33%.

Quantization & Memory Breakdown

To answer the most common technical question upfront—exact bit-widths (bpw), base checkpoints, and what is compressed in each GGUF:

Note on MoE Expert Routing (Top-K): By default (Lossless), DeepSeek-V4-Flash evaluates all 6 routed experts + 1 shared expert (Top-6) per layer with zero expert dropping (1.00000 logit cosine similarity vs the unpatched runtime on the same checkpoint). Setting LLAMA_MOE_TOP_K=5 or 4 is an optional runtime flag for users who want to trade a small perplexity delta for higher decode throughput.

Quickstart

Linux & macOS (Automated Setup & Chat)

git clone https://github.com/synchopate/llama-modchip.git
cd llama-modchip
./modchip chat

Direct Run (Pre-Compiled Linux x86_64 Binary + Hugging Face .gguf)

Compatibility Note (GGML_TYPE_Q2_0 = 40 & GGML_TYPE_VQ3_0 = 43): Fractal-64 and LUT-VQ3-64 Surgical Hybrid models use custom block-64 BF16-scale tensors (GGML_TYPE_Q2_0 = 40, GGML_TYPE_VQ3_0 = 43) and deepseek4 / Stockfish-PVS kernels. Unpatched llama.cpp, Ollama, or LM Studio binaries will reject them with unknown tensor type 40/43. Use ./modchip above or the standalone pre-compiled binary (llama-server-modchip-linux-x86_64, zero external .so dependencies) included in the Hugging Face repositories and GitHub Release v2.5.0 (prebuilt static Windows .exe binaries are also available at v2.4.0-windows-cpu):

# 1A. 30B Nemotron-3.5-A3B Surgical Hybrid VQ3-64 + Fractal-64 (13.30 GiB — 16 GB RAM at 24.2–60.4 tok/s):
hf download Synchopate/NVIDIA-Nemotron-3.5-30B-A3B-VQ3-64-CPU-GGUF \
  llama-server-modchip-linux-x86_64 NVIDIA-Nemotron-3.5-30B-A3B-VQ3-64.gguf --local-dir .
chmod +x llama-server-modchip-linux-x86_64
./llama-server-modchip-linux-x86_64 -m NVIDIA-Nemotron-3.5-30B-A3B-VQ3-64.gguf -c 262144 -t 16 --port 8080

# 1B. 30B Nemotron-3.5-A3B Pure Fractal-64 (14.35 GiB — 16–32 GB RAM at 25.0–60.4 tok/s):
hf download Synchopate/NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64-CPU-GGUF \
  llama-server-modchip-linux-x86_64 NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64.gguf --local-dir .

# 2A. 284B DeepSeek-V4-Flash Surgical Hybrid VQ3-64 + Fractal-64 (56.45 GiB — fits in 64 GB RAM):
hf download Synchopate/DeepSeek-V4-Flash-0731-VQ3-64GB-CPU-GGUF --local-dir ./dsv4-vq3
chmod +x ./dsv4-vq3/llama-server-modchip-linux-x86_64
./dsv4-vq3/llama-server-modchip-linux-x86_64 -m ./dsv4-vq3/DeepSeek-V4-Flash-0731-VQ3-00001-of-00003.gguf -c 32768 -t 16 -fa on --port 8080

# 2B. 284B DeepSeek-V4-Flash Pure Fractal-64 (56.56 GiB — fits in 64 GB RAM):
hf download Synchopate/DeepSeek-V4-Flash-0731-Fractal-64GB-CPU-GGUF --local-dir ./dsv4

Windows 10/11 (Hybrid GPU + CPU MoE Pack)

Open windows/ and run windows/AVVIA_MODCHIP.bat (or ./modchip start in Git Bash). It detects physical CPU cores, free RAM, and GPU VRAM (nvidia-smi), downloads llama.cpp CUDA 12.4 / Vulkan binaries, computes the --n-cpu-moe split, and launches the Web Chat UI (http://127.0.0.1:8080).

Hardware Readiness Check (Zero Model Download Required)

curl -sL https://raw.githubusercontent.com/synchopate/llama-modchip/main/scripts/check_cpu_readiness.py | python3

Submit your terminal scorecard via a GitHub Benchmark Issue or GitHub Discussions to add your CPU to the leaderboard below.

Supported Models & Model-Specific Optimizations

Each model family has a dedicated technical guide with mathematical derivations, operator-level latency profiles, and quality gate benchmarks:

Supported Chips & Processor-Specific Optimizations

Microarchitectural details and tuning flags for each processor and GPU hybrid are documented in docs/HARDWARE_OPTIMIZATIONS.md:

Headline Benchmarks: Pure CPU & Hybrid Summary

1. DeepSeek-V4-Flash 284B (13.17B Active) on a 64 GB Zen 5 CPU (AMD Ryzen 9 9955HX, 0 GPUs)

Operator-level profiling and mathematical derivations in docs/DEEPSEEK_V4_OPTIMIZATIONS.md (probabilities and KL-divergence measured relative to the unpatched Top-6 execution on the 56.56 GiB Fractal checkpoint):

2. NVIDIA-Nemotron-3.5-30B-A3B (3.0B Active) at 256K Context (AMD Ryzen 9 9955HX, 0 GPUs)

Full ablation tables and 100/100 Quality Gate report in docs/NEMOTRON_OPTIMIZATIONS.md:

3. DeepSeek-V4-Flash 284B (13.17B Active) on Apple Silicon (Apple M5 Max, 64 GB Unified Memory)

• Prefill Throughput (pp512): 169.4 ± 29.6 tok/s (Apple Metal accelerated)
• Decode Throughput (tg128): 27.0 ± 0.4 tok/s in llama-bench
• Real Served Generation (llama-server): 20.7 – 23.6 tok/s across realistic coding (code_edit: 20.7 tok/s, code_new: 23.3 tok/s, free_text: 23.6 tok/s) with 0 swapouts (iogpu.wired_limit_mb=57344)
• Architecture & Placement: Apple M5 Max (18-Core CPU + 40-Core Metal GPU), 64 GB LPDDR5X. 5 pure-IQ1_S MoE layers offloaded to CPU (blk.(10|12|14|16|18).ffn_.*_exps.=CPU), remainder on Metal.
• Driver & Patches: patches/apple-silicon/0001 (Fractal-64 Q2_0 NEON/Metal), 0002 (Metal tensor run mapping), 0003 (Metal register unpack). Built via apple-silicon/build-dsv4-engine.sh and launched with apple-silicon/run-dsv4.sh. Contributed by Bobby Grayson (@notactuallytreyanastasio). Full report: apple-silicon/results/dsv4-m5max.md.

Community CPU & Hybrid Leaderboard

Run ./modchip bench (or windows\BENCHMARK_PC.bat on Windows) and Submit Your Scorecard (GitHub Issue) (pure-CPU entries sorted by Refactor decode):

llama-modchip/
├── modchip                                    # Unified CLI (setup | start | chat | bench | eval | quantize | web | claude | stop)
├── install.sh                                 # Automated Installer (Linux)
├── README.md / ARCHITECTURE.md / FAQ.md       # Overview, Full Mathematical Paper & Engineering FAQ
├── RELEASE_NOTES.md                           # v2.4.0 Release Notes
├── docs/
│   ├── NEMOTRON_OPTIMIZATIONS.md              # Deep-dive on Nemotron-3.5-30B-A3B & Nemotron-H-8B optimizations
│   ├── DEEPSEEK_V4_OPTIMIZATIONS.md           # Deep-dive on DeepSeek-V4-Flash 284B profiling, SVD & L2-quadrature MoE
│   ├── HARDWARE_OPTIMIZATIONS.md              # Chip-by-chip optimizations (Zen 5, Threadripper, Vulkan, CUDA, Metal, ARM64)
│   └── WINDOWS_PORT.md                        # Native Windows 11 patched-engine build & Ryzen 7 9800X3D 1M-context report
├── patches/
│   ├── apple-silicon/                         # Apple Silicon engine patches (Fractal-64 Q2_0 NEON/Metal, host-ptr run mapping, register unpack)
│   ├── deepseek-v4-flash-fractal-256k.patch   # Unified v2.4.0 patch (DeepSeek-V4-Flash 284B + Nemotron-3.5 30B, Linux/macOS/Windows)
│   ├── dsv4-64gb-cpu.patch                    # DeepSeek-V4-Flash 284B CPU engine patch (llama.cpp d2f83055 base)
│   └── zen5-nemotron-256k-optimizations.patch # Standalone ggml/llama core patch for Nemotron-3.5
├── windows/                                   # Windows 10/11 Hybrid GPU+CPU MoE Pack (AVVIA_MODCHIP.bat, modchip_win.ps1)
├── apple-silicon/                             # Apple Silicon M4/M5 setup scripts (build-dsv4-engine.sh, run-dsv4.sh, M5 Max benchmarks & docs)
├── fedora/                                    # Fedora 44 / AMD Ryzen 7 8745HS (GEEKOM A8) all-in-one script
├── speculative/                               # RouteCrack + Stockfish-PVS v2 (rc_stockfish.h, ngram-map.cpp)
├── benchmarks/                                # eval_quality_and_tools.py (100-pt Quality Gate) & results/*.json
├── gateway/sse_gateway.py                     # Agentic SSE Proxy + Compact Tool Table + Dark-Mode Web Chat HUD (:8787)
└── scripts/                                   # build_fractal_moe_gguf.sh, transcode_dsv4.{c,py}, chat.py, llama_modchip.py