GitHub - wici-ai/inferssd: Run models larger than your RAM or VRAM by streaming MoE experts from SSD. Built on llama.cpp.

Large MoE models, powered by your SSD

Website  ·  Blog  ·  Slack  ·  Apache-2.0

macOS / Metal  ·  Linux / CUDA  ·  Windows (in progress)

Run models larger than your RAM or VRAM. InferSSD builds on llama.cpp to run large mixture-of-experts (MoE) models from SSD, using standard GGUF files, with browser chat and an OpenAI-compatible API. It currently supports DeepSeek-V4.1-Flash-1017, Muse-Spark-Open and Qwen3.6-35B-A3B, with more models coming.

These are the tested GGUF files, community conversions of the named model releases. DeepSeek's Q8_0 keeps the official bit widths: routed experts stay native 4-bit MXFP4, and FP8 tensors become 8-bit Q8_0. See weight-format details.

Performance benchmarks

4K prompt tier · 256 output tokens · decode measured over tokens 129–256. All rates are in token/s; higher is faster. The multiple next to each InferSSD rate is its speedup over llama.cpp on the same hardware. On the MacBook Pro, Colibri runs the official Safetensors on CPU; llama.cpp and InferSSD run the GGUF with Metal. See measurement details.

Test hardware

Unified memory is shared by the CPU and GPU. SSD capacities are nominal. See hardware details for SSD models, CPU core layouts, and configuration notes.

DeepSeek-V4.1-Flash-1017

Muse-Spark-Open

N/S: Muse is not currently supported by Colibri.

Get started

Install on Apple Silicon macOS or on Linux with an NVIDIA GPU:

curl -fsSL https://github.com/wici-ai/inferssd/releases/latest/download/install.sh | sh

InferSSD currently installs from source, so the engine is compiled for your machine's CPU and GPU; prebuilt packages are not available yet. Building takes about 2 to 12 minutes, depending on the machine.

The installer works in a private environment and uses no sudo. It needs Git, Python 3.9 or later, and a C/C++ compiler. macOS also needs the Xcode Command Line Tools; Linux also needs systemd with cgroup v2, and the CUDA toolkit on a machine with an NVIDIA GPU. Follow the printed PATH instruction if it shows one.

Then start a model:

# Chat in your terminal
inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

# Or start the web UI and an OpenAI-compatible API at http://127.0.0.1:8080
inferssd serve -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

For Muse, use unsloth/Muse-Spark-Open-816B-A42B-GGUF; for Qwen3.6, unsloth/Qwen3.6-35B-A3B-GGUF. With serve, open http://127.0.0.1:8080 in your browser, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1.

The first run downloads the model files (about 470 GiB for DeepSeek or Muse, 21 GiB for Qwen3.6) to ~/.local/share/inferssd/models and measures your machine. Use --model-dir /path/on/ssd to choose the drive, or -m /path/to/first-shard.gguf for files you already have. There is no memory setting to choose: InferSSD uses the memory available. Stop with Ctrl+C, or /exit in chat.

See the quick start for upgrades and uninstalling.

Further reading

• How it works
• Benchmark methodology

Contributing

Contributions are welcome, including hardware results and improvements to expert I/O, caching, and batching. See the contributing guide.

Acknowledgments and license

Built on llama.cpp and ggml, with thanks to their contributors and the model-adaptation community.

InferSSD is licensed under Apache-2.0; bundled llama.cpp/ggml and other third-party components keep their original licenses and notices; see NOTICE. Model weights remain subject to their publishers' licenses.