GitHub - andreaborio/ds4: DeepSeek 4 Flash local inference engine for Metal and CUDA

DwarfStar

Specialized local inference for models that do not fit in memory.

A transparent research and co-development fork of antirez/ds4, focused on Metal, adaptive SSD streaming, common 16–64 GB Apple Silicon systems, and measured experimentation.

Quick start · Benchmarks · Models · DSBox · Upstream diff · Documentation

DwarfStar is a small, self-contained inference engine optimized around a narrow set of very large models. It includes native model loading, prompt rendering, tool calling, RAM/on-disk KV state, an HTTP server, a coding agent, GGUF tooling, and correctness and speed tests. It is intentionally not a generic GGUF runner; arbitrary GGUF files are not expected to work.

Why this fork exists

Upstream DwarfStar provides the core engine and leads the high-memory and distributed paths. This fork asks a complementary question:

How far can the same specialized design be pushed on the 16–64 GB Macs many developers already own, when the SSD becomes an active model-memory tier?

The current work concentrates on:

• adaptive Metal residency and routed-expert cache policies across memory tiers;
• SSD streaming that accounts for page cache, wired memory, swap, I/O, and throughput together;
• safer model-backed experiments near macOS memory limits;
• measured GLM 5.2 work and Qwen3.6-35B-A3B bring-up;
• GGUF calibration, incremental quantization, and expert-analysis tooling;
• keeping useful changes small enough to validate and send upstream.

This is primarily a learning and systems-research project. The fork lets work continue while upstream changes are under review; it is not a parallel rewrite or a competing inference ecosystem.

Co-development with upstream

The boundary is explicit and reviewable:

This is mandatory, not aspirational: every fork change applicable to an upstream-supported path will be opened upstream once its scope, correctness, and performance evidence are ready. Fork development can continue while that review is in progress.

Current upstream work includes #434 (quality-score build fix), #520 (GLM streamed-prefill correctness), and #528 (GLM indexed-prefill prepare). The DeepSeek regression found on the GLM line is tracked in #532. See FORK_NOTES.md for the status of each fork change and MERGE_LOG.md for sync history. The same policy is part of CONTRIBUTING.md. The current main delta is always inspectable in GitHub's upstream/fork comparison.

Quick start

Requirements: Apple Silicon, Xcode Command Line Tools, and enough SSD space for the selected model. A 64 GB Mac is the practical reference tier for DeepSeek Flash streaming; the 16 GB path is an experimental low-memory tier, not a speed guarantee.

xcode-select --install
git clone https://github.com/andreaborio/ds4.git
cd ds4

./download_model.sh q2-imatrix
make
./ds4 --build-info
./ds4 -m ./ds4flash.gguf --nothink

On macOS, AUTO residency keeps the model resident when it safely fits. Otherwise it selects SSD streaming and derives an expert-cache budget from the model geometry and live host memory. Force the SSD path only when you need a controlled run:

./ds4 -m ./ds4flash.gguf --ssd-streaming --ctx 32768 --nothink

Start the local API with:

./ds4-server -m ./ds4flash.gguf --ctx 32768

How SSD streaming uses memory

flowchart LR
    GGUF["Model GGUF on SSD"] --> AUTO["AUTO memory planner"]
    AUTO -->|"safe fit"| RES["Resident model"]
    AUTO -->|"model exceeds budget"| STREAM["SSD-streamed model"]
    STREAM --> FIXED["Mapped fixed / non-routed state"]
    STREAM --> CACHE["Adaptive routed-expert cache"]
    GGUF -->|"cache miss"| CACHE
    FIXED --> METAL["Metal graph"]
    CACHE --> METAL
    METAL --> TOKEN["Next token"]

The fixed model state, KV cache, graph scratch, and macOS file-backed cache all need headroom. The routed-expert cache is the variable tier; making it larger can help only until it starts displacing the pages and allocations the rest of the runtime needs.

Model status

DeepSeek expert-major v2 format

DS4 includes the experimental, self-describing ds4.expert_major.v2 layout for DeepSeek V4. It stores each layer as adjacent gate/up/down expert records without requantizing and without keeping a second routed-weight copy. Canonical GGUFs remain byte-for-byte compatible with every existing backend; native v2 files currently require a complete local Apple Metal model and fail early elsewhere.

Conversion, full byte-level verification, compatibility limits, and the model-backed promotion gate are in docs/deepseek-expert-major-v2.md. Until a dated canonical/native benchmark gate is complete, the canonical DeepSeek GGUF remains the release reference. The first M5 Pro SSD tranche is recorded in docs/benchmarks/2026-07-17-deepseek-native-expert-major.md. The distinctly named experimental artifact is DeepSeek-V4-Flash-DS4-ExpertMajor-v2-GGUF, with full conversion provenance and compatibility limits in its model card.

Qwen3.6 Metal AUTO path

The main branch is qualified and measured with one normalized text-only artifact. The recommended release download is the single-layout Qwen3.6-35B-A3B-DS4-ExpertMajor-v1-Q4_K_S.gguf from andreaborio/Qwen3.6-35B-A3B-DS4-ExpertMajor-v1-GGUF. It stores routed weights once in DS4's expert-major order and activates automatically; no sidecar variables are needed. The canonical Qwen3.6-35B-A3B-ds4-Q4_K_S.gguf remains supported during migration. The release artifact is 20,808,970,240 bytes (only 406,816 bytes larger than the canonical input) with SHA-256 fb2b344d49f0c3dfd854cfc11d92ffc873cc93a1d30bf4664e5aea6f1bfef839.

This is not generic Qwen or arbitrary community-GGUF support. The literal environment guard is the experimental opt-in; Metal, power 100, and AUTO residency are the Apple defaults, but are shown below for reproducibility:

DS4_QWEN_EXPERIMENTAL_METAL=1 ./ds4 \
  -m /absolute/path/to/Qwen3.6-35B-A3B-DS4-ExpertMajor-v1-Q4_K_S.gguf \
  --metal --power 100 --ctx 8192 --nothink

ds4.expert_major.v1 is an explicit DS4 GGUF extension. Other loaders must reject this artifact unless they implement the layout; use the canonical file for llama.cpp, MLX, or other runtimes. Format details, conversion commands, compatibility boundaries, and measured parity are in docs/qwen-expert-major-store.md.

Qwen AUTO selects the full-model mapped Metal mode only when both the fixed Metal working-set budget and a point-in-time host-memory pressure check pass. Under pressure it falls back to SSD and lazily grows the routed-expert cache to the largest complete routing tier admitted by the current conservative snapshot. Above 16 GiB the planner independently reserves the 2.50 GiB static page set, context/runtime memory, and system headroom. On a 16 GiB Mac, AUTO keeps the complete static charge but lets those unpinned, pageable GGUF pages share system headroom. It selects the largest complete 320-expert cache cycle admitted by the remaining live and platform budgets rather than imposing a fixed low-RAM floor. Bounded file-backed inactive pages receive full credit only while macOS reports normal pressure; unknown or elevated pressure retains half-credit and fails closed near the boundary. --resident fails unless both admission checks pass; because pressure can change after the snapshot, this is a conservative admission policy rather than a future-memory guarantee. --ssd-streaming remains the reproducible forced-streaming override. In SSD mode Qwen grows its Metal expert cache in 321-expert slabs (about 0.529 GiB) instead of taking the generic 4 GiB first slab.

Here resident means that DS4 maps the complete tensor payload, disables its explicit SSD expert cache, and executes full-tensor Metal kernels. Metal's residency request is a budgeting hint: it neither pre-faults every GGUF page nor proves that every page remains physically resident as later pressure changes. That stronger physical-residency claim requires separate runtime measurement. All neural math in the supported Qwen path is on Metal. The CPU still performs tokenization, sampling, route readback, cache bookkeeping, and streamed GGUF I/O; a CPU+GPU split of layers or experts is not implemented in this path.

The hard SSD cache floor is 321 complete routed experts (about 0.53 GiB); 640 (about 1.06 GiB) is a useful controlled small-cache tier. Startup and the per-layer path fail closed if the effective locked cache falls below the floor. The runtime has completed model-backed resident and SSD generation on an M5 Pro with 64 GiB, plus bounded SSD generation on a physical M1 Pro with 16 GiB. On production main bd62a0b, AUTO started with 321 cached experts for prefill and grew toward 2,241 for decode. One cold request completed at 10.56/8.24 prefill/generation t/s; four subsequent distinct short prompts had medians of 15.04/9.77 t/s, with normal memory pressure and no new swapouts. This recheck used the canonical migration GGUF; the native ExpertMajor v1 artifact was not copied to the 16 GiB host because only 3.6 GB of disk space was free. The older 4.06/7.03 result remains a conservative 321-expert compatibility floor, not the current production speed. See tests/qwen/README.md for the exact artifact contract, reproducible evidence, and current limitations.

Metal on Apple Silicon is the current proving ground for fork-specific optimization. The inherited CUDA/DGX Spark and ROCm/Strix Halo DeepSeek paths remain supported targets, but a Metal result is not advertised as a Blackwell or Strix Halo result until it is re-measured on that backend.

Measured results

Best retained local results so far. These rows are not cross-model rankings: each model uses a different artifact, context, and runtime path.

DeepSeek hardware reference bests from the standard speed-bench sweep:

Full commands, samples, and caveats are in docs/benchmarks/2026-07-15-qwen-ds4-vs-llamacpp.md, docs/benchmarks/2026-07-14-m5-pro.md, SSD_STREAMING_VERIFICATION.md, and docs/ENGINE_REFERENCE.md.

Memory safety is part of performance

More expert-cache RAM is not automatically faster. On memory-constrained Macs, an oversized cache can evict the file-backed pages SSD streaming needs and make decode slower even when Activity Monitor appears to show free memory. AUTO therefore treats the routed-expert cache as variable and preserves headroom for fixed weights, KV, scratch, Metal allocations, and the macOS page cache.

During development, a model-backed test bypassed SSD streaming and attempted to make an 80.76 GiB GGUF resident with a 100,000-token context on a 64 GiB Mac. Global wired memory reached roughly 61.36 GiB before a watchdog kernel panic. Crashing the host is not an acceptable test outcome.

Current main includes hardware-aware AUTO residency, fail-closed cache admission, bounded benchmark guards, and GPU cleanup before model mappings are released (1523b26). A stricter guard that rejects resident mappings larger than 90% of physical RAM is tested and published on fix/refuse-oversized-resident-maps at 06fd005, but is not yet on main. Until it is merged, it must not be described as a mainline guarantee.

Prefer a desktop interface?

DSBox is the companion desktop interface, inspired by Unsloth Studio: discover compatible models, manage ds4, chat locally, connect coding agents, and observe memory, swap, disk, and token throughput without hand-assembling every command. DSBox is a separate project and still a work in progress.

_{DSBox is an optional companion UI, maintained in a separate repository.}

Documentation

• docs/ENGINE_REFERENCE.md: complete model, runtime, server, agent, KV-cache, distributed, backend, and debugging guide.
• tests/qwen/README.md: experimental Qwen artifact contract, oracle procedure, Metal + SSD commands, measurements, and limits.
• docs/qwen-expert-major-store.md: DS4-native GGUF layout, transactional converter, compatibility, and parity evidence.
• docs/expert-major-v2-roadmap.md: generic expert-major manifest and separate DeepSeek/GLM qualification plan.
• CONTRIBUTING.md: upstream-first contribution policy and correctness/performance gates.
• FORK_NOTES.md: fork delta and upstreamability ledger.
• MERGE_LOG.md: upstream synchronization history.
• GOLD_METAL_SSD.md: Metal build identity, AUTO residency, and benchmark promotion gates.
• SSD_STREAMING_VERIFICATION.md: independent SSD-streaming verification campaign.
• ONEDGE_IMATRIX.md: live, privacy-preserving imatrix collection.
• STREAMING_MIXED_PRECISION.md: mixed-precision expert streaming design and validation.
• EXPERT_PRUNE.md: expert profiling and prune-mask research.
• gguf-tools/README.md: GGUF, imatrix, quantization, and quality tooling.

ds4-server -m model.gguf --imatrix-out edge.dat                  # collect from live traffic
ds4-server -m model.gguf --imatrix-out edge.dat --imatrix-every 128 --imatrix-min-requests 32

# 1. build the 2-bit base once
gguf-tools/deepseek4-quantize --hf FP --template base.gguf --imatrix coder.dat \
  --out coder-iq2.gguf

# 2. every boost variant reuses the base's unchanged layers, re-quantizing only the boosted ones
gguf-tools/deepseek4-quantize --hf FP --template base.gguf --imatrix coder.dat \
  --reuse coder-iq2.gguf \
  --tensor-type blk.30.ffn_gate_exps.weight=q4_k …  --out coder-q4boost.gguf

reuse: PRIOR.gguf shares the weights key (…) but not the imatrix — copying
       imatrix-independent tensors, regenerating the steered ones

# the mask lives in the CPU router, so enable it (streaming-IQ2 path), then prune:
DS4_METAL_ENABLE_STREAMING_IQ2_CPU_ROUTER=1 DS4_EXPERT_PRUNE_MASK=mask.txt \
  ds4 -m coder-iq2.gguf -p "…" --ssd-streaming
# -> "ds4: expert prune mask ACTIVE (N experts pruned) from mask.txt"

Full engine reference

The long-form guide now lives in docs/ENGINE_REFERENCE.md. It covers model downloads, full-resident and SSD-streamed operation, distributed inference, power controls, the native agent, benchmarking, capability evaluation, CLI, server/tool calling, disk KV cache, backends, steering, test vectors, and debugging.

Keeping the manual separate makes this README a reviewable landing page while preserving the full operational reference.

Status, credit, and license

DwarfStar is beta software and ds4-agent remains alpha. The core engine and upstream direction come from antirez/ds4. The project also exists thanks to the kernels, formats, and engineering work of llama.cpp and GGML.

Released under the MIT license. Contributions follow the upstream-first policy.