radiance

· Codeberg.org

22 min read Original article ↗

An inference server for AMD GPUs, in C++ and HIP: language models, and image, video, speech and music generation. Its HTTP API speaks OpenAI's chat, completions, Responses, images, audio and video APIs, Anthropic's Messages API and llama-server's native one, so their clients and SDKs work unchanged. Model architectures, kernel libraries and quantisers are .so plugins; the ones in this tree are built and loaded the same way a third party's are (docs/PLUGIN.md).

The bundled device kernels target AMD RDNA4 (libr4d: gfx1200, gfx1201), RDNA3 (libr3d: gfx11xx) and Strix Halo (libhalo: gfx1151). The serve settings below are for two 32 GB cards (Radeon AI PRO R9700). The engine starts on any AMD GPU family; another card needs a kernel library for it. A machine with no GPU builds the host backend, whose AVX-512/AVX2 kernels (libavx) serve every model here: the small ones (EmbeddingGemma 2, Qwen3-ASR, the 0.8B-3B models) at interactive speed on a desktop CPU, the 27B and larger far more slowly than a card.

To support the development of radiance: Patreon.

Models

model container get it fleet file
Qwen3.8-Flash-Next: IQR-V4 experts, int8 trunk, MTP, vision qwen3.8-next-flash-iqr-v4.rad (113 GiB) download tp1.yaml, tp2.yaml
Qwen3.8-27B IQ8R (rotated int8), DFlash2 drafter, vision qwen3.8-27b-iq8r-dflash.rad (29 GiB) download tp2.yaml
Qwen3.8-27B IQR-V4, 2.5 to 8 bits a weight (docs/IQR.md), DFlash2 drafter, vision qwen3.8-27b-iq2r-v4.rad .. qwen3.8-27b-iq8r-v4.rad (10 to 28 GiB) download tp1.yaml (IQ4R-L)
Qwen3.6-35B-A3B IQ8R, MTP qwen3.6-35b-a3b-iq8r.rad (35 GiB) download —
MiniCPM5-2B IQ8R, DSpark drafter minicpm5-2b-iq8r.rad (3 GiB) download tp1.yaml, tp2.yaml
MiniCPM5-2B bf16 (served without its drafter) minicpm5-2b-bf16.rad (5 GiB) download —
Spark X2.6 4B bf16: thinking and tool calls, one to four cards spark-x2.6-4b-bf16.rad (8.3 GiB) convert —
Gemma 4 31B: GPTQ int4 trunk, int8 activations, MTP, vision gemma4-31b-w4a8.rad (20 GiB) download —
Gemma 4 31B w5: GPTQ int5 trunk (a scale and a zero point per 128 weights), MTP, vision gemma4-31b-w5.rad (23 GiB) download —
Gemma 4 31B heretic-ara (refusals removed): GPTQ int4 trunk, int8 activations, MTP, vision gemma4-31b-heretic-ara-w4a8.rad (20 GiB) download —
Gemma 4 31B: int8 trunk and lm_head (no calibration), MTP, vision gemma4-31b-i8g.rad (33 GiB) download —
Gemma 4 31B heretic-ara (refusals removed): int8 trunk and lm_head (no calibration), MTP, vision gemma4-31b-heretic-ara-i8g.rad (33 GiB) download —
Gemma 4 31B Q8: int8 trunk and lm_head (Q8_0's grid at group 128), int8 DFlash drafter, vision gemma4-31b-dense-q8-df.rad (33 GiB) convert —
Gemma 4 12B bf16: MTP, vision, audio gemma4-12b-bf16.rad (25 GiB) download —
Gemma 4 12B int8 (i8a8): MTP, vision, audio gemma4-12b-i8g.rad (14 GiB) download —
Qwen-Image 2.1 IQ8R: text to image and image edits qwen-image-2.1-iq8r.rad (16 GiB) download tp1.yaml, tp2.yaml
Qwen-Image 2.1 Turbo IQ8R: the same in 8 steps qwen-image-2.1-turbo-iq8r.rad (16 GiB) download —
Krea 2 Turbo IQ8R: text to image, one card krea-2-turbo-iq8r.rad (16 GiB) download —
Krea 2 Raw IQ8R: the same, undistilled krea-2-raw-iq8r.rad (16 GiB) download —
FLUX.3 [klein] IQ8R: text to image and edits from up to ten pictures, one card flux-3-klein-iq8r.rad (9.4 GiB) download —
FLUX.3 [klein] base IQ8R flux-3-klein-base-iq8r.rad (9.3 GiB) download —
LTX-2.5 distilled IQ8R: video with its stereo soundtrack from text or a first frame, one card ltx-2.5-distilled-iq8r.rad (37 GiB) download —
LTX-2.5 base IQ8R ltx-2.5-iq8r.rad (37 GiB) download —
MiniMax-H3, int8 DiT: video with its stereo soundtrack from text minimax-h3-i8.rad (63 GiB) download tp2.yaml
FastH3 8-Step V2, int8 DiT: the same in eight steps fasth3-8-step-v2-i8.rad (65 GiB) download —
MiniMax-Music3 bf16: songs from lyrics and a description minimax-music3-bf16.rad (20 GiB) download tp1.yaml, tp2.yaml
Unlimited-OCR bf16: document parsing with the DeepEncoder tower unlimited-ocr-bf16.rad (6.2 GiB) download —
Breeze-TTS-2 bf16: speech, whole, streamed or live, one card breeze-tts2-bf16.rad (5.5 GiB) download tp1.yaml, tp2.yaml
TranslateGemma 27B, int8 trunk: translation translategemma-27b-i8.rad (28 GiB) download —
Qwen3.8-27B bf16 qwen3.8-27b-bf16.rad (50 GiB) convert —
Qwen3.8-Flash-Next bf16, a disk-streamed reference qwen3.8-flash-next-bf16.rad (335 GiB) convert —
Stable Diffusion XL bf16 and its finetunes (Illustrious, NoobAI, Pony): text to image, one card illustrious-xl-v2.0-bf16.rad (6.4 GiB), from any single-file SDXL checkpoint convert —
EmbeddingGemma 2 bf16: text, image, video and audio embeddings, one card embeddinggemma2-bf16.rad (1.4 GiB) convert tp1.yaml, tp2.yaml
nomic-embed-text-v2-moe bf16: text embeddings, one card nomic-embed-v2-moe-bf16.rad (959 MB) convert —
LiquidAI LFM3-1.7B IQ8R: rotated int8 trunk, bf16 head; text, tool calls and reasoning, one card lfm3-1.7b-iq8r.rad (2.5 GiB) convert with data/recipes/lfm3-1.7b-iq8r.recipe —
LiquidAI LFM3-1.7B bf16: text, tool calls and reasoning, one card lfm3-1.7b-bf16.rad (3.8 GiB) convert —
Meta Muse Glimmer 30B bf16: text, images and video; sliding-window rotary layers between NoPE full ones, a windowed ViT muse-glimmer-30b-bf16.rad (56 GiB) convert —
Meta Muse Glimmer 30B w4a8: int4 trunk, images and video, one card muse-glimmer-30b-w4a8.rad (16 GiB) convert —
Meta Muse Glimmer 30B IQ8R: rotated int8 trunk, int8 head, DFlash drafter, images and video muse-glimmer-30b-iq8r-dflash.rad (36 GiB) download —
Recursive Flint (39.8B, about 2.5B active) IQ8R: rotated int8 trunk and experts, two cards flint-iq8r.rad (38 GiB) download —
Ling-3.0-tiny IQ8R: rotated int8 trunk and experts, one card ling-3.0-tiny-iq8r.rad (7.9 GiB) download —
Nemotron 3.5 Embed 1B BF16: text and image embeddings, one card the checkpoint as it ships NVIDIA checkpoint —
Nemotron 3.5 Embed 8B BF16: text and image embeddings, one card the checkpoint as it ships NVIDIA checkpoint —
Nemotron 3.5 Rerank 8B BF16: text/image document relevance, one card the checkpoint as it ships NVIDIA checkpoint tp1.yaml, tp2.yaml
Qwen3-ASR 1.7B bf16: speech to text, /v1/audio/transcriptions and live over /v1/realtime, 30 languages, one card the checkpoint as it ships (4.1 GiB) Qwen/Qwen3-ASR-1.7B-hf tp1.yaml, tp2.yaml

Not served by this version. A container whose weights are block FP8, MXFP4, M2XFP4 or ParoQuant, or whose experts read E4M3 activations, is refused at startup by the name of the first such weight. That is every FP8 and MXFP4 container of the models above, and the models that have no int8 container yet: Ornith-1.5 (35B-A3B and 122B), Sage-A1.5, NVIDIA-Nemotron-3.5-Bolt-74B-A9B, GLM-5.3-Flash, Ling-3.0-flash-Sante, Fish Audio drama-3 and Anima's FP8 container (Anima converted to bf16 is served). Each returns with an int8 container. A container with an int8 or bf16 trunk and a block-FP8 drafter (NVIDIA-Nemotron-3.5-Bolt-E4B, the Gemma 4 31B DFlash and DFlash2 containers, MiniCPM5-2B bf16) serves without the drafter under num-speculative-tokens: 0.

A container holds everything the server needs: the weights, the tokeniser, the chat template and any drafter. Each download link's repository has the recipe and the exact command that made the file. Download one into your models directory (the transfer resumes if interrupted):

mkdir -p /srv/models/rad && cd /srv/models/rad
curl -LC - -O https://huggingface.co/StillDeadcode/minicpm5-2b-iq8r/resolve/main/minicpm5-2b-iq8r.rad

The Gemma 4 container is keptn42's: GPTQ calibrated in transformers, with an int8 lm_head; its repository has the recipe and the calibration step. To convert Google's bf16 checkpoint yourself, data/recipes/gemma4-31b-dense-bf16-w4a8.recipe quantises without calibration (round-to-nearest with a scale search, bf16 lm_head); see Convert a checkpoint. On two R9700s that container decodes at 103 tok/s on prose and 141 on code, against 112 and 150 for keptn42's, and accepts 0.46 of its drafts on prose against 0.50.

TranslateGemma 27B is converted from Google's bf16 checkpoint with data/recipes/translategemma-27b-i8a8.recipe, an int8 trunk rounded without calibration data. Its chat template takes structured turns naming the two languages and refuses the plain turn the server renders at startup, so its entry serves a plain Gemma 3 template, deploy/compose/gemma3.jinja (override-chat-template names it; in a compose file, mount it into the container), and a client sends on /v1/completions the prompt the checkpoint's template renders.

The Qwen-Image containers serve OpenAI's images API (/v1/images/generations, /v1/images/edits): pictures from a prompt, and edits of up to ten pictures with an optional mask, 2048x2048 by default. On two R9700s a 2048x2048 picture takes 61 s at the base model's 40 steps and 13 s with Turbo (56 and 12 s with tp-wire: wht6); 1024x1024 takes 6 s and 2.6 s. Qwen-Image 2.1 is under Qwen's research license, which is non-commercial. Krea 2 is under the Krea 2 Community License and Krea's acceptable use policy, FLUX.3 [klein] under the FLUX Kommunity License (non-commercial use), LTX-2.5 under the LTX-2.x Community License, and an SDXL finetune under its checkpoint's own license.

The MiniMax-H3 containers serve OpenAI's video jobs (/v1/videos): a video with its stereo soundtrack from a prompt, 5 to 14 s, up to 1344x768. On two R9700s a 5.17 s clip at 1344x768 takes about 68 s with FastH3 and 8.8 minutes at MiniMax-H3's 50 steps; both read about 39 GiB of pinned host memory. With LBH-123-AI's latent upscaler in the container, "latent_upscale": 2 runs the first steps at half the size and refines at the full size: MiniMax-H3's clip then takes 3.4 minutes (docs/GUIDE.md §5.16). A ComfyUI node, integrations/comfyui/ComfyUI-Radiance-MiniMax, renders through /v1/videos from a ComfyUI graph as a client of the server. MiniMax-H3's license, which FastH3 carries, does not cover the EU, the UK, the Republic of Korea or the USA.

MiniMax-Music3 and Breeze-TTS-2 answer /v1/audio/speech, whole or streamed as they are made: a song from lyrics and a description, 44.1 kHz stereo, made at about the speed it plays on two cards; and speech in a voice the request describes, made in 0.6 of its playing time on one card, its first streamed audio 0.21 s after the request. Breeze-TTS-2 also speaks text while it is being written, over the /v1/audio/speech/stream WebSocket; it is licensed for research and non-commercial use only. Fish Audio's drama-3 speaks a script whose speakers and direction are written in it (<|speaker:1|>, [laughs]), each speaker in the model's voice or one cloned from a reference clip and its transcript, at 0.3 of its playing time on two cards, its first streamed audio 0.23 s after the request; under the Fish Audio Community License a service that lets others generate speech with it needs a licence from Fish Audio, and imitating a real person's voice needs their consent.

Run with Docker

The host needs Docker and an amdgpu driver that supports ROCm 10.1; it does not need ROCm. The card works best with Resizable BAR on (Above 4G Decoding and Resizable BAR in the firmware setup); without it the engine keeps kernel arguments in pinned host memory, and every dispatch reads them over the link. The image is stilldeadcode/radiance, with its kernels built for gfx1201 (RDNA4), gfx11-generic (RDNA3) and gfx1151 (Strix Halo). Pull it, clone this repository for its compose files, then start one of them:

docker pull stilldeadcode/radiance
git clone https://codeberg.org/StillDeadcode/radiance.git
cp -r radiance/deploy/compose ~/radiance-compose && cd ~/radiance-compose
cp .env.example .env                         # set RADIANCE_MODELS, RADIANCE_API_KEY, the GIDs
docker compose -f tp2.yaml up -d             # two cards; tp1.yaml for one (RADIANCE_GPU)
docker compose -f tp2.yaml logs -f
docker compose -f tp2.yaml down

A file serves all its models from one container behind one port, and brings a model onto the cards when a request names it (docs/MULTIMODEL.md): a request's model is the entry's name (flashnext, qwen3.8-27b, minicpm5-2b, ...). The entries are in tp1.radiance.yaml and tp2.radiance.yaml beside the compose files, the server's configuration as it reads it. Delete the entries of the containers you do not have; any other model from the table is added as an entry with its keys from the table under Run without Docker. The two files take the same port, so run one at a time. Running without compose, and the reason for each docker run option, is in docs/DOCKER.md.

To build the image yourself, for example for other cards, run docker/build.sh -t 'gfx1200;gfx1201' in the clone (it needs docker buildx and tests the build as it goes). Then set RADIANCE_IMAGE=radiance:latest in .env.

Build from source

You need Linux, git, a C++20 compiler, CMake 3.21+ and Ninja. For the device kernels you also need ROCm 10.1 or 7.2 (hipcc; set ROCM_PATH if it is not /opt/rocm). Clone the repository and build in it:

git clone https://codeberg.org/StillDeadcode/radiance.git && cd radiance
./build.sh                                   # configure, build for the cards present, run the fast tests
./build.sh --no-hip                          # host backend only
cmake -S . -B build -G Ninja -DRAD_GPU_TARGETS='gfx1201'   # build for cards this machine lacks

To install, name the prefix when the build directory is first configured (-DCMAKE_INSTALL_PREFIX=/opt/radiance), build, then run cmake --install build. The default plugin path, <prefix>/share/radiance, is compiled in from that first configure, so cmake --install build --prefix DIR alone gives a tree that looks for its plugins elsewhere. The prefix gets bin/, include/radiance/, lib/cmake/radiance/, libexec/radiance/, share/radiance/ and share/doc/radiance/.

With ROCm 10.1 the build turns the compiler's VOPD pairing (dual-issue instruction pairs) off: its LLVM 24 pairs in an order that follows heap addresses, so two builds of the same source can differ. The Docker image builds its compiler with the fix and keeps the pairing, which is 1.4-3.5% faster on prefill GEMMs; a compiler that has the same fix is declared with -DRAD_HIP_VOPD_REPEATABLE=ON. ROCm 7.2 is unaffected.

Run without Docker

The server reads its settings from a YAML file: server: holds the listener, memory: the per-card VRAM budget, and models: one entry whose name is the model id requests use. Every key is named as in the docs/GUIDE.md §4.11 reference; radiance --help prints it.

# radiance.yaml
server:
  port: 8000
models:
  minicpm5-2b-iq8r:
    model: /srv/models/rad/minicpm5-2b-iq8r.rad   # or a Hugging Face repo id: StillDeadcode/minicpm5-2b-iq8r
export RADIANCE_HOME=$PWD/build/radiance_home   # the plugins; an install finds its own
build/bin/radiance --config radiance.yaml --check   # validate the file, serve nothing
build/bin/radiance --config radiance.yaml

model also takes a Hugging Face repo id, as vLLM does. The server fetches the repo with curl and jq (scripts/hfget.sh) into ~/.cache/radiance/hf (server.download-dir puts it elsewhere), serves the container it holds, and checks it for changes at every start; a start that cannot reach the hub serves the copy it has. The Docker image carries both tools; in a container, server.download-dir must name a writable mount, since the compose files mount /models read-only.

The server takes no other command-line options. A command line written for an earlier release (radiance --model ... --tp 2) is refused with the same settings printed as YAML, and rad-yaml FLAGS... prints that file without starting anything: rad-yaml --model x.rad --tp 2 > radiance.yaml. docs/MULTIMODEL.md §4 describes the file's levels.

Each row holds the keys a model is served with in a file of its own (the Flash-Next rows leave out prefix-cache-mib). They go under the model's entry, except gpu-headroom-mib, which goes under memory: and in a file of several models holds for all of them. tp1.radiance.yaml and tp2.radiance.yaml carry their models' entries as set for sharing the cards; a model added to either takes its row from here.

model keys
Qwen3.8-Flash-Next tp: 2, tp-wire: wht6, max-num-seqs: 8, max-model-len: 200000, placement: expert_tiered, host-pool-mib: 24576, gpu-headroom-mib: 96, kv-cache-dtype: i8, num-speculative-tokens: 3, max-num-batched-tokens: 2048
Qwen3.8-Flash-Next, one card tp: 1, max-num-seqs: 4, max-model-len: 200000, placement: expert_tiered, host-pool-mib: 43000, gpu-headroom-mib: 96, kv-cache-dtype: i8, num-speculative-tokens: 3, max-num-batched-tokens: 4096, checkpoint-interval: 4096
Qwen3.8-Flash-Next, four ranks tp: 4, max-num-seqs: 8, max-model-len: 200000, placement: expert_tiered, host-pool-mib: 49152, gpu-headroom-mib: 96, kv-cache-dtype: i8, num-speculative-tokens: 3, max-num-batched-tokens: 2048
Qwen3.8-Flash-Next, three ranks (the shipped container as it is, docs/TP3.md) tp: 3, max-num-seqs: 8, max-model-len: 200000, placement: expert_tiered, host-pool-mib: 24576, gpu-headroom-mib: 96, kv-cache-dtype: i8, num-speculative-tokens: 3, max-num-batched-tokens: 2048
Qwen3.8-27B IQ8R and IQR-V4, Qwen3.6-35B-A3B tp: 2, max-num-seqs: 32, max-model-len: 200000, kv-cache-dtype: i8
Qwen3.8-27B bf16 tp: 2, max-num-seqs: 8, max-model-len: 200000, kv-cache-dtype: i8
MiniCPM5-2B IQ8R and bf16 tp: 2, max-num-seqs: 32, max-model-len: 131072, kv-cache-dtype: i8, gpu-headroom-mib: 160, max-num-batched-tokens: 512
Qwen3.8-Flash-Next bf16 tp: 2, max-model-len: 32768, placement: expert_tiered, weights-disk-tier: true, host-pool-mib: 24576, gpu-headroom-mib: 768, deterministic: true, max-num-seqs: 32, max-num-batched-tokens: 32768, num-speculative-tokens: 0
Qwen-Image 2.1 and 2.1 Turbo tp: 2, tp-wire: wht6
MiniMax-H3 and FastH3 tp: 2, vision-tower-placement: ram
MiniMax-Music3 tp: 2, or tp: 1 (20.5 GiB of weights on the card)
Breeze-TTS-2 tp: 1
Gemma 4 31B (GPTQ int4) tp: 2, max-num-seqs: 8, max-model-len: 65536, kv-cache-dtype: i8, num-speculative-tokens: 3, max-num-batched-tokens: 2048, temp: 1.0, top-p: 0.95, top-k: 64
Gemma 4 31B w5, heretic-ara int4, int8 and heretic-ara int8 tp: 2, max-num-seqs: 8, max-model-len: 65536, kv-cache-dtype: i8, vision-tower-placement: ram, num-speculative-tokens: 3, max-num-batched-tokens: 2048, temp: 1.0, top-p: 0.95, top-k: 64
Gemma 4 31B Q8 tp: 2, max-num-seqs: 8, max-model-len: 65536, kv-cache-dtype: i8, num-speculative-tokens: 0, max-num-batched-tokens: 2048, temp: 1.0, top-p: 0.95, top-k: 64
Gemma 4 12B bf16 tp: 2, max-num-seqs: 8, max-model-len: 65536, kv-cache-dtype: bf16, num-speculative-tokens: 3, max-num-batched-tokens: 2048, mm-max-patches: 1120, temp: 1.0, top-p: 0.95, top-k: 64, chat-template-kwargs: {"enable_thinking":false}
Gemma 4 12B int8 tp: 1, max-num-seqs: 8, max-model-len: 65536, kv-cache-dtype: bf16, num-speculative-tokens: 3, max-num-batched-tokens: 2048, mm-max-patches: 1120, temp: 1.0, top-p: 0.95, top-k: 64, chat-template-kwargs: {"enable_thinking":false}
NVIDIA-Nemotron-3.5-Bolt-E4B bf16, int8 and 4-bit, without the drafter tp: 2, max-num-seqs: 8, max-model-len: 65536, kv-cache-dtype: bf16, num-speculative-tokens: 0, mm-max-patches: 2520, vision-tower-placement: ram, temp: 1.0, top-p: 0.95, top-k: 64, chat-template-kwargs: {"enable_thinking":false}; tp: 1 on one card
TranslateGemma 27B tp: 2, max-num-seqs: 8, max-model-len: 16384, kv-cache-dtype: i8, and override-chat-template naming deploy/compose/gemma3.jinja
Unlimited-OCR tp: 1, max-num-seqs: 8, max-model-len: 32768, kv-cache-dtype: i8, gpu-headroom-mib: 96, and override-chat-template naming deploy/chat-templates/unlimited-ocr-parser.jinja (/chat-templates/unlimited-ocr-parser.jinja in the compose files' container)
Krea 2 Turbo and Raw, FLUX.3 [klein] and its base, Stable Diffusion XL tp: 1
LTX-2.5 distilled and base tp: 1, vision-tower-placement: ram
EmbeddingGemma 2 tp: 1, max-num-seqs: 64, max-num-batched-tokens: 8192
Qwen3-ASR 1.7B tp: 1, max-num-seqs: 16

The 27B in IQ8R and IQR-V4, the 35B-A3B, MiniCPM5 and the Flash-Next bf16 reference run at up to 32 sequences a step. The other Flash-Next rows and the 27B in bf16 keep 8, the one-card Flash-Next row 4: on a hybrid model each sequence holds its own recurrent state, and 32 of them take that memory from the attention cache and, on Flash-Next, from the expert slab. See docs/GUIDE.md §4.4.

On one card a Flash-Next prefill step copies every pooled expert across the link once, so a 4096-token step shares that copy among twice the tokens of a 2048-token one (26K-token prompts: 2545 against 1413 tok/s); the checkpoint interval matches the step because a prefill chunk never crosses a checkpoint. At 4 sequences that row's cache holds 301K tokens. The Flash-Next bf16 row is a reference to calibrate and compare against, a step taking tens of seconds whatever its size; its 32K of context keeps QSA's score plane for a 32K-token step at 1 GiB.

At tp: 4, decode and verify steps carry the exact all-reduce in the gated residual's write, with the MoE gather in front of it; a prefill chunk runs the unfused chain. That chain's exact all-reduce is one-shot up to 1 MiB a message and two-shot above, which sends half the bytes on a prefill chunk; both produce the same bits. The four-rank row runs exact. At three and four ranks tp-wire: wht6 selects a different lossy wire, tiered int8 over the two-shot, above tp-wire-min-kb. Eight ranks have no lossy wire, so tp: 8 with tp-wire: wht6 stops at startup naming the all-reduce.

Four ranks is a capacity configuration: its per-step rate against tp: 2 on the same cards has not been measured, and its tiering keys are the two-rank recipe carried over. As at two ranks, each rank owns a slice of every expert's width: each parity's 128-column blocks are cut contiguously over the ranks and the odd block alternates with the parity, so every rank's share is whole blocks and every block has one owner. The resident expert bytes per rank are smaller, so the elastic window has a different shape: read the boot log's elastic line and re-derive host-pool-mib before trusting it.

host-pool-mib is what a model pins for its weights in all, each rank an equal share: the four-rank row pins 48 GiB of RAM (12 GiB a rank) that nothing else can use and the kernel cannot swap out. The amdgpu driver caps what all processes together pin at the machine's RAM less 1/64 and 1.5 GiB; the pool is locked user memory, which ttm.pages_limit does not count. A 64 GiB machine serves the one-card Flash-Next row alone, with prefix-cache-mib: 4096, at about 49 GiB pinned (docs/GUIDE.md §7). In tp1.yaml its entry carries prefix-cache-mib: 1024 so its host pins fit such a host beside the 8 GiB the fleet keeps free; there it is unloaded when another model loads, and on a 61 GB host it answered its next request after 17 to 19 s, loaded from its container again.

The Gemma 4 rows sample as the checkpoint's generation_config states, and a request switches thinking (chat_template_kwargs). vision-tower-placement: ram keeps the vision tower in host memory, so its VRAM goes to the KV cache. The 31B takes its full 262,144-token context at max-model-len: 262144, max-num-seqs: 2. The 12B keeps a bf16 cache because an 8-bit one costs it more than it saves; on Bolt-E4B an E4M3 cache cost 9x the KL and the int8 cache is unmeasured. The Bolt-E4B row serves without speculation because the published containers' drafters are block FP8.

EmbeddingGemma 2 takes an input in one step, so its max-num-batched-tokens is the longest input as well as the batch. A live Qwen3-ASR session holds one sequence, so its max-num-seqs is the most sessions served at once. On one card LTX-2.5's adapter slots fit at max-lora-rank: 16, max-loras: 1 (433 MiB).

In a file of several models, Qwen-Image's grant counts two sequences of max-model-len tokens of cache, which at the model's own 33600 tokens is 55.8 GiB on one card and 32.6 GiB a card on two, more than the cards. tp1.yaml sets max-model-len: 8192 (it needs 29.7 GiB; pictures up to about 1.9 megapixels, 1024x1024 being 4096 tokens) and tp2.yaml 20480 (25.4 GiB a card; the default 2048x2048 is 16384 tokens). Alone in a file it loads at its own length, its cache taking what the cards have left. It sizes its buffers itself and warns that it does not take image-gen-max-pixels or image-gen-max-refs.

Add server.api-key: KEY (or server.api-key-env: VAR, which reads it from the environment) to require Authorization: Bearer KEY (or X-Api-Key: KEY). debug-graph: true on the model prints every op and the kernel it resolved to, then exits without serving. A chat request:

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"minicpm5-2b-iq8r","messages":[{"role":"user","content":"hi"}],"max_tokens":50}'

An image model answers OpenAI's images API instead, with base64 pictures:

curl -s localhost:8000/v1/images/generations -H 'Content-Type: application/json' \
  -d '{"prompt":"a lighthouse on a cliff at sunset, oil painting","size":"1024x1024"}' \
  | python3 -c 'import sys,json,base64; open("out.png","wb").write(base64.b64decode(json.load(sys.stdin)["data"][0]["b64_json"]))'

http://localhost:8000/ is the dashboard. docs/GUIDE.md covers the API, memory budgets, long context, speculative decoding and operating a server.

Convert a checkpoint

rad-convert turns a Hugging Face checkpoint into a container, quantising what a recipe in data/recipes/ names. Download checkpoints with scripts/hfget.sh <org/repo> <dir>.

# Spark X2.6 uses the checkpoint's BF16 weights, tokenizer and chat template; no recipe required.
build/bin/rad-convert /srv/models/hf/XHToken-Spark-X2.6-4B -o spark-x2.6-4b-bf16.rad --max-ctx 4096
build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-27B -o qwen3.8-27b-bf16.rad \
    --recipe data/recipes/q38-27b-bf16.recipe
build/bin/rad-convert /srv/models/hf/Qwen/Qwen3.8-Flash-Next -o qwen3.8-flash-next-bf16.rad --max-ctx 4096
build/bin/rad-convert /srv/models/hf/google/gemma-4-31B-it -o gemma4-31b-w4a8.rad \
    --recipe data/recipes/gemma4-31b-dense-bf16-w4a8.recipe \
    --set quantization_config.quant_method=compressed-tensors \
    --draft-model /srv/models/hf/google/gemma-4-31B-it-assistant
build/bin/rad-convert /srv/models/hf/google/gemma-4-31B-it -o gemma4-31b-dense-w4a8-df.rad \
    --recipe data/recipes/gemma4-31b-dense-bf16-w4a8-df.recipe \
    --set quantization_config.quant_method=compressed-tensors \
    --draft-model /srv/models/hf/z-lab/gemma-4-31B-it-DFlash
build/bin/rad-convert /srv/models/hf/google/gemma-4-31B-it -o gemma4-31b-dense-q8-df.rad \
    --recipe data/recipes/gemma4-31b-dense-bf16-q8-df.recipe \
    --set quantization_config.quant_method=compressed-tensors \
    --draft-model /srv/models/hf/z-lab/gemma-4-31B-it-DFlash
# Meta's Muse Glimmer 30B, as it ships (bf16), with an int4 trunk and int8 attention values, or
# with an IQ8R trunk and an int8 head
build/bin/rad-convert /srv/models/hf/meta-models/Muse-Glimmer-30B -o muse-glimmer-30b-bf16.rad \
    --recipe data/recipes/museglimmer-30b-bf16.recipe
build/bin/rad-convert /srv/models/hf/meta-models/Muse-Glimmer-30B -o muse-glimmer-30b-w4a8.rad \
    --recipe data/recipes/museglimmer-30b-w4a8.recipe \
    --set quantization_config.quant_method=compressed-tensors
build/bin/rad-convert /srv/models/hf/meta-models/Muse-Glimmer-30B -o muse-glimmer-30b-iq8r-dflash.rad \
    --draft-model /srv/models/hf/meta-models/Muse-Glimmer-30B-assistant \
    --recipe data/recipes/museglimmer-30b-iq8r.recipe \
    --set quantization_config.quant_method=compressed-tensors
build/bin/rad-convert /srv/models/hf/ukisai/Swift-1.5-Qwen3.8-27b-W4A16-AWQ -o swift15-27b-w4a16.rad \
    --recipe data/recipes/q38-27b-w4a16.recipe
# Any single-file SDXL checkpoint (SDXL 1.0, Illustrious, NoobAI, Pony, a civitai finetune), with
# CLIP's tokenizer.json from openai/clip-vit-large-patch14
build/bin/rad-convert /srv/models/hf/OnomaAIResearch/Illustrious-XL-v2.0/Illustrious-XL-v2.0.safetensors \
    -o illustrious-xl-v2.0-bf16.rad --recipe data/recipes/sdxl-bf16.recipe \
    --tokenizer /srv/models/hf/openai/clip-vit-large-patch14/tokenizer.json

With Docker, use docker run --rm -v /srv/models:/models --entrypoint rad-convert stilldeadcode/radiance .... In the image the recipes are in /opt/radiance/share/radiance/recipes/. A checkpoint served at its own precision needs no container: its directory is the entry's model (model: /srv/models/hf/Qwen/Qwen3.8-27B). A release someone else quantised to an integer grid is written at its own numbers where the kernels read them (an int4 W4A16 release, compressed-tensors' AWQ and GPTQ or AutoRound's, on the 4-bit GEMMs with data/recipes/q38-27b-w4a16.recipe). Block FP8, MXFP4, NVFP4 and ParoQuant releases are not served: convert the model's bf16 checkpoint with an int8 recipe. docs/GUIDE.md names the recipes of record with their rad-convert commands.

Spark X2.6 runs BF16 at tp: 1, 2, 3 or 4 -- and 8, the worlds libr4d has an exact all-reduce for -- without a drafter or quantisation recipe. Its checkpoint's chat template enables thinking by default; replies return reasoning in reasoning_content. Set chat_template_kwargs.enable_thinking to false for a non-thinking request. Tool calls use the checkpoint's tagged format and are returned as OpenAI tool_calls.

Test

ctest --test-dir build -L fast -j8         # the inner loop, under 15 s; build.sh runs it
ctest --test-dir build -LE 'gpu|oot' -j8   # everything that runs without a card (about 5 minutes)
ctest --test-dir build -L gpu              # the card tests; they need the cards to themselves
ctest --test-dir build -L oot              # rebuilds every bundled plugin out of tree against an install

CMakePresets.json names the same runs (ctest --preset fast, host, gpu, full, asan), and docs/GUIDE.md §1.3 describes the tiers and the test binaries' options.

Documentation

docs/GUIDE.md installing, converting, serving, the HTTP API, operating a server
docs/TOOLS.md every flag of every tool, and the recipe syntax
docs/DIAGNOSE.md diagnose: the engine measures itself, and what makes a run a measurement
docs/DOCKER.md the image and the compose files
docs/PREPARING_YOUR_RIG.md a two-R9700 machine from firmware setup to a serving model
docs/PLUGIN.md writing a kernel library, architecture or quantiser plugin, including one for another AMD card
docs/ARCHITECTURES.md the supported model families and their geometry
docs/TP3.md any world size: the attention world and uneven delta-net heads; three ranks
docs/OPS.md, docs/KERNELS.md the op vocabulary, and what rad-kbench last measured
spec.md the design

License

Apache-2.0, copyright Deadcode and the radiance contributors. Keep NOTICE with any copy or derivative. It credits the authors and lists the MIT-licensed code the tree includes. The model weights are not covered: each keeps the license of the model it was converted from. The Docker image also holds third-party libraries under their own licenses, GPL and LGPL among them; /usr/share/doc/THIRD-PARTY in the image lists each one with its version and source.