Two card classes, tuned properly, instead of every card tuned adequately. A mechanism that wins on the RTX PRO 6000 and loses on the RTX 5090 becomes a per-device default keyed on the device — not a compromise applied to both — so a naked command runs at that card's measured best. Speculative, graphed and batched serving are each gated byte-identical to plain decode, per request, so the speed is not paid for in silent output drift.
MIT. Clone it, point it at a supported checkpoint and a Blackwell card, and it serves — no account, no key. No card? A hosted instance runs at inference.tiyuvta.ai; the lab is tiyuvta.ai, same author, and its terms and prices are documented there rather than here.
| What | Inference engine, Rust + CUDA, OpenAI-compatible serving |
| Tuned for | RTX PRO 6000 Blackwell (sm_120a) and RTX 5090, with a compile-gated H100 (sm_90a) lane |
| Format | safetensors first — the tuned path from here on. GGUF stays supported, and is still the path most models take |
| Shape | One model per GPU, replicas across cards behind an admission proxy, PP-2 when a model does not fit one card; placement validated through four GPUs. Tensor parallel, P2P and 3-stage PP are in progress |
| Author | Avi Fenesh · lab tiyuvta.ai · hosted instance inference.tiyuvta.ai |
| Licence | MIT |
Jump to — Quick start · Speed · Which models run · What the server does · Docs · Request a model
Use it if you serve on a PRO 6000 or a 50-series card and want a naked command to run at that card's measured best.
Look elsewhere if you serve on datacenter parts — the sm_90a lane is compile-gated,
secondary, and sets no defaults — or if you want a Python library to import rather than a Rust
binary behind an HTTP surface. If the model you need is not in the table below,
ask for it: support is per (model, quantization, drafter), so a format alone
never carries a checkpoint here.
Current tag: releases/latest. main runs
ahead of it. A version number in prose is stale the day after it is written, so this file does
not repeat one.
Install
The release installer is the shortest path. It selects the published sm_120a, sm_90a, or
sm_89 prebuilt, verifies the release checksum, and installs memra-server, run-gen,
run-spec, and kernel-check.
curl -fsSL https://raw.githubusercontent.com/avifenesh/memra/main/tools/install.sh | sh export PATH="$HOME/.local/bin:$PATH"
Prebuilt binaries require Linux x86_64, glibc 2.35 or newer, NVIDIA driver 580 or newer,
and the CUDA runtime libraries. They do not require nvcc. Set MEMRA_INSTALL_DIR to
override ~/.local/bin.
To build all workspace binaries from source:
git clone https://github.com/avifenesh/memra.git cd memra cargo build --release export PATH="$PWD/target/release:$PATH"
Source builds require Rust 1.85 or newer and the CUDA 13.1 toolkit. Architecture is detected
at build time; MEMRA_CUDA_ARCH is the documented override.
Quick start
Point the command at a supported Hugging Face checkpoint directory or GGUF artifact to run one chat-templated generation. New upstream model onboarding starts from the official checkpoint:
MEMRA_CHAT=1 run-gen /absolute/path/to/hf-checkpoint \
--prompt "Explain KV caches in one sentence."run-gen also accepts a supported GGUF or an hf:owner/repo[:file] spec. The hf: form
downloads and caches the selected artifact on first use.
Start the OpenAI-compatible server in one terminal:
MODEL=/absolute/path/to/hf-checkpoint
MEMRA_MODELS="qwen=$MODEL" memra-serverThe default bind address is 127.0.0.1:8080. From another terminal, stream a chat completion:
curl -sS -N http://127.0.0.1:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "qwen", "messages": [{"role": "user", "content": "Explain KV caches in one sentence."}], "max_tokens": 128, "stream": true }'
The qwen value is the alias from MEMRA_MODELS. Bearer authentication, multiple model
aliases, MTP drafter attachment, and deployment probes are covered in
docs/SERVING.md.
Speed
Flagship model, RTX PRO 6000 Blackwell, safetensors with the masked-ranks trim — the leading path. Every figure carries its conditions in docs/PERFORMANCE.md, which is also where regression re-measures land.
| Metric | Measured |
|---|---|
| TTFT p50, cold | 0.156 s (c=1) — ≤0.32 s through c=4 |
| TTFT, cached conversation turn | 0.130 s on a 5.7k-token context (full prefix restore) |
| Decode p50, single stream | RTX PRO 6000: 140 tok/s (rep medians 138–141) · RTX 5090 Laptop: 75 tok/s (range 71–80) — safetensors with the masked-ranks trim, the leading path; plain decode 75 and 44 |
| Sampled-config throughput | top-p/top-k/min-p requests sample on-device — sampled aggregate equals greedy (240–245 tok/s at c=16–32) |
| Aggregate completion | 238–245 tok/s at c=16, flat to c=32, zero sheds across capacity mixes |
| Sustained soak | 576/576 requests, 0 errors, 0 sheds, −0.27% drift |
| Spec ON/OFF exactness | 8/8 byte-identical; verify gate: zero differing logits at T=1..4, K=1/3/8 |
Image and video input on the same endpoint (OpenAI image_url /
video_url content parts, base64 data URIs; videos as animated GIF, decoded
in-process) — the checkpoint's native ViT tower runs in-engine, gated by a per-token
cosine parity oracle against the HF reference before serving (images min-cos 0.9997,
video 0.99999); vision tokens bill as ordinary prompt tokens.
Tuning on Qwen3.8-27B is finished on both paths, and safetensors is the leading one — the decode figures above are its numbers, measured with the masked-ranks trim. Step-3.7-Flash already serves on the GGUF path and its current tuning is on FP8, not Q8. Gemma-4 31B's serving stack shipped in v0.89.0: an NVFP4mix GGUF trunk (Q6_K embedding + ffn_down), batched decode and an assistant-drafter speculative path on by default, and capacity-keyed kernel mirrors that engage only on the 96GB card class they were measured on — banked on RTX PRO 6000 (450W host cap) at 65.7 tok/s plain c1, 144.9/218.9 spec prose/code, 282.6 aggregate at c8 cached, 287.5 at c16 (receipts: research/gemma-vision-20260816/, SERVED-SPEC.md; spec figures re-banked 2026-08-17 after a GGUF chat_template fix — the earlier 138.5/243.0 rows were measured under a ChatML prompt wrap a mint bug caused). A default only ships for a card class it was measured on.
Per-model detail lives in docs/MODELS.md: the three published rank flavours and the corpus behind each, why the trim cannot move output, what the prefix cache does and does not cover on a GDN-hybrid, and two measured results that went against expectation — cache depth alone swinging throughput 3.1x, and speculative decoding losing 4x on cache-carried shapes.
Same-rig samples
The README carries only representative regression samples. The full boards, methodology, thermal regime, N, and open cells live in docs/PERFORMANCE.md.
| Model / scenario | Path | tok/s |
|---|---|---|
| Qwen3.5-9B — speculative decode, short / medium / long-agentic prompt classes | GGUF NVFP4, MTP + masked-ranks trim | 281.0 / 211.7 / 187.1 |
| Gemma-4 31B dense — speculative decode at 1.7k context (NVFP4 safetensors tuning in progress) | GGUF Q4_0, MTP K=6 + trim | 97.3 |
| Qwen3.6-35B-A3B MoE — plain decode, tg128 at 512-token context | GGUF IQ4_XS, plain decode | 187.0 |
| Qwen3.6-35B-A3B MoE — the same, at 6.3k-token context | GGUF IQ4_XS, plain decode | 177.1 |
RTX 5090 Laptop, measured 2026-08-02 — medians of N=5 interleaved reps. The flagship's own numbers are in the table above; these are the rest of the roster on the second card. Conditions, thermal regime, open cells and the engine-to-engine pairings: docs/PERFORMANCE.md; raw per-run logs: research/tune-data/.
Which models run
Support is specific to a model, quantization and drafter — never to a format. A checkpoint loading is not a model being supported: each family has its own tensor census, quantization arithmetic and topology, and it counts as supported once it has passed its own gates.
safetensors is the tuned path from here on. GGUF stays supported and is still the path most models take. There is no promise that a format keeps getting new work, in either direction — a GGUF landing tomorrow may or may not be picked up, and a safetensors checkpoint can load without being on the tuned path. Where a model is supported on both, both keep working.
| Family | Supported on | Tuning now |
|---|---|---|
| Qwen3.8-27B | both, both tuned — safetensors NVFP4 and NVFP4+Q5_K GGUF | done; safetensors leads |
| Gemma-4 31B | GGUF — QAT Q4_0 and the NVFP4mix shipQ6K serving trunk, both tuned | NVFP4 safetensors |
| Step-3.7-Flash 196B-A11B | GGUF (IQ4_XS + Q8_0 MTP head, two-card PP-2) | FP8 |
| everything else | GGUF | — |
Full roster, per-card targets and the reasoning: docs/MODELS.md.
Tensor parallel, P2P and 3-stage pipeline parallel are being built now — named here as unfinished rather than listed as features.
What the server does
Table stakes, present and not worth a paragraph each: /v1/chat/completions and
/v1/completions, blocking or SSE; streaming tool_calls, separated reasoning output and
reasoning_effort mapped through each model's chat template; response_format json_object
and json_schema enforced during decode; prefix caching with same-window dedup and
session affinity; bearer auth, per-tenant cache boundaries and caps, health and readiness
probes, metrics, graceful drain; /v1/models reporting each loaded model's
supported_parameters; the model's full 262,144-token window. Two more dialects serve
over the same core with identical auth and accounting: /v1/messages (Anthropic Messages
API) and a stateless /v1/responses (OpenAI Responses API) — so Anthropic-format and
Responses-only agent clients point here directly (docs/API-SURFACES.md).
Two things here are less common:
- Vocab-masked draft heads. The drafter proposes over 32,768 frequency-ranked tokens while
verification runs the target's full 248,320-token vocabulary — so the mask moves the acceptance
rate and cannot move output. On a safetensors trunk a published
.txtof ranks drives the trim at load time (MEMRA_FRSPEC_TRIM=<ranks.txt>) and no separate draft file is needed; three flavours are published, one per workload (docs/MODELS.md). - The decode mode is a per-request decision. Speculation is not free: on shapes carried by the prefix cache it costs more than it returns, so the server drops to plain or batched decode where those measure faster, PP-2 included — and each mode is byte-identity gated against plain decode before it is allowed to serve.
Request fields, response shapes, capability gates, auth, cache semantics and admission behaviour: docs/SERVING.md.
Docs
| Read this | For |
|---|---|
| docs/MODELS.md | Every supported model, which path it runs on, per-card targets |
| docs/SERVING.md | API contract, caching, auth, admission, PP-2, and operations |
| docs/API-SURFACES.md | The /v1/messages and /v1/responses translation surfaces |
| docs/PERFORMANCE.md | Full boards, methodology, rigs, gaps, and model qualification |
| docs/FLAGS.md | Audited environment-variable catalog |
| ARCHITECTURE.md | Engine structure and the sm_120a implementation ledger |
| ARCHITECTURE-H100.md | H100-specific mechanisms and evidence |
| CONTRIBUTING.md | Development workflow and GPU correctness gates |
| docs/decisions/ | Why a default, format, target or arm was chosen — and what was rejected, with the measurement that settled it |
| releases | The changelog. Generated from conventional commits by tools/changelog.sh — what changed lives there, not in these docs |
| docs/RELEASING.md | Versioning, target-rig battery, tags, and publication |
Request a model
The support list is a series of decisions, not a plan — so the most useful thing you can send is which model you want served and on what card. Open an issue or a discussion. A concrete checkpoint with a reason carries more weight than a wishlist, and it gets read.
Contributing
Issues and PRs are welcome. Start with CONTRIBUTING.md; validation on a new GPU should use the hardware report template.
License
MIT — see LICENSE.