DeepSeek-V4-Flash-0731-Latent-Reasoning. A self-contained model that does thinking in latent space, NVFP4-quantized, with a production vllm form for serving runtime.
https://huggingface.co/nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning
Published on blog.n.ichol.ai
Quick overview
The goal of this model is to employ latent thinking in an open source model. The idea is to compress thinking tokens and keep them within latent forms not in the output emission.
The backbone is quantized down to NVFP4.
Model: nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning
HuggingFace repo:
The DeepSeek-V4-Flash-0731 backbone. Roughly 79 GiB of weights per GPU at TP=2 (158–164 GiB total).
The DSpark draft block (3 layers), preserved from the source.
The trained latent reasoning head. 35.7M params, loaded from a single
latent_reasoning_head.safetensors(~152 MB).
first evals
BBH (BIG-Bench Hard), cot_zeroshot, 27 subtasks: aggregate 0.94 ± 0.008. Measured with lm-evaluation-harness 0.4.12 against an OpenAI-compatible endpoint. Deepseek latent reevaluated the results after giving separate 0.94 vs 0.88 exact match. Thinking enabled. 50 items per subtask, 1350 items total.
| Subtask | Score | Subtask | Score |
|---|---|---|---|
| tracking_shuffled_objects_three_objects | 1.00 | date_understanding | 0.92 |
| tracking_shuffled_objects_five_objects | 1.00 | sports_understanding | 0.88 |
| tracking_shuffled_objects_seven_objects | 1.00 | logical_deduction_five_objects | 0.88 |
| penguins_in_a_table | 1.00 | web_of_lies | 0.86 |
| formal_fallacies | 1.00 | snarks | 0.84 |
| boolean_expressions | 1.00 | ruin_names | 0.84 |
| word_sorting | 0.98 | movie_recommendation | 0.84 |
| temporal_sequences | 0.98 | salient_translation_error_detection | 0.76 |
| object_counting | 0.98 | geometric_shapes | 0.74 |
| navigate | 0.98 | causal_judgement | 0.66 |
| logical_deduction_three_objects | 0.98 | disambiguation_qa | 0.58 |
| reasoning_about_colored_objects | 0.96 | dyck_languages | 0.26 |
| hyperbaton | 0.96 | ||
| multistep_arithmetic_two | 0.94 | ||
| logical_deduction_seven_objects | 0.94 |
It is strongest where reasoning means multi-step state tracking. tracking_shuffled_objects, boolean_expressions, formal_fallacies, penguins_in_a_table all hit 1.00. It is weakest on the mechanical, syntax-heavy jobs. dyck_languages (bracket matching) sits at 0.26, the clear outlier. That is a genuine weakness, not a measurement artifact.
Lm eval uses
flexible-extract, notstrict-match. BBH'sstrict-matchregexes for the literal phraseThe answer is X. This model does not emit that phrase. Its near-zerostrict-matchscore is an answer-formatting artifact, not a reasoning failure.Per-subtask values carry about ±0.05–0.07 at 50 items each. The aggregate of 0.880 is the reliable number.
Architecture
layer 35 hidden (4096-d)
|
v LayerNorm
+--------- ReasoningCompressionHead ----------+
| Linear 4096 -> 2048 . SiLU |
| Linear 2048 -> 2048 . SiLU |
| Linear 2048 -> 2048 -> [mu, log_sigma] |
| |
| stop_head: |
| Linear 4096 -> 1024 . SiLU |
| Linear 1024 -> 1 | -> end-of-reasoning
+---------------------------------------------+
| mu (1024-d latent)
v LayerNorm
+-------------- LatentDecoder ----------------+
| Linear 1024 -> 2048 . SiLU |
| Linear 2048 -> 2048 . SiLU |
| Linear 2048 -> 4096 |
+---------------------------------------------+
|
v written back into the residual stream
DeepSeek-V4-Flash-0731 backbone (frozen, NVFP4)
| Config | Value |
|---|---|
| hidden_size | 4096 |
| latent_dim | 1024 |
| mlp_dim | 2048 |
| source_layer / target_layer | 35 / 42 |
| activation | SiLU |
| learned stop head | yes |
| head + decoder params | 35.7M (float32) |
| backbone layers | 43 |
The head is a variational compression. It predicts [mu, log_sigma] and clamps log_sigma. The decoder redistributes the latent back into the 4096-d stream. The whole thing is a single flat tensor dict distinguished by key prefix. The geometry lives in the checkpoint's own metadata, so the serving runtime needs zero configuration; it reads the head's shape straight from the file. target_proj, a frozen projection that defined the regression target during training, is included for completeness but is not used at inference.
VLLM support (sort of)
| Repository | What it is |
|---|---|
nickmitchko/ds4-reasoning-addon |
The latent-reasoning addon: the closed-loop driver, serve script, pinned requirements and GPU-sizing guide. Start here. |
nickmitchko/vllm-ds4-sm120 |
The DS4 vLLM fork it runs on (branch ds4-sm120-preview-dev). Required. Upstream vLLM cannot serve this model. |
Installing the addon does nothing by itself. The plugin registers but stays dormant until VLLM_DS4_REASONING_CKPT is set. That is a deliberate safety choice; it is safe to leave installed. The supported entrypoint is a single script that sets all the measured-good defaults for you:
# 1. the engine (a full build takes a while)
git clone https://github.com/nickmitchko/vllm-ds4-sm120.git && cd vllm-ds4-sm120
git checkout ds4-sm120-preview-dev
export CUDA_HOME=/usr/local/cuda-13.0 PATH=/usr/local/cuda-13.0/bin:$PATH
export TORCH_CUDA_ARCH_LIST="12.0"
pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu130
pip install -e . --no-build-isolation
# 2. the addon + its pinned deps
git clone https://github.com/nickmitchko/ds4-reasoning-addon.git && cd ds4-reasoning-addon
pip install -e . --no-deps
pip install -r release/requirements-serve.txt \
--extra-index-url https://flashinfer.ai/whl/cu130/torch2.11
# 3. serve: no arguments needed, the head is resolved from the model repo
release/serve_ds4_reasoning.sh
The serve script defaults to the bundled model head (latent_reasoning_head.safetensors, ~152 MiB, cached after the first run), so the recommended configuration is already applied. If you trained your own head, point HEAD_BUNDLE at it and you are done.
Why a fork at all?
The stock vLLM can't serve this model.
DeepSeek-V4 routes MoE experts by a hash keyed on input_ids. vLLM's native prompt_embeds path nulls input_ids when you supply embeddings, which would crash the engine at startup with DeepSeek V4 hash MoE routing requires input_ids. So the addon cannot use the standard embeddings path.
Instead, injection overwrites the embed_tokens output at the target positions with the decoded latent, while token ids keep flowing normally so hash-MoE routing still works. Latent steps carry a reserved pad token id purely for accounting. A latent step occupies a real KV position, so it must advance num_tokens in lockstep with num_computed_tokens. That id never reaches the client, and its embedding is never read (the row is marked is_token_ids=False, so the injected latent survives). Because this runs at the execute_model seam, it stays on the cudagraph fast path. No enforce_eager, and it batches across concurrent requests.
Speculative drafting is suppressed during the latent phase (VLLM_DS4_SUPPRESS_LATENT_DRAFTS, on by default). Leave it on. The rider injects one embedding at the last query row, so a draft slot would steal it and the real position would get the pad token's embedding instead of the latent.
Runtime Configs
Server-wide environment variables (defaults = measured-good)
| Variable | Default | What it does |
|---|---|---|
MAX_LATENT |
256 | Safety cap on latent steps (bounds a stop-head misfire). Matches the head's K=256 training. |
MIN_LATENT |
4 | Floor on latent steps, so answers never no-think. |
USE_STOP |
1 | Learned stop; 0 uses a fixed-N cap. |
STOP_THRESHOLD |
0.5 | Stop-head threshold. 0.5 is correct. Don't "fix" warmup by lowering it. |
MIN_OUTPUT_TOKENS |
4096 | Floor on the answer's token budget. |
RIDER |
1 | 0 serves the bare backbone (A/B baseline against latent reasoning). |
DEBUG |
0 | 1 prints per-request latent stats. |
MAX_MODEL_LEN |
262144 | Context window. |
MAX_NUM_SEQS |
2 | Concurrent sequences. Trades against context. |
GPU_UTIL |
0.95 | Narrow viable band. 0.97 OOMs, and lower can fail the KV check. |
TP |
2 | Tensor-parallel size. |
Same as before, plus one new one. These win over env defaults and work on both /v1/chat/completions and /v1/messages (useful for clients like Claude Code that set headers but not body fields).
| Header | Effect |
|---|---|
x-ds4-thinking |
Enable thinking without a body field. |
x-ds4-max-latent |
Per-request latent cap. |
x-ds4-min-latent |
Per-request latent floor. |
x-ds4-use-stop |
Toggle the learned stop. |
x-ds4-stop-threshold |
Per-request stop threshold. |
x-ds4-min-output-tokens |
Per-request answer-budget floor. |
Bounding latent thinking
MAX_LATENT is a safety cap, not a reasoning-depth dial. The learned stop normally ends the phase. The cap only bounds a misfire. Where you set it matters: pushing far past the K=256 the head saw in training drifts into garbage rather than thinking harder. The bundle's compression_factor is also inert at serve time. Nothing in the rider reads it. It describes what the head learned, not a budget the loop enforces.
MIN_OUTPUT_TOKENS exists because the latent phase and the answer share one budget. Each latent step bills one reserved accounting token. A client sending a modest max_tokens can spend the whole budget thinking, then get an empty answer back with stop_reason=length. The floor only ever raises a client's max_tokens. The learned stop still ends generation early once the answer is done, so a generous floor does not force verbosity.
Common client configs
Two non-obvious requirements for any client that talks to the server. Both were learned the hard way.
1. You must ask for thinking. The reasoning phase is gated on chat_template_kwargs={"thinking": true} (or the header x-ds4-thinking: 1). Without it the output is garbage. That is not hyperbole.
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8001/v1", api_key="dummy")
resp = client.chat.completions.create(
model="nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning",
messages=[{"role": "user", "content": "Write a Python LRU cache."}],
extra_body={"chat_template_kwargs": {"thinking": True}}, # REQUIRED
max_tokens=4096,
temperature=0.6,
)
print(resp.choices[0].message.content)
2. Give the answer token headroom. Reasoning and answer share one budget. A tight max_tokens can be consumed entirely by the latent phase. When in doubt, use the MIN_OUTPUT_TOKENS floor (default 4096) rather than hand-tuning per request.
Known limitations
The first request or two after startup can be degenerate repetition. Low draft acceptance, "be be be" output, even with the stop head firing normally. It settles by itself and then stays correct. Send one throwaway request after startup and treat a single bad early answer as unwarmed, not broken. And do not lower STOP_THRESHOLD or force SPEC_TOKENS=1 in response. Neither is the cause.
An armed rider and a silently-dormant one produce identical server logs. If VLLM_DS4_REASONING_CKPT is not set, or the request does not ask for thinking, the plugin quietly does nothing and you have no idea. VLLM_DS4_REASONING_DEBUG=1 is the only way to confirm injection is actually happening. It prints steps=N stop_step=... max_p=... end=stop|cap per completion, and says so explicitly when nothing was injected.
What it takes to run it
The backbone is NVFP4, so this is Blackwell-class hardware (sm120) or better. There is no way around it. The sm120 sparse-MLA kernel path is specific to this fork, and Hopper and earlier are untested. Sizing is entirely about fitting the ~158–164 GiB of weights plus the fp8 KV cache.
| VRAM (total) | Verdict |
|---|---|
| ≥ 192 GiB (2× 96, or 4× 48+) | Recommended. Long context (256k) with room for the KV cache. |
| 160–192 GiB | Workable. Weights fit; drop MAX_MODEL_LEN to 32k–64k so KV fits. |
| < 160 GiB | Not enough for NVFP4 weights at any context. |
The verified configuration: 2× RTX PRO 6000 Blackwell Max-Q (96 GiB each), TP=2, context 262,144 (443,012 fp8 KV tokens), gpu_memory_utilization 0.95, throughput 11.1 ms/token with the head plus DSpark spec decode (1.40×), and 89–91% draft acceptance.
max_num_seqstrades against context. At 256k the fp8 KV cache only holds a couple of full-length sequences. Raising concurrency starves KV and either OOMs at startup or silently truncates usable context. Raise it only alongside a lowerMAX_MODEL_LEN.If the engine fails its KV-cache check at startup, lower
MAX_MODEL_LENfirst, thenMAX_NUM_SEQS.gpu_memory_utilizationhas a narrow viable band in both directions.
what actually happens when a request hits the server: The backbone prefills your prompt once, then takes autoregressive latent decode steps. Each step's input embedding is decoder(head(previous layer-35 hidden)), fed back from the model's own hidden state via a per-request anchor store. This continues until the learned stop head crosses its threshold, emits response, and the answer decodes as ordinary tokens.
h_src (layer 35) --layer_norm--> head -> mu (latent, 1024-d)
mu --layer_norm--> decoder -> hidden vector (4096-d)
This runs batched across concurrent requests and on the cudagraph fast path (no enforce_eager, no --max-num-seqs 1), and it streams. One latent step stands in for roughly six reasoning tokens. The model thinks compressed, then speaks.
Limitations and Future Work
Blackwell. NVFP4 needs sm120 native kernels, and the sparse-MLA path is fork-specific.m.
The surfaced trace is not the computation. Reasoning happens in latent space, so the text you see is not a faithful token-level record of the thinking that produced the answer. The latent space remains opaque.
Evaluation is BBH-only at 50 items/subtask. No multi-task or long-context suite reported. The aggregate is solid; the coverage is still narrow.
dyck_languagesat 0.26 is a real weak spot, Syntax-heavy, mechanical tasks are precisely what compressed-latent reasoning struggles with.
Future work includes:
A broader benchmark suite: multi-task and long-context, so the BBH 0.880 is not one number carrying the whole story.
Making the latent reasoning interpretable, so we can see what the model thinks, not just that it thinks.
Better handling of the mechanical-syntax tasks (
dyck_languages) that are the current weak point.Pushing the DSpark interplay further. The closed latent loop and speculative decoding together already give 1.40× at 89–91% draft acceptance. There is more to squeeze there.
This blog was authored with the help of nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning
Implementation: nmitchko/DeepSeek-V4-Flash-0731-Latent-Reasoning · ds4-reasoning-addon · vllm-ds4-sm120Blog: blog.n.ichol.ai