The branch for a batch with no draft tokens gave every request one logit row, ignoring the model state's num_new_sampled_tokens_per_step. A DiffusionGemma prefill step declares zero and never reads them, so each prefill paid a full pass over the 262144-row vocabulary matrix (8.8 ms on a GB10) for nothing. Every other model state declares one, so nothing changes for them. The warmup's grammar bitmask now skips an empty batch. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
The logprobs stashed on a request's converging step went through the top-k path only, so a request's logprob_token_ids were ignored. For a choice over many single-token options that is a real defect: at a 26-way slot only one to eight of the options rank in the top 20, the rest get no value, and the answer changes with the k requested (measured: Swahili at k=10, Greek at k=20 and 32, with most of the mass on tokens that spell the option name). The sampler now keeps the logprob token-id state like the autoregressive sampler does and passes it to compute_topk_scores with a per-row slot mapping, so every requested id gets its exact logprob at every canvas position. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
…s via extra_args
SamplingParams.extra_args (vllm_xargs on the OpenAI server) gains three
optional fields for structured reads on a DiffusionGemma model:
diffusion_seed_canvas list[int], exactly canvas_length ids. Replaces the
all-noise initial canvas after prefill; the caller
puts its answer template in known positions and
random ids in the slots it wants the model to fill.
diffusion_max_steps int. Denoise steps before the request commits.
diffusion_read_only bool. Commit the argmax canvas as soon as the step
cap is reached instead of waiting for convergence,
so a single-step request returns one forward's
logprobs at every canvas position.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
The sampler read diffusion_seed_canvas, diffusion_max_steps and diffusion_read_only straight out of extra_args on the GPU worker. A seed id outside the vocabulary reached the embedding gather as a device-side assert, and a wrong type or length raised inside the engine loop, both of which take the engine down rather than the one request. SamplingParams.verify now checks the three fields and returns a 400. The input processor passes the served DiffusionConfig so the seed length is checked against the configured canvas, not the model's default. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
A read-only request emitted its argmax canvas and then, unless the client had set max_tokens at or below the canvas length and ignore_eos, rolled into a second block from fresh noise with the seed gone. The request now carries that contract itself: verify caps max_tokens at the served canvas length and sets ignore_eos, so the scheduler ends the request on the first canvas and an end-of-turn token drawn into a noise slot cannot cut it short. A smaller client max_tokens still trims the output. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Every denoise step ran a GPU sync for the read-only mask (`.any()` and `.tolist()`) whether or not any read-only request was in the batch, and each finished prefill uploaded its seed canvas in a Python loop, one host-to-device copy per slot. DiffusionGemmaRequestStates now keeps host-side sets of the seeded and read-only slots next to the GPU flags. The sampler checks the sets first, so ordinary generation pays nothing. Seeds live in a [max_num_reqs, canvas] tensor uploaded once at add_request and applied to a whole prefill batch with one masked copy. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
The logprobs stashed on the converging step came from the sampler's schedule-tempered logits (temperature 0.8 on a first step), so a caller after calibrated probabilities had to know the schedule and rescale. A read-only request now stashes logprobs from the model's own logits at temperature 1. The argmax, and so the emitted canvas, is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
The per-slot cap was a float tensor compared against a float cast of the int32 step counter inside the compiled step. Both are now int32. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
A read-only slot never enters the encoder phase, so `~is_committing` was always true for it. Also state why remove_request leaves the GPU flags to add_request. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
The schema-to-canvas work (template rendering, single-token label slots, noise seeding, averaging over draws) lives behind an OpenAI-shaped server so clients never build a canvas by hand. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Comments say the why once, in the active voice, without asides. Two long lines are rewrapped. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
The compiled step computes the next step's self-conditioning soft embed, probs @ embed_weight, for every tile. That is a full pass over the vocabulary matrix (6.5 ms on a GB10) that no slot capped at one denoise step ever reads. The sampler keeps a host-side set of one-step slots and passes a flag that selects the variant without the matmul when a whole tile is in it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Async scheduling schedules one step ahead and guards against stepping past max_tokens on the assumption that a step yields one token plus drafts that may be rejected. A diffusion denoise step yields nothing until it emits a whole canvas, so the guard never fired and every read-only request ran one extra 32-token forward that the scheduler then discarded, a third of the request's GPU time. A read-only request with diffusion_max_steps emits on exactly that denoise step and ends. Each in-flight denoise step holds one canvas of output placeholders, so once the cap's worth of placeholders is in flight the scheduler skips the request. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
With a long option list most labels never rank in the top-k, and the model's mass sits on tokens that spell the option name instead. The server now sends the union of its label ids as logprob_token_ids so every label gets its exact logprob at every position. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
`diffusion_canvas_length` in extra_args runs a request on a canvas narrower than the served one. The server keeps its canvas for buffers and graph capture; the scheduler schedules the request's width as draft tokens each step, and the sampler already pads a short canvas back to the served width and commits only the real positions. Under async scheduling the placeholder spec-token list is cut to the width, and without it the draft-token handler reports the width per request, from a new model-state hook that the DiffusionGemma state answers from a per-slot width array. The example server picks the smallest multiple of 16 that holds a schema's answer template, so one wide server serves short reads at short-canvas cost. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
The user message may carry OpenAI image parts, passed through to vLLM untouched, so a schema can ask questions about an image. A schema whose answer lines do not fit the canvas is split into chunks that run together and share the request's state. Each chunk lists its own questions: a chunk that answers a subset of a longer list loses alignment and confidence (on per-word PII, 3 of 42 decisions flipped with a shared list, none with own lists). "ask" picks a subset of question ids for one read and "chunk_rows" caps the rows per chunk. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
"sequential": true runs a decision's chunks in order and prefills each chunk's argmax answer lines before the next, so later answers condition on earlier ones. On coupled questions (a coin-flip choice and a consequence that must match it) this gives 24/24 consistent pairs against 17/24 for one joint read, at about twice the latency; on independent questions it changes nothing, so it is off by default. The continuation goes through /v1/completions as token ids: the chat template strips the thought block from an assistant message and trims its trailing newline, and a continuation without the thought block made the model flip the consequence with confidence. Rendering the chat prefix with the tokenizer's own template and appending the block and the lines exactly avoids that. Sequential chunks therefore take text states only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Up to ten questions answer as "id: label" lines. Past that the id runs
straight into the label, space separated ("0yes 1no"), three tokens a
question against four or five, so more questions fit one read.
Measured against the line format: 42 per-word booleans agree on every
word in two chunks or four, ten 26-way choices and twenty 5-level scores
agree within one answer, which is the run-to-run spread of the line format
itself. Two tokens a question, or no id at all, lose alignment past about
twenty questions.
A continuation template now starts with the join between questions, so a
sequential chunk's tokens match what one joint template would produce
(word ids absorb a preceding space into their token, digits do not).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Matt Mastracci <matthew@mastracci.com>