TL;DR
- Before: Qwen3.8-27B on one RTX 5090, with a 262k-token context, reading prompts at about 10,000 tokens/s and generating about 200 tokens/s for one chat, or about 800 tokens/s in total across eight at once.
- What I did: extracted the part of Cloudflare’s Clef decision model that differs from Qwen3.8-27B as a LoRA adapter, and patched vLLM so that one copy of the weights serves both Qwen for chat and Clef for decisions.
- Result: a local decision model running beside Qwen3.8-27B, answering in 150–310 ms on prompts of up to 32k tokens whose start it has seen before (reading a new 32k-token prompt in full takes 4.9 s). The cost to chat: the context drops to 181k tokens, and at most six chats run at once instead of eight, about 570 tokens/s in total instead of 800.
- Code: the adapter extraction, the vLLM plugin, the decision service and setup instructions are in clef-lora-vllm.
I have one RTX 5090 at home, and I want two kinds of model on it: a chat model good enough to drive coding agents, and a decision model like Jev.
For chat and coding, I have found that Qwen3.8-27B fits this card well at the moment. I run it with DFlash2, a community vLLM setup for RTX 50-series cards that serves a 4-bit copy of the model with speculative decoding, with a few settings changed for my PC. It decodes about 200 tokens/s for one conversation and about 800 tokens/s in total with eight requests at once, reads prompts at about 10,000 tokens/s, and uses all of the card’s 32 GB.
On 2026-09-30 Cloudflare released Clef, a decision model that accepts the same requests as TypeSafe’s hosted decision model, Jev. Cloudflare published a leaderboard with it, on which Clef ranks first on decision score of 73 decision models, ahead of Jev. Clef is built from Qwen3.8-27B, however, so to run as released I would need to figure out a way to run a second 27-billion-parameter model. The first approach I considered was to switch models on the GPU as needed, but a switch takes several minutes.
It turns out it was possible to patch the chat server so it can answer Clef’s requests too, from the copy of the model it already holds. Comparing Clef’s weights with the base model’s showed that Clef is Qwen3.8-27B with only its last 24 of 64 layers changed, and that the change has the shape of a merged rank-256 LoRA fine-tune. Factored back out, the change is a LoRA adapter of 700 million parameters, which vLLM applies only to requests that name it, so the same vLLM process serves chat and decisions at the same time. Part of Clef is an additional small scoring network that can run on the GPU or, with a small performance cost, on the CPU. I set this patch up as a Docker Compose overlay file on top of the existing chat setup.
While patching, I also added something Clef doesn’t support out of the box: reusing the cached start (prefix) of a prompt, so that a large request is answered quickly when its start was seen before. A 16,000-token prompt grown by four lines takes 180 ms instead of 2.1 s, and a 32,000-token one 0.31 s instead of 4.9 s.
To fit Clef’s overlay parts into VRAM, I needed to reduce chat’s context pool to 181k tokens from 262k, which leaves room for six concurrent chat requests instead of eight.
1. The chat setup before Clef
| Decode, one conversation, a fixed coding prompt at temperature 0 | 243 tokens/s |
| Total decode across eight conversations | about 800 tokens/s |
| Time to first token, short prompt | 90 ms |
| Prompt reading, 10,000 tokens | 10,100 tokens/s |
| Prompt reading, 90,000 tokens | 4,079 tokens/s |
With the server running, about 200 MiB of the card’s 32,607 MiB is free.
2. What Clef does
Clef takes a prompt that describes the state and every option to decide on, runs the model over it in a single forward pass, and takes the last layer’s
output at every prompt position (its hidden states: one vector of 5,120 numbers per token). A small scoring
network shipped with Clef’s release (joint_head.safetensors, 256 MB) returns one score per option,
and a softmax per question turns the scores into probabilities. Cloudflare reports a median of
209 ms per request on an H200, and scores of 97.7% on ARC-Challenge, 94.2 macro-F1 on BANKING77
intent detection and 97.4 on CLINC150.
Clef uses the same API as Jev: it answers at POST /v1/systemone.
Answering a Clef request therefore needs two things the chat server does not provide: the forward pass has to run with Clef’s weights rather than the base model’s, and the hidden states have to reach the scoring network, whereas vLLM’s API returns only tokens.
3. Clef is the base model plus a low-rank change
Qwen3.8-27B's weights Clef's change, as a rank-256 LoRA adapter the two added together, as Clef ships them
In each of the 186 changed matrices: WClef = WQwen + B · A
As released: two models
Qwen3.8-27B
27B
27,356,728,560
parameters
Clef
layers 40–639.13B changed
layers 0–39identical
27B
27,356,728,560
parameters
Two copies: 54.7B parameters
LoRA extracted from Clef
Qwen3.8-27B
adapter
0.7B
700,366,848
parameters
rank 256, on
layers 40–63
27B
27,356,728,560
parameters
- chat requests: the base alone
- requests for
clef: base plus adapter
One copy and the adapter: 28.1B parameters
4. Running Clef inside vLLM
vLLM can apply a LoRA adapter per request. Started with --enable-lora and
--lora-modules clef=/models/clef-adapter, it applies the adapter to any request that names
the model clef, and serves every other request from the base weights as before, in the same
batches. The hidden states need code inside vLLM, because its HTTP API returns only tokens. A
plugin loaded into vLLM, clef_capture (about 1,220 lines of Python), takes the last layer’s
output for decision requests while it is still on the graphics card, runs Clef’s scoring network
there, and writes the option scores to a file for the decision service, which answers POST /v1/systemone.
The plugin, the service and the steps to set them up are in the
clef-lora-vllm repository.
What the RTX 5090's 31.8 GiB of graphics memory holds
Qwen3.8-27B weights, 4-bit18.1 GiB Clef adapter, bf161.3 GiB scoring network and its working memory0.5 GiB DFlash draft model1.3 GiB key-value cache4.75 GiB CUDA graphs1.0 GiB Windows desktop and CUDA context1.6 GiB everything not measured separately3.3 GiB
Decisions added the adapter and the scoring network, 1.8 GiB together, which came out of the key-value cache: 4.75 GiB instead of 6.5.
- It names the base model, so vLLM applies no adapter.
- Layers 0–63 run with Qwen3.8-27B's weights alone.
- The last layer's output gives the next token, and the pass repeats for each token of the reply.
- It names the model
clef, so vLLM applies the adapter, for this request only. - Layers 0–39 run with Qwen3.8-27B's weights alone, as for chat.
- In layers 40–63, each changed matrix also runs the adapter's two thin matrices and adds their output, which gives the same result as Clef's weights (teal).
- The plugin takes the last layer's output at every prompt position and runs Clef's scoring network on it, on the card.
- The network returns one score per option, and the plugin writes them to a file for the decision service.
5. Reusing the start of long prompts
One of the uses I have in mind for decisions is a game. Each prompt holds a large fixed text (the rules and instructions on how to play), then a game log that only ever grows at its end, then a question such as which step to take next, with the possible steps as options. I expect all of these to add up to a 14,000–16,000-token prompt, which would take about 2 seconds to process. With regular chat models we expect vLLM to reuse the already processed start and process only the new end, which brings the latency down to a fraction of a second. The original Clef setup does not support prefix caching. I was able to make adjustments to support it in my plugin.
Clef processes input in the following way:
- The prompt goes through the model, which produces one vector per token: 5,120 numbers.
- The scoring network has two parts:
- First part, per token: it turns each token’s vector into a 12 × 1,024 matrix. It does this for each token separately, without looking at any other token.
- Second part, scoring: it takes the question and option tokens and computes one score per option. To do that it looks across the matrices of every token in the prompt.
The second part needs one more input. Besides the matrices, it also uses the original vectors of the question and option tokens themselves, for example the average vector of each option’s tokens.
A token’s vector depends only on the tokens before it, because the model reads left to right, and the first part of the scoring network handles each token on its own. A token’s matrix is therefore the same every time the prompt starts the same way, so the plugin keeps a history of these matrices. When vLLM reuses the start of the prompt, the plugin takes the matrices for that start from the history instead of computing them again. The plugin stops vLLM’s reuse before the first question, so the question and option tokens are always computed fresh and their vectors are available.
Measured on the game
A test game used a 14,000-token fixed text and a game log that gained five turns before each of 12 decisions, each decision asking which of six steps to take and whether energy is low:
| Prompt | Reused | Total | Of which the scoring network | |
|---|---|---|---|---|
| first decision, nothing to reuse | 14,434 tokens | 0 | 2.06 s | 97 ms |
| decisions 2–12, with reuse | 14,555–15,841 tokens | 13,968 | 0.17–0.45 s | 27–48 ms |
vLLM reuses a cached start only in whole blocks, which for this model are 1,552 tokens, so with reuse it reads only the part of the prompt after the last whole matching block, 600 to 1,900 tokens here (the end of the old game log, the new turns and the question), which is why the time grows within the game until the state passes the next 1,552-token boundary.
6. Latency by prompt length, and against Jev
The figures so far come from one game. To see how decision time grows with the prompt, and how it compares with Jev, I set up a script that sends one 6-option choice question about a support transcript of varying length:
| Prompt tokens | Read in full | The identical request again | Transcript grown by four lines | Tokens reused |
|---|---|---|---|---|
| 310 | 137 ms | 141 ms | 147 ms | 0 |
| 739 | 146 ms | 140 ms | 143 ms | 0 |
| 1,015 | 151 ms | 141 ms | 147 ms | 0 |
| 2,022 | 330 ms | 144 ms | 148 ms | 1,552 |
| 4,002 | 475 ms | 142 ms | 145 ms | 3,104 |
| 8,019 | 1,032 ms | 152 ms | 157 ms | 7,760 |
| 16,002 | 2,054 ms | 179 ms | 180 ms | 15,520 |
| 32,013 | 4,926 ms | 308 ms | 311 ms | 31,040 |
TypeSafe publishes no latency figures for Jev 1.13. Its limits are 64k tokens per request and 32k
for the state plus the longest question. I ran a quick benchmark on 2026-10-13 to see how Jev’s latency depends on prompt length,
using the free Jev model through OpenCode (jev-1.13-free):
| Jev’s token count | New transcript, median | The identical request again, median |
|---|---|---|
| 491 (a one-line transcript) | 222 ms | 263 ms |
| 1,400 | 278 ms | 260 ms |
| 5,183 | 272 ms | 256 ms |
| 15,549 | 373 ms | 360 ms |
| 28,790 | 616 ms | 513 ms |
Side by side, at the nearest prompt lengths (token counts come from different tokenizers, each includes its own question template, and Jev’s times include the network to OpenCode and OpenCode’s hop to TypeSafe):
| Prompt tokens, Jev / here | Jev | Here, read in full | Here, grown by four lines |
|---|---|---|---|
| 491 / 310 | 222 ms | 137 ms | 147 ms |
| 1,400 / 1,015 | 278 ms | 151 ms | 147 ms |
| 5,183 / 4,002 | 272 ms | 475 ms | 145 ms |
| 15,549 / 16,002 | 373 ms | 2,054 ms | 180 ms |
| 28,790 / 32,013 | 616 ms | 4,926 ms | 311 ms |
For my use, a game whose prompt grows at the end, the local setup answers a grown prompt in 145–311 ms, against Jev’s 222–616 ms for prompts of about the same lengths.
7. Several requests at once
Chat
This used the DFlash2 project’s own concurrency benchmark: five essay prompts, 512 tokens each at temperature 0. On these prompts only about 30% of draft tokens are accepted, so one stream runs at about 150 tokens/s, rather than the 200 or more of code.
| Streams | Total, tokens/s | Before Clef |
|---|---|---|
| 1 | 141–149 | 125 |
| 2 | 277–298 | 244 |
| 4 | 507–567 | 456–470 |
| 8 requested, 6 running | 568–581 | 801, all eight running |
Up to four streams, chat scales as it did before, at about 150 tokens/s each. Of eight streams, two waited about 4 s for a free place in every round; with six running, vLLM reported 639 tokens/s in all.
Decisions
| At once | 300 tokens: median, decisions/s | 16k tokens, read in full: median (slowest), decisions/s |
|---|---|---|
| 1 | 0.14 s, 7.3 | 2.4 s, 0.42 |
| 2 | 0.21 s, 9.4 | 3.3 s (4.3 s), 0.47 |
| 4 | 0.31 s, 12.8 | 5.4 s (8.3 s), 0.48 |
| 8 | 0.49 s, 15.6 | 9.3 s (16.2 s), 0.49 |
Short decisions scale well, to about 15 per second. Long ones are limited by reading the prompt, about 8,000 tokens per second in all, so sending more at once only makes the queue longer.
For the game, the case that matters is several decisions about the same state, for example several questions asked of one position at the same moment. A test game with a 14,000-token fixed text sent N decisions at once in each round, with the same state and different questions:
| At once | New state, nothing cached | Five new turns | Same state, new questions | Five more turns |
|---|---|---|---|---|
| 1 | 2.53 s | 0.21 s | 0.23 s | 0.18 s |
| 2 | 2.04 s | 0.34 s | 0.27 s | 0.34 s |
| 4 | 2.10 s | 0.55 s | 0.55 s | 0.46–0.63 s |
| 8 | 2.31–2.52 s | 0.54–1.04 s | 0.52–1.03 s | 0.49–1.24 s |
When nothing is cached yet, only the first request reads the start. The others wait in vLLM’s queue and then reuse all 13,968 cached tokens, so eight decisions take about as long as one full read. With the start cached, every request reuses it, and the time grows from about 0.2 s for one request to about 1 s for eight, because each still reads its own new turns and question and is scored on its own. No request had to be read twice.
Other people published ways to run Clef in the week after its release. The work described here was done separately and does not use their code or files.
- trevest’s Clef-LoRA-for-Qwen3.8-27B-GGUF,
published on 2026-10-03, with an MLX version on 2026-10-05, also recovers Clef’s change to
layers 40–63 as an adapter on Qwen3.8-27B, so that one copy of the model answers both chat and
decisions, with the scoring network on the CPU behind
/v1/systemone. It runs in a patched llama.cpp and in MLX. Its model card reports, for the bf16 base model plus its adapter against Clef’s own hidden states, a median relative difference of 0.6% and a cosine similarity of at least 0.9995. It limits a decision to 4,096 tokens and reads every prompt in full. - kurcontko’s clef-NVFP4, published on 2026-10-04, is a vLLM plugin that serves the whole of Clef as a model of its own, with the scoring network inside vLLM, from a 4-bit copy of Clef’s weights. Its model card reports 94.7% agreement with bf16 Clef on 1,400 records. taylorsm9’s clef-vllm-spark, published on 2026-10-07, runs that plugin with Clef’s bf16 weights on a DGX Spark.
Running Clef as a separate vLLM model needs graphics memory for a second copy of the 27B weights, which the card described here does not have. The setup in this post shares one copy as trevest’s does, and, like kurcontko’s plugin, runs the scoring network inside vLLM. It differs from trevest’s in running on vLLM beside speculative decoding, in accepting prompts up to 32,768 tokens, and in reusing the start of prompts that grow at the end.