Theoretical Upper Bounds for LLM Performance

Note: this post is AI generated, adapted from my working notes behind the Our Models calculator. It is a work in progress and may lack rigor in places. If you find an issue in the formulation, please email me at [email protected].

How fast can a machine serve a large language model? I built Our Models, a database of open models, AI hardware, and audited model profiles, to answer that question for any hardware and model pair. This post derives, from first principles, the math the calculator runs.

The math gives a ballpark ceiling on how many tokens per second a given hardware and model pair can serve, a roofline the pair cannot exceed under a stated set of assumptions. Nothing in it is specific to local machines. The decode bound needs only two numbers from the hardware, memory capacity and memory bandwidth, and the prefill bound adds a third, compute throughput, so the same formulas cover a MacBook and a datacenter GPU node. The post focuses on local hardware because local machines are the weak ones, and the weaker the machine, the harder good throughput is to get and the more the ceiling matters. A real implementation lands somewhere below these ceilings.

The argument builds in layers, each created by a problem the previous layer cannot solve. A memory system is two numbers, capacity and bandwidth, and their product turns out to be a natural measure of what the system can fund. That product yields a clean but loose throughput ceiling for batched decoding. The loose ceiling ignores per-session context traffic, so it is too generous for real serving, and repairing it gives the bound the calculator actually uses. The repaired bound is then filled in per architecture by small adapters and checked against real hardware. That covers decode, the phase that generates tokens. A second part runs the same program for prefill, the phase that processes the prompt, where the limiting resource switches from memory bandwidth to compute.

Interactive figures accompany the derivation. They all use the same toy setup, a 32B-class dense model at 4-bit (18 GB of weights, 6 GB of runtime overhead, 0.26 GB of KV per thousand tokens of context) on a 128 GB machine with 800 GB/s of sustained bandwidth, so the numbers stay comparable from figure to figure.

Four questions

Serving an LLM raises four separate questions, and mixing them up is an easy way to be wrong about a machine.

Question	What it asks
Resident fit	Can the model plus runtime overhead be held in memory at all?
Single-session speed	What is the memory-side ceiling for one active conversation?
Useful serving throughput	Across many active sessions, how many tokens per second can the device produce while each session stays above a minimum useful rate?
Prompt wait	How long does a user wait for the first token while their prompt is processed?

The third question is the hard one for decode. A machine can fit many sessions in memory and still be too slow per session at that concurrency, so the calculator must report both whether sessions fit and whether the fitting sessions are fast enough to be worth running. The fourth question belongs to prefill, runs on a different resource, and gets its own part of the post.

Every number produced here is an upper bound. A real system can fall below it for reasons the model deliberately ignores, among them kernel quality, quantization overhead, scheduling, CPU involvement, paging, interconnects, and thermal throttling. The decode bounds also ignore compute, which is safe because decode is memory-bound. Prefill is usually compute-bound, so its bounds have to include it. The value of a clean upper bound is that it tells you the best case you are allowed to hope for, and therefore how much room an implementation still has.

Memory power

Before any model-specific detail, ask what a memory system fundamentally offers. When people compare accelerators for local inference they list many specs, from capacity and bandwidth through compute throughput, cache hierarchy, PCIe lanes, and thermals. For the decode phase of autoregressive generation, two of these recur in almost every bound. How much state can the memory hold, and how fast can it move that state? We start there.

Two numbers and their product

Model an idealized memory system by exactly two quantities.

$$C = \text{usable memory capacity}, \qquad R = \text{sustained memory bandwidth}.$$

Capacity is a stock, the amount of resident state that can exist at once. Bandwidth is a flow, the amount of state that can be moved per second. They have different units and answer different questions, so any single product of them needs justifying before we rely on it.

Define memory power as

$$D = C R .$$

If $C$ is in GB and $R$ is in GB/s, then $D$ is in $\mathrm{GB}^2/\mathrm{s}$. Power is meant in the colloquial sense of capability, as in computing power, and no watts appear anywhere in this model. The units look strange, so the rest of this section explains why this particular combination of $C$ and $R$ is the right scalar.

For a feel of the scale, a 24 GB GPU at 1000 GB/s has $D = 24 \times 1000 = 24{,}000\ \mathrm{GB}^2/\mathrm{s}$. One thing to be careful with throughout is that every memory quantity in a given calculation must use the same unit system. The equations are identical in bytes, GB, or bits, and only the numeric value of $D$ changes.

Feasibility theorem

Here is the toy problem that justifies $CR$. The models in this formulation are deliberately simple, and they are meant as rules of thumb, useful when deciding on hardware for a model or when checking whether an existing setup is getting the most out of the hardware it runs on.

A pure memory workload is a pair $w = (h, r)$, where $h$ is the resident information it must keep alive and $r$ is the information flow rate it must sustain. The workload is feasible on the system $M(C, R)$ when the memory can both hold the state and carry the traffic. There are exactly two ways to fail.

The first is a capacity failure. If a workload needs more resident state than the memory can store, it cannot fit, and no scheduling trick repairs that.

$$h \le C .$$

The second is a flux failure. If a workload needs more traffic per second than the interface can deliver, it cannot be sustained, and no surplus capacity repairs that.

$$r \le R .$$

In the idealized model these two conditions are also sufficient. If $h \le C$ and $r \le R$, allocate $h$ of state and stream at rate $r$. Therefore the feasible set is precisely the rectangle

$$F(C, R) = \{(h, r) : 0 \le h \le C,\ 0 \le r \le R\},$$

whose area is

$$\mu(F) = C R .$$

So memory power has a concrete meaning. It is the measure of the feasible workload region. A workload lives at a point $(h, r)$ in the plane, the system can serve every workload inside its rectangle and none outside, and the size of that rectangle is $CR$.

A caution before building on this. The rectangle is a toy model, and the world is messier in both directions. Real hardware delivers only a fraction of its catalog bandwidth. Even a perfectly sequential read falls short of the spec number, and the achieved fraction depends on the access pattern, on whether enough parallel work is in flight to hide memory latency, and on the hardware itself. The sufficiency claim is idealized too, because the rate a machine reaches depends on which bytes are read and in what order. The bounds below survive this, since real inefficiency only pushes a machine further below its ceiling. The cost lands on comparisons between machines. If one machine sustains 85% of its spec bandwidth and another 60%, ceilings computed from spec numbers make the second machine look better than it really is. Read $R$ as sustained bandwidth where a measured number exists, and read every comparison in this post as approximate.

A caveat on the metric

One misreading is worth heading off. The product $CR$ measures the workload-feasibility region, the set of jobs the device can support at an instant. A different question, “how many distinct memory histories can this device produce over a time $T$”, has a different answer, namely the resident information plus the information streamed over that window,

$$C + R T .$$

The feasibility-region reading is the one an inference-throughput bound will need, because serving means keeping sessions resident in memory and streaming data for them.

A first decode bound

The feasibility theorem describes static workloads. Decoding is dynamic, emitting tokens over time. This section turns memory power into a throughput ceiling for batched decoding, and in doing so shows where the product $CR$ enters a real performance bound.

A toy decoder

Consider the decode phase of an autoregressive transformer, simplified to the memory system alone. Let

$$W = \text{model weight footprint}, \qquad K = \text{per-session KV/state footprint},$$

let $b$ be the batch size (number of concurrent sequences), and let $s$ be the number of decode steps per second. Each decode step emits one token per active sequence, so aggregate output throughput is

$$T = b \, s .$$

Throughput is a product of two things, how many sequences run in parallel and how fast the shared model can be swept, and the memory system caps each factor separately.

Capacity limit

The model weights must be resident once, and every active sequence needs its own KV cache, the per-conversation attention state that grows with context length. So the resident state is $W + bK$, and it must fit.

$$W + bK \le C \quad\Longrightarrow\quad b \le \frac{C - W}{K} .$$

This is the capacity limit, and it sets the maximum parallelism. It is exactly the $h \le C$ condition of the feasibility theorem, with $h = W + bK$.

Bandwidth limit

In a dense transformer, each decode step must apply the model weights. In the memory-bound idealization, applying the weights means streaming roughly $W$ of data per step. With bandwidth $R$, the step rate obeys

$$s W \le R \quad\Longrightarrow\quad s \le \frac{R}{W} .$$

This is the bandwidth limit, and it sets the maximum step rate. It is the $r \le R$ condition, with the per-step traffic playing the role of the flux.

Memory-power decode bound

Multiply the two caps. Throughput is parallelism times step rate, and each is separately bounded, so

$$T = b\, s \le \frac{C - W}{K}\cdot\frac{R}{W} = \frac{R\,(C - W)}{K W} .$$

Substituting $D = CR$ and factoring out $C$ exposes the memory-power term.

$$T \le \frac{D}{K W}\left(1 - \frac{W}{C}\right).$$

When the model is much smaller than memory, $W \ll C$, the correction vanishes and the bound collapses to the memorable form

$$T \lesssim \frac{D}{K W} .$$

This is the memory-power decode bound. The product $CR$ appears because maximum throughput genuinely factors into maximum parallelism times maximum step rate.

$$\underbrace{\frac{C}{K}}_{\text{how many sessions resident}}\times \underbrace{\frac{R}{W}}_{\text{how many sweeps per second}} = \frac{CR}{KW} = \frac{D}{KW} .$$

The numerator $D = CR$ is the machine. The denominator $KW$ is the workload, the per-session state times the model sweep size. The bound reads as throughput is memory power divided by memory cost per active model-token.

Scope of the bound

The memory-power decode bound governs batched throughput. Set $b = 1$ and it degenerates to

$$T \lesssim \frac{R}{W},$$

so for a single session only bandwidth matters and capacity is merely a fit constraint. That is the correct behavior, and it makes clear what each metric is for.

Use case	The metric that governs it
Single-user local chat	bandwidth $R$, with capacity $C$ as a fit gate
Largest model that fits	capacity $C$ first, then bandwidth
Maximum batched decode throughput	memory power $D = CR$

Memory power is the right scalar precisely for the third row. The next section explains why even that bound is too optimistic.

Missing traffic

The memory-power decode bound assumes the only per-token memory traffic worth counting is the model sweep $W$, shared across the batch. Real decoding also reads each session’s growing KV cache, and that traffic is private, so it does not amortize over the batch. Ignoring it makes the bound promise throughput that long-context serving can never reach. This section introduces the correct per-token accounting and shows the memory-power bound falls out of it as a loose corollary.

Universal resource bound

Step back to the most general statement, which holds regardless of architecture. If each output token costs at least $q_{\min}$ of unavoidable memory traffic and at least $a_{\min}$ of unavoidable compute, and the device delivers at most $R$ bytes/s and $F$ FLOP/s, then over all setups that fit,

$$T_{\max} \le \max_{\text{setup fits}} \min\!\left(\frac{R}{q_{\min}},\ \frac{F}{a_{\min}}\right).$$

Two rooflines, and the workload lives under the lower of them. For decode I assume the memory roofline is the lower of the two, which is the usual case for local serving, and keep only its half of the minimum,

$$T_{\max} \le \frac{R}{q},$$

where $q$ is the memory traffic per output token. Everything in the decode part now reduces to estimating $q$ honestly. The compute half of the minimum is not discarded, though. It returns as the binding limit in the prefill part, which computes where the two rooflines cross. Dropping it here is also practical. A hardware catalog can collect capacity and bandwidth consistently across consumer and workstation devices, while comparable sustained-compute numbers are much harder to obtain, a price the prefill bound will have to pay.

Bytes per token

Split the per-token traffic into the two kinds that behave differently under batching. The model (or active-expert) weights are shared, since one sweep serves the whole batch, so their per-token cost is divided by the batch. The KV/context read is private, since each session reads its own cache, so its per-token cost is not divided at all. With $W_{\mathrm{active}}$ the shared weight traffic per iteration, $\rho$ the tokens emitted per session per iteration (one for ordinary decoding), and $K_{\mathrm{read}}(L)$ the private context traffic per output token at active context length $L$,

$$q_{\text{simple}}(b) = \frac{W_{\mathrm{active}}}{b\,\rho} + K_{\mathrm{read}}(L).$$

The first term shrinks as the batch grows, because more sessions share each weight sweep. The second term does not move. A larger batch does not make any session’s context cheaper to read, and this asymmetry is the main reason long-context serving behaves differently from short-context serving.

Interactive figure: bytes per output token as the batch and the read context change. Enable JavaScript to explore it.

Memory-power bound as a corollary

Treat $q_{\text{simple}}$ as an optimistic lower estimate of the true bytes per token, $q_{\text{simple}}(b) \le q_{\text{actual}}(b)$. Dividing $R$ by a smaller denominator gives a larger quotient, so substituting $q_{\text{simple}}$ keeps the result an upper bound.

$$T_{\max} \le \max_{b\,:\,W_{\mathrm{resident}} + b K_{\mathrm{store}} + O \le C}\ \frac{R}{\dfrac{W_{\mathrm{active}}}{b\rho} + K_{\mathrm{read}}(L)} .$$

Here the maximization runs over batches that fit in memory, with $W_{\mathrm{resident}}$ the resident model footprint, $K_{\mathrm{store}}$ the KV memory stored per session, and $O$ the runtime overhead. This is the honest simple bound, bandwidth divided by shared-per-token cost plus private-per-token cost, maximized over fitting batches.

Now recover the previous section’s bound by deliberately throwing information away. Since $K_{\mathrm{read}}(L) \ge 0$, dropping it only loosens the denominator.

$$\frac{R}{\dfrac{W_{\mathrm{active}}}{b\rho} + K_{\mathrm{read}}(L)} \le \frac{R}{\dfrac{W_{\mathrm{active}}}{b\rho}} = \frac{b\,\rho\,R}{W_{\mathrm{active}}} .$$

The capacity constraint caps the batch at $b \le (C - W_{\mathrm{resident}} - O)/K_{\mathrm{store}}$, so

$$T_{\max} \le \rho\,\frac{R\,(C - W_{\mathrm{resident}} - O)}{K_{\mathrm{store}}\,W_{\mathrm{active}}} = \rho\,\frac{D}{K_{\mathrm{store}}\,W_{\mathrm{active}}}\left(1 - \frac{W_{\mathrm{resident}} + O}{C}\right).$$

This is the memory-power decode bound again. It is what you get by discarding the private context term and using the largest batch that fits. That is why it can sit far above achievable throughput while remaining a true ceiling. The ordering is

$$\text{actual throughput} \;\le\; \text{simple (KV-aware) bound} \;\le\; \text{memory-power bound}.$$

The memory-power bound is the orientation line. The KV-aware bound is the one to serve from, and the next section develops it into the operational calculator.

KV-aware bound

This section turns the simple bytes-per-token bound into the model the calculator runs. Three refinements are needed. Context must be split into the part that controls memory and the part that controls speed. The shared weight traffic must be allowed to grow with the batch, which matters for mixture-of-experts (MoE) models, models that route each token through a small subset of their weights. And the batch must be filtered so that we never count concurrency at which every session has become uselessly slow.

Two context lengths

A serving system usually reserves KV space for a long maximum context but reads, on average, a shorter active context. These two lengths drive different parts of the bound, so we keep them separate.

$$\begin{aligned} L_{\mathrm{alloc}} &= \text{reserved (maximum) context}, \\ L_{\mathrm{read}} &= \text{average active context}, \\ L_{\mathrm{read}} &\le L_{\mathrm{alloc}}. \end{aligned}$$

The allocation length controls how much KV memory each session reserves, and therefore how many sessions fit. The read length controls how much context each output token must stream, and therefore per-token cost. Collapsing them into one number either overcharges memory or overcharges speed.

Model quantities

A model contributes five quantities. Two are about fitting, two are about speed, and one is about decoding style.

Symbol	Role
$W_{\mathrm{resident}}$	Full resident footprint (for MoE, all resident weights, including the inactive experts)
$W_{\mathrm{batch}}(n)$	Shared weight traffic for one iteration that routes $n$ tokens in total, evaluated at $n = b\rho$ in decode
$K_{\mathrm{alloc}}(L_{\mathrm{alloc}})$	KV/cache memory reserved per session, which controls concurrency
$K_{\mathrm{read}}(L_{\mathrm{read}})$	Private context traffic per output token, which controls decode cost
$\rho$	Tokens emitted per session per iteration ($\rho = 1$ ordinary, $\rho > 1$ speculative)

The split between $K_{\mathrm{alloc}}$ and $K_{\mathrm{read}}$ mirrors the split between the two context lengths. Allocation controls how many sessions fit, read controls how fast each one decodes. Note that $W_{\mathrm{batch}}$ takes the iteration’s total token routings as its argument, $n = b\rho$ for a decode iteration, for a reason the adapter section explains. Prefill will later feed the same function far larger token counts.

Memory-fit batch

The first gate is whether sessions fit. Load the model, reserve overhead, and divide the remainder by the per-session allocation.

$$b_{\mathrm{mem}}(L_{\mathrm{alloc}}) = \left\lfloor \frac{C - W_{\mathrm{resident}} - O}{K_{\mathrm{alloc}}(L_{\mathrm{alloc}})} \right\rfloor .$$

This is memory-fit concurrency only. It is necessary but not sufficient, and it is exactly the trap that makes a machine look like it can serve a hundred sessions when it cannot serve them usefully.

Interactive figure: how many sessions fit in memory as capacity and reserved context change. Enable JavaScript to explore it.

Aggregate and per-session ceilings

The per-token traffic is the shared weight sweep amortized over the emitted tokens, plus the private context read.

$$q_{\mathrm{KV}}(b, L_{\mathrm{read}}) = \frac{W_{\mathrm{batch}}(b\rho)}{b\,\rho} + K_{\mathrm{read}}(L_{\mathrm{read}}) .$$

Then the memory roofline $T \le R/q$ gives the aggregate ceiling at batch $b$,

$$T(b, L_{\mathrm{read}}) \le \frac{R}{q_{\mathrm{KV}}(b, L_{\mathrm{read}})},$$

and dividing by the batch gives the per-session rate,

$$r(b, L_{\mathrm{read}}) = \frac{T(b, L_{\mathrm{read}})}{b} = \frac{\rho R}{W_{\mathrm{batch}}(b\rho) + b\,\rho\,K_{\mathrm{read}}(L_{\mathrm{read}})} .$$

As $b$ grows, the aggregate $T$ rises but the per-session $r$ falls. That tension is the whole serving tradeoff, and it is why a fit-only bound is not enough.

Usable-batch correction

The fix is to refuse batches at which a session would crawl. Impose a per-session floor $r_\star$, the minimum useful tokens/s/session, and solve $r(b) \ge r_\star$ for $b$. Replacing the batch-dependent $W_{\mathrm{batch}}(b\rho)$ by its shared lower bound $W_{\mathrm{active}}$ keeps a closed form. Because $W_{\mathrm{batch}}(b\rho) \ge W_{\mathrm{active}}$, the substitution only weakens the condition, so the implication runs one way, and the closed form is a necessary condition on the admissible batch rather than a sufficient one.

$$r(b) \ge r_\star \quad\Longrightarrow\quad b \le \frac{\rho R / r_\star - W_{\mathrm{active}}}{\rho\,K_{\mathrm{read}}(L_{\mathrm{read}})} ,$$

which defines a rate-limited batch

$$b_{\mathrm{rate}}(L_{\mathrm{read}}, r_\star) = \left\lfloor \frac{\rho R / r_\star - W_{\mathrm{active}}}{\rho\,K_{\mathrm{read}}(L_{\mathrm{read}})} \right\rfloor .$$

The usable batch is whichever gate binds first.

$$b_{\mathrm{usable}} = \min\!\big(b_{\mathrm{mem}}(L_{\mathrm{alloc}}),\ b_{\mathrm{rate}}(L_{\mathrm{read}}, r_\star)\big).$$

Because $b_{\mathrm{rate}}$ comes from a necessary condition, $b_{\mathrm{usable}}$ is itself an upper bound on the truly admissible batch, and the calculator applies the exact floor test with the true $W_{\mathrm{batch}}(b\rho)$ in the next step. This is what stops the “hundred sessions” illusion. As context grows, $K_{\mathrm{read}}(L)$ grows, so $b_{\mathrm{rate}}$ falls quickly even while $b_{\mathrm{mem}}$ stays large. The KV slots fit while the useful rate does not.

Interactive figure: aggregate and per-session ceilings against batch size, with the memory gate and the per-session floor. Enable JavaScript to explore it.

The bound the calculator uses

Collecting the pieces, define the usable batch set as the fitting batches that also clear the floor,

$$\mathcal{B}(L_{\mathrm{alloc}}, L_{\mathrm{read}}, r_\star) = \left\{ b : 1 \le b \le b_{\mathrm{mem}}(L_{\mathrm{alloc}}),\ \frac{1}{b}\,\frac{R}{q_{\mathrm{KV}}(b, L_{\mathrm{read}})} \ge r_\star \right\},$$

and take the best aggregate over that set.

$$T_{\max}(L_{\mathrm{alloc}}, L_{\mathrm{read}}, r_\star) \le \max_{b \in \mathcal{B}}\ \frac{R}{\dfrac{W_{\mathrm{batch}}(b\rho)}{b\,\rho} + K_{\mathrm{read}}(L_{\mathrm{read}})} .$$

This is the KV-aware bound, the main practical formulation. In words, try every batch that fits, reject the ones too slow per session, and for the rest take bandwidth divided by bytes per output token, keeping the best.

The looser memory-power bound is its corollary, obtained as before by dropping the private term and using the largest fitting batch.

$$T_{\max} \le \rho\,\frac{D}{K_{\mathrm{alloc}}(L_{\mathrm{alloc}})\,W_{\mathrm{active}}}\left(1 - \frac{W_{\mathrm{resident}} + O}{C}\right), \qquad D = CR.$$

The two stand in a fixed relation, which is the main result of the derivation, written first by name and then in full.

$$\begin{aligned} T_{\max} \;&\le\; \text{KV-aware bound} \;\le\; \text{memory-power bound} \;\le\; \text{large-memory limit} \\ T_{\max} \;&\le\; \max_{b \in \mathcal{B}} \frac{R}{\dfrac{W_{\mathrm{batch}}(b\rho)}{b\rho} + K_{\mathrm{read}}(L_{\mathrm{read}})} \\ \;&\le\; \rho\,\frac{D}{K_{\mathrm{alloc}}(L_{\mathrm{alloc}})\,W_{\mathrm{active}}} \left(1 - \frac{W_{\mathrm{resident}} + O}{C}\right) \\ \;&\le\; \rho\,\frac{D}{K_{\mathrm{alloc}}(L_{\mathrm{alloc}})\,W_{\mathrm{active}}} \end{aligned}$$

The gap across these terms is the point of the whole derivation. The KV-aware line is the tight, practical bound. The memory-power line shows the memory system’s large theoretical capacity-bandwidth product, and the distance between them comes from private context traffic, expert diversity, and the per-session floor. The final term drops the resident-model factor as well, so the right-hand side is exactly the simplified $D/(KW)$ from the memory-power decode bound, now with $K = K_{\mathrm{alloc}}(L_{\mathrm{alloc}})$ and $W = W_{\mathrm{active}}$. It is the loosest, most optimistic reading, since the resident model and overhead always claim a real share of $C$.

Single session

Set $b = 1$ to recover the latency-style bound for one conversation. There is no batch to amortize the weight sweep over.

$$T_{1,\max} \le \frac{R}{W_{\mathrm{batch}}(\rho)/\rho + K_{\mathrm{read}}(L_{\mathrm{read}})},$$

and for ordinary decoding ($\rho = 1$) this is just $R / (W_{\mathrm{batch}}(1) + K_{\mathrm{read}}(L_{\mathrm{read}}))$. Capacity has dropped out except as the gate that decides whether the model fits at all, consistent with the earlier observation that bandwidth governs single-session speed.

Speculative decoding

Speculative decoding lets $\rho > 1$. A draft model proposes $\gamma$ tokens, the target verifies all of them in one forward pass, and $\rho$ is the expected number it accepts per session per iteration, bounded as $1 \le \rho \le \gamma + 1$. If the target accepts each draft token with probability $\alpha$, the expected count is $\rho = (1 - \alpha^{\gamma+1})/(1 - \alpha)$, so a measured acceptance rate gives a usable $\rho$. The formulas of the previous sections keep $\rho$ visible and are exact at $\rho = 1$, and this section gives $\rho > 1$ its honest accounting, which changes the earlier formulas in both directions.¹

The first change charges costs the earlier formulas skip. The draft must be resident, so its footprint and its own cache allocation join the capacity gate, $W_{\mathrm{resident}} + W_{\mathrm{draft,res}} + O + b,(K_{\mathrm{alloc}} + K_{\mathrm{draft,alloc}}) \le C$. The draft is also autoregressive, so proposing $\gamma$ tokens sweeps its weights $\gamma$ times and reads its own cache $\gamma$ times, and every iteration carries that traffic. The sweep of the target grows too, since all $b(\gamma+1)$ candidates route through the model whether or not they survive verification, so the MoE expert count takes $b(\gamma+1)$ routings rather than $b\rho$.

The second change runs the other way and removes an overcharge. Reading the formulas above at $\rho > 1$ charges $K_{\mathrm{read}}$ once per accepted token, so $\rho$ times per iteration. A verification pass does less work than that, because it scores all $\gamma + 1$ candidates against the session’s cache in one sweep, the same way a prefill chunk reads a prefix once for many tokens. The honest private charge is one read per iteration, or $K_{\mathrm{read}} / \rho$ per accepted token. The overcharge matters because it pushes the computed ceiling below what a clean speculative implementation can reach at long context, and a bound an implementation can beat has stopped being an upper bound.

Collecting both changes, the traffic per accepted token is

$$q_{\mathrm{spec}}(b) = \frac{W_{\mathrm{batch}}(b(\gamma+1)) + \gamma\, W_{\mathrm{draft}}}{b\,\rho} + \frac{K_{\mathrm{read}}(L_{\mathrm{read}}) + \gamma\, K_{\mathrm{draft,read}}}{\rho},$$

where $W_{\mathrm{draft}}$ is the draft’s weight sweep, shared across the batch like the target’s, and $K_{\mathrm{draft,read}}$ is the draft’s per-step cache read, private like the target’s. At $\gamma = 0$ there is no draft, $\rho = 1$, and the formula collapses to the ordinary $q_{\mathrm{KV}}(b)$. Whether speculation wins on a given workload comes down to the acceptance rate, since $\rho$ rises with it while every $\gamma$-proportional cost stays fixed. One more limit applies at large batches. A verification pass pushes $b(\gamma+1)$ tokens through the model at once, which is as much work as a small prefill chunk. The compute roofline of the prefill part below therefore applies to the pass, and the same crossover test decides whether memory or compute limits it.

The two bounds side by side

The gap between the memory-power bound and the KV-aware bound is easiest to see as memory traffic. Below, two copies of the same machine decode side by side. Each board is the machine’s usable memory, with the weights packed into an orange container of equal-sized cells and each session’s blue KV cells packed into a small container of its own.

Each decode iteration must move every byte that its accounting charges, at the same bandwidth on both machines, so the charged cells light up one by one and the board resets when the iteration completes. The left board charges only the shared weight cells, so it resets quickly and its token counter races ahead. The right board also charges the read cells of every session’s context, so its iterations stretch as the batch and the context grow. A real iteration takes milliseconds, so time runs in slow motion here.

Interactive animation: memory-power accounting and KV-aware accounting decoding side by side on the same machine. Enable JavaScript to watch it.

Both boards run on the same silicon at the same bandwidth, and only the bookkeeping differs between them. The left counter is the memory-power accounting at the chosen batch, and the right one is what the KV-aware bound admits once private context reads are charged.

The sliders show the two context lengths at work. The reserved context sets how many cells each session’s container holds and can push the machine past its capacity, so raising it eventually makes the containers stop fitting. The read context sets how many of those cells light up every iteration, and the longer it gets the smaller the weights’ share of each iteration becomes. It follows the reservation at an adjustable fraction, 90 percent by default, with the unread rest capped at 32k, and the sliders for all of this sit under Advanced. Growing the model itself slows both boards down in step, while the orange container eats the room the blue ones need.

The full memory-power bound goes one step further than the left board. It grows the batch until memory is completely full of KV cells, which is exactly what the default maximize batch mode does, and the line under the left board reports that number. Shrinking the reserved context makes it explode.

At a 4k reservation about a hundred sessions fit and the bound climbs past 4,000 tok/s on this toy machine, and at a 1k reservation it would pass 17,000. Those numbers are true ceilings and useless forecasts at the same time. What stops a real machine long before then is reading each session’s context, which is exactly the traffic the right board charges.

Prefill

Everything so far bounds decode, which produces tokens one iteration at a time. Before decode can start, the machine has to push the whole prompt through the model once to build the KV cache. That phase is prefill, and its limits work differently. A decode iteration re-reads the weights for every token it emits, while prefill reads them once and serves every prompt token in the batch with that single sweep. The weight term that dominated the decode denominator nearly vanishes, and the limit moves to the compute roofline $F/a$ that the decode part dropped. The one new hardware number is $F$, the device’s sustained compute throughput in FLOP/s, and prefill quantities carry a pf subscript throughout this part.

Prefill workload

A prefill batch holds $b$ prompts of $L$ tokens each, so one iteration processes $bL$ input tokens. The prompt length $L$ is a third length next to the decode part’s two. $L_{\mathrm{read}}$ counted the average context a decode step reads back, while $L$ counts the tokens of the prompt being pushed through now. Write $T_{\mathrm{pf}}$ for aggregate prefill throughput in prompt tokens per second, matching the decode part’s use of $T$. If the batch takes $t_{\mathrm{pf}}$ seconds, then

$$T_{\mathrm{pf}} = \frac{bL}{t_{\mathrm{pf}}}, \qquad t_{\mathrm{pf}} = \frac{bL}{T_{\mathrm{pf}}},$$

and $t_{\mathrm{pf}}$ is the wait for the first token. Dividing $L$ by the aggregate rate would understate that wait by a factor of $b$, since all $b$ prompts share the machine, and the two readings agree only at $b = 1$. Queueing, tokenization, sampling, and the first decode step sit outside this model.

Prefill capacity gate

Everything the batch needs has to fit in memory before speed matters. The gate is the decode gate plus a workspace term,

$$W_{\mathrm{resident}} + O + b\,K_{\mathrm{alloc}}(L_{\mathrm{alloc}}) + \Omega_{\mathrm{pf}} \le C,$$

where $\Omega_{\mathrm{pf}}$ is the temporary activation memory prefill holds in flight. A runtime that grows the KV cache on demand instead of reserving it replaces $b,K_{\mathrm{alloc}}(L_{\mathrm{alloc}})$ with $b,K_{\mathrm{store}}(L)$, the cache actually written for an $L$-token prompt. For a conventional KV cache each prompt token writes

$$\kappa_{\mathrm{KV,w}} = 2\, N_{\mathrm{layers}}\, N_{\mathrm{kv}}\, d_h\, e_{\mathrm{KV,store}}$$

bytes, so $K_{\mathrm{store}}(L) = \kappa_{\mathrm{KV,w}}, L$. This is the per-token form of the KV-allocation formula in the adapter section below. The workspace has no decode counterpart, and this post models it only roughly, as a few activation-sized tensors over the tokens in flight. Leaving it out of the gate is ceiling-safe, because a looser gate only admits more batches and a maximum over more batches can only rise.

Traffic per prompt token

The decode part defined $W_{\mathrm{batch}}(n)$ as the shared weight traffic of an iteration with $n$ token routings. A one-shot prefill iteration routes $bL$ tokens, so its sweep is $W_{\mathrm{batch}}(bL)$, and each token’s share is $W_{\mathrm{batch}}(bL)/(bL)$. For a dense model the sweep is one full read of the active weights however large $bL$ grows, which is why prefill costs so much less memory traffic per token than decode. Adding the KV bytes each token writes and the kernel-dependent terms gives a minimal traffic model,

$$q_{\mathrm{pf}}(b, L) = \frac{W_{\mathrm{batch}}(bL)}{bL} + \kappa_{\mathrm{KV,w}} + q_{\mathrm{attn}}^{\mathrm{HBM}} + q_{\mathrm{act}},$$

where $q_{\mathrm{attn}}^{\mathrm{HBM}}$ is the attention kernel’s memory traffic and $q_{\mathrm{act}}$ the traffic from activations. The memory-side ceiling is $R / q_{\mathrm{pf}}$.

One warning applies to the attention term. In decode, every output token reads the whole cache, so the decode bound charges $K_{\mathrm{read}}(L_{\mathrm{read}})$ per token. Prefill does not work that way. All the tokens of one prompt share the same keys and values, and how many bytes actually move depends on the kernel. FlashAttention never writes the full $L \times L$ attention matrix out to memory, so this traffic stays small even though the amount of math stays quadratic. The safest loose ceiling drops the two kernel-dependent terms entirely, which only raises $R/q_{\mathrm{pf}}$ and keeps it a true ceiling.

Mixture-of-experts models need one correction. The adapter section below derives the expected distinct-expert count $m(n)$, which applies with $n = bL$, the token routings of the prefill iteration, where decode fed it $n = b\rho$. Prefill routes so many tokens at once that $m(bL)$ reaches all $E$ experts almost immediately, so an honest prefill sweep reads every expert,

$$W_{\mathrm{batch}}(bL) \approx e_w\, P_{\mathrm{total}},$$

whereas decode at modest batch sizes touches far fewer.

Compute per prompt token

Every prompt token has to be multiplied through the model’s weight matrices. Let $P_{\mathrm{lin,active}}$ count the parameters in those multiplications, meaning the attention projections and whichever MLP or expert matrices are active, but leaving out embedding lookups and any output projection that runs once per request. This is deliberately narrower than the MoE section’s $P_{\mathrm{active}}$, which was introduced for weight accounting rather than as a FLOP count. Counting one multiply-add as two FLOPs, the linear layers cost $A_{\mathrm{linear}} = 2, b L, P_{\mathrm{lin,active}}$ FLOPs per batch.

Attention adds the part that grows with context. In a causal sequence of length $L$, each token attends to itself and everything before it, which makes $L(L+1)/2$ query-key pairs per layer, and computing both $QK^\top$ and $PV$ costs about $4d$ FLOPs per pair, where $d = N_q d_h$ is the query-side hidden width and $N_q$ the query-head count. Summing over layers and adding a remainder term $A_{\mathrm{extra}}$ for softmax, normalization, rotary embeddings, routing, dequantization, and logits,

$$A_{\mathrm{pf}}(b, L) = 2 b L P_{\mathrm{lin,active}} + 2 b N_{\mathrm{layers}} d L (L+1) + A_{\mathrm{extra}},$$

and dividing by $bL$ gives FLOPs per prompt token,

$$a_{\mathrm{pf}}(L) = 2 P_{\mathrm{lin,active}} + 2 N_{\mathrm{layers}}\, d\, (L + 1) + a_{\mathrm{extra}}.$$

The compute-side ceiling is $F / a_{\mathrm{pf}}$. For short and medium prompts the weight multiplications dominate and the ceiling is roughly $F / (2 P_{\mathrm{lin,active}})$. For long prompts the attention work takes over, growing with $L$ per token and with $L^2$ per request, and layers that only attend to a sliding window stop growing once $L$ passes the window width $w$, the same split the adapter section below uses for $K_{\mathrm{read}}$.

One refinement matters when precisions mix, because $F$ is really several numbers on one chip. The linear layers can run at the weight precision’s tensor rate, while the attention matmuls run at the cache and activation precision, usually BF16, and the two families share the same tensor cores, so their times add rather than overlap. A tighter compute floor charges each term at its own rate,

$$t_{\mathrm{compute}} \ge \frac{A_{\mathrm{linear}}}{F_{\mathrm{linear}}} + \frac{A_{\mathrm{attention}}}{F_{\mathrm{attention}}}.$$

Charging everything at the fastest path’s rate stays a valid ceiling, only a looser one, and the gap opens exactly where attention dominates, at long context. The worked example below charges attention at the BF16 rate for this reason.

Combined ceiling and crossover

Putting the two sides together, prefill at a fixed batch and prompt length obeys

$$T_{\mathrm{pf}}(b, L) \le \min\left(\frac{R}{q_{\mathrm{pf}}(b, L)},\ \frac{F}{a_{\mathrm{pf}}(L)}\right),$$

for any batch that passes the capacity gate. In time terms, with total traffic $Q_{\mathrm{pf}} = bL,q_{\mathrm{pf}}$ and total compute $A_{\mathrm{pf}} = bL,a_{\mathrm{pf}}$, the batch cannot finish faster than the slower of its two jobs,

$$t_{\mathrm{pf},\min}(b, L) = \max\left(\frac{Q_{\mathrm{pf}}}{R},\ \frac{A_{\mathrm{pf}}}{F}\right).$$

Which side is the limit comes down to arithmetic intensity, the ratio of math to bytes. At short context, one weight sweep of $W_{\mathrm{batch}}(S)$ bytes serves $S$ tokens costing $2 P_{\mathrm{lin,active}}$ FLOPs each, so the two rooflines cross where the sweep time equals the math time,

$$S^\star = \frac{W_{\mathrm{batch}}(S^\star)\, F}{2\, P_{\mathrm{lin,active}}\, R}.$$

For a dense model the sweep is the active parameters themselves, so the parameters cancel and $S^\star \approx e_w F / (2R)$. At $F = 100$ TFLOP/s, $R = 800$ GB/s, and 4-bit weights that is about 30 tokens, so dense prefill turns compute-bound almost immediately. A sparse MoE does not cancel. Prefill sweeps every expert while each token only computes through its active parameters, so

$$S^\star \approx \frac{e_w F}{2R} \cdot \frac{P_{\mathrm{total}}}{P_{\mathrm{lin,active}}},$$

the dense figure times the sparsity ratio. A model with 14 times more total than active parameters crosses in the thousands of tokens instead of 30, and the worked example below lands between about 2,100 and 8,300 tokens per sweep. So compute is the usual limit for dense prefill, while a sparse MoE processed in modest chunks on bandwidth-poor hardware can stay memory-bound. Decode sits below the crossover in either case, which is why the decode part could ignore compute entirely.

One formula for both phases

Real servers rarely run a prompt in one piece. Let $\ell$ be how much of the prompt is already cached and $c$ how many new tokens this iteration processes, so one iteration handles $bc$ tokens and the weight sweep $W_{\mathrm{batch}}(bc)$, including the MoE expert count $m(bc)$, amortizes over all of them. Each new token attends to the $\ell$ cached tokens plus the new ones before it, which makes $c\ell + c(c+1)/2$ attention pairs per request, so the compute per new token generalizes to

$$a_{\mathrm{pf}}(c, \ell) \approx 2 P_{\mathrm{lin,active}} + 4 N_{\mathrm{layers}}\, d \left(\ell + \frac{c+1}{2}\right) + a_{\mathrm{extra}},$$

and an optimistic memory model to

$$q_{\mathrm{pf}}(b, c, \ell) \approx \frac{W_{\mathrm{batch}}(bc)}{bc} + \frac{K_{\mathrm{read}}(\ell)}{c} + \kappa_{\mathrm{KV,w}} + q_{\mathrm{attn}}^{\mathrm{HBM}} + q_{\mathrm{act}}.$$

The $K_{\mathrm{read}}(\ell)/c$ term assumes the cache of the old prefix is read about once per chunk and shared by the chunk’s $c$ new tokens. Averaged over a prompt split into equal chunks, the prefixes run $0, c, 2c, \ldots$, so the average prefix is $(L-c)/2$, while the compute average $\ell + (c+1)/2$ sums to $(L+1)/2$ however the prompt is divided, since chunking changes the rereads and the scheduling but never the total attention arithmetic.

The earlier one-shot forms are the case $c = L,\ \ell = 0$, where the reread term vanishes because there is no earlier chunk. At $c = 1$ the formula reproduces the decode bound of the first part, up to the KV write term $\kappa_{\mathrm{KV,w}}$, which the decode formulation drops as negligible next to $K_{\mathrm{read}}$ (a tenth of a megabyte against gigabytes at long context, and dropping it only loosens that ceiling). Speculation has no counterpart before the first output token, but its verification pass is itself a chunk, $c = \gamma + 1$ candidates of which $\rho$ survive, and that is where the speculative section’s division of the cache read by $\rho$ comes from. One formula covers both phases, and the chunk width $c$ is what separates them.

Chunk size belongs to the serving software, so a ceiling for a hardware and model pair must range over it. Within this model the answer is simple. Shrinking the chunk only adds rereads and shrinks the weight amortization, while the total attention arithmetic never changes, so the combined ceiling is largest at $c = L$ and the one-shot case is the true ceiling. A number computed at a fixed smaller chunk is a policy ceiling, the best a scheduler that caps its chunks at $c$ can reach, and a one-shot implementation may legitimately exceed it. What pushes real systems toward smaller chunks sits outside the bound, in the workspace $\Omega_{\mathrm{pf}}$ and in the waiting decode requests that Sarathi-Serve style scheduling lets cut in between chunks.

TTFT floor

The decode bound rejects batches where a session falls below $r_\star$ tokens per second. The prefill counterpart is a budget $\tau_\star$ on the wait for the first token. A setting is now a batch and a chunk size, and the settings that fit in memory and finish within the budget form the set

$$\mathcal{B}_{\mathrm{pf}} = \left\lbrace (b, c) :\ W_{\mathrm{resident}} + O + b K_{\mathrm{alloc}} + \Omega_{\mathrm{pf}} \le C,\quad t_{\mathrm{pf},\min}(b, c) \le \tau_\star \right\rbrace,$$

and the prefill ceiling at prompt length $L$ is the best throughput over the admissible settings,

$$T_{\mathrm{pf,max}}(L) \le \max_{(b, c) \in \mathcal{B}_{\mathrm{pf}}} \min\left(\frac{R}{q_{\mathrm{pf}}},\ \frac{F}{a_{\mathrm{pf}}}\right).$$

The budget is optional. Without one, the set is simply every setting that fits, and a caller who supplies $\tau_\star$ tightens it without hiding the unrestricted result. One-shot prefill maximizes throughput and minimizes the wait at the same time, so the maximum lands at $c = L$ whenever it fits. As with the decode floor, passing the test does not guarantee the budget is met, since scheduling and kernel overhead can still push the real wait past $\tau_\star$.

Ceilings across prompt lengths

$T_{\mathrm{pf,max}}(L)$ is a ceiling for one prompt length, and prefill has no length-independent speed. Short prompts amortize the weight sweep over fewer tokens, long prompts pay more attention work, and the fastest ceiling can sit at an intermediate length. A summary that avoids asking for $L$ takes the range over every prompt length that fits,

$$\mathcal{L} = \left\{L : 1 \le L \le L_{\mathrm{model,max}},\ W_{\mathrm{resident}} + O + K_{\mathrm{alloc}}(L) \le C\right\},$$

where $L_{\mathrm{model,max}}$ is the model’s context limit, and reports the two ends of the curve,

T_{\mathrm{pf,lo}} = \min_{L \in \mathcal{L}} T_{\mathrm{pf,max}}(L), \qquad T_{\mathrm{pf,hi}} = \max_{L \in \mathcal{L}} T_{\mathrm{pf,max}}(L). $$Both extremes must carry the prompt length where they occur, and$T_{\mathrm{pf,lo}}$ is still an upper ceiling on throughput at its length, never a lower bound on what real software achieves. ### Ordering of the prefill bounds The decode part arranged its bounds in one chain, each looser than the last, because every bound lived on the same resource and each step dropped a positive term from the same denominator. Prefill cannot be arranged that way. The memory ceiling $R/q_{\mathrm{pf}}$and the compute ceiling$F/a_{\mathrm{pf}}$are incomparable, since neither is smaller everywhere and which one is the limit flips at the crossover$S^\star$. What survives is a chain within each resource. Dropping the kernel-dependent traffic terms loosens the memory side toward a memory-power analogue,

T_{\mathrm{pf,MP}} \le \frac{L, D}{K_{\mathrm{alloc}}(L_{\mathrm{alloc}}), W_{\mathrm{active}}} \left(1 - \frac{W_{\mathrm{resident}} + O}{C}\right), $$

obtained by multiplying the capacity cap on $b$ with the sweep-rate cap $R/W_{\mathrm{active}}$, and dropping the attention term loosens the compute side toward $F/(2P_{\mathrm{lin,active}})$. The full ordering is

$$\text{actual} \le \min\!\left(\frac{R}{q_{\mathrm{pf}}},\ \frac{F}{a_{\mathrm{pf}}}\right) \le \frac{R}{q_{\mathrm{pf}}} \le T_{\mathrm{pf,MP}}$$

on the memory side and

$$\min\!\left(\frac{R}{q_{\mathrm{pf}}},\ \frac{F}{a_{\mathrm{pf}}}\right) \le \frac{F}{a_{\mathrm{pf}}} \le \frac{F}{2 P_{\mathrm{lin,active}}}$$

on the compute side, two independent chains hanging off a shared minimum, a lattice rather than a line. The practical consequence is that prefill has no single loosest bound to serve as an orientation line, and a loose bound from the wrong chain can sit orders of magnitude above the true ceiling. The memory-power analogue above is exactly such a bound. The decode part was the special case where one resource was known in advance to always win, and that knowledge is what collapsed the lattice into a chain.

Model adapters

The KV-aware bound is architecture-agnostic, and an architecture enters only through a short list of quantities. So each model family is captured by a small adapter that supplies them, and the prefill bound extends the same adapter with two compute entries described at the end of this section.

$$\mathrm{Adapter}(M) = \big[\, W_{\mathrm{resident}},\ W_{\mathrm{batch}}(n),\ K_{\mathrm{alloc}}(L_{\mathrm{alloc}}),\ K_{\mathrm{read}}(L_{\mathrm{read}}),\ \rho \,\big].$$

Three adapters cover the catalog. They handle dense transformers, mixture-of-experts, and hybrid/sliding/recurrent attention.

Dense transformers

For a dense model with $P_{\mathrm{total}}$ parameters at $e_w$ bytes each, all weights are touched every step, so the resident footprint and the per-step sweep coincide.

$$W_{\mathrm{resident}} = W_{\mathrm{batch}}(n) = P_{\mathrm{total}}\, e_w .$$

With $N_{\mathrm{layers}}$ layers, $N_{\mathrm{kv}}$ key/value heads, head dimension $d_h$, and KV byte widths $e_{\mathrm{KV,store}}$ and $e_{\mathrm{KV,read}}$, full-context attention reserves and reads

$$K_{\mathrm{alloc}}(L) = 2\, N_{\mathrm{layers}}\, N_{\mathrm{kv}}\, d_h\, e_{\mathrm{KV,store}}\, L, \qquad K_{\mathrm{read}}(L) \approx 2\, N_{\mathrm{layers}}\, N_{\mathrm{kv}}\, d_h\, e_{\mathrm{KV,read}}\, L,$$

where the factor $2$ counts keys and values. Weight precision and KV precision are independent settings. NVFP4 weights (NVIDIA’s 4-bit floating-point format) do not imply an NVFP4 cache, so $e_w$ and $e_{\mathrm{KV}}$ are tracked separately.

Mixture-of-experts

An MoE model is where the constant-$W_{\mathrm{batch}}$ assumption breaks, and fixing it is the single most important adapter correction. Let $P_{\mathrm{total}}$ be total parameters, $P_{\mathrm{active}}$ the active parameters per token, $E$ the number of routed experts, and $k$ the experts selected per token. Assuming uniformly sized routed experts, each routed expert holds

$$p_{\mathrm{expert}} = \frac{P_{\mathrm{total}} - P_{\mathrm{active}}}{E - k},$$

and the always-on remainder (dense trunk, shared experts, embeddings, attention) is

$$P_{\mathrm{fixed}} = P_{\mathrm{active}} - k\, p_{\mathrm{expert}} .$$

The naive model assumes a batch touches the same active experts every session, keeping $W_{\mathrm{batch}}$ constant. That is false. Independent sessions route to different experts, so a larger batch touches more distinct experts. With $n = b\rho$ token routings, each independently missing a given expert with probability $1 - k/E$, the expected number of distinct experts touched is

$$m(n) = E\left(1 - \left(1 - \frac{k}{E}\right)^{n}\right),$$

and the per-iteration shared traffic is the fixed part plus the touched experts.

$$W_{\mathrm{batch}}(n) = e_w\big[\, P_{\mathrm{fixed}} + p_{\mathrm{expert}}\, m(n) \,\big].$$

At $n = 1$ this reduces to the active-parameter footprint, and as $n \to \infty$ it saturates at all $E$ experts. This rising $W_{\mathrm{batch}}(n)$ is why MoE batching does not amortize for free, and why the MoE rows in the worked table reach their throughput optimum at modest batch sizes.

One caveat applies here. Every other traffic term in the bound is a deliberate under-estimate of real traffic, which is what makes $R/q$ a true ceiling. The expert count $m(n)$ is different. It is an expectation under independent, uniform routing rather than a lower bound. Real routing is correlated, since load-balancing losses push toward uniform while hot experts and topically similar sessions pull the other way, and correlated routing touches fewer distinct experts than the formula predicts. In that case the modeled traffic overstates the actual traffic, and the computed ceiling can sit below the true one. When a guaranteed ceiling is required, replace $m(n)$ by its minimum $k$, which replaces $W_{\mathrm{batch}}(n)$ by $W_{\mathrm{active}}$. The expectation form is the better estimate, the floor form is the safe bound.

Hybrid, sliding, and recurrent attention

Models with local or sliding-window attention, compressed or latent attention, or linear/recurrent state must not use the full-KV formula blindly, because their cache does not grow linearly in $L$ everywhere. Split both KV terms into global, local, and fixed-state parts.

$$K_{\bullet}(L) = K_{\mathrm{global},\bullet}(L) + K_{\mathrm{local},\bullet}(L) + K_{\mathrm{state},\bullet}, \qquad \bullet \in \{\mathrm{alloc}, \mathrm{read}\}.$$

A simple read approximation with sliding-window width $w$ is

$$K_{\mathrm{read}}(L) = \kappa_{\mathrm{global}}\, L + \kappa_{\mathrm{local}}\,\min(L, w) + K_{\mathrm{state}} .$$

Full-attention layers pay for the whole context, sliding-window layers pay only up to the window, and recurrent or latent state adds a fixed or slowly growing term. The same shape covers Gemma-style local/global attention and DeepSeek-style compressed/sparse attention, with only the coefficients changing.

Prefill adapter fields

The prefill bound reuses almost the whole decode adapter. $W_{\mathrm{resident}}$ and $K_{\mathrm{alloc}}$ serve the same capacity gate, $W_{\mathrm{batch}}(n)$ is the same function fed the iteration’s token count, $K_{\mathrm{read}}$ reappears as the chunked prefix reread, and the write coefficient $\kappa_{\mathrm{KV,w}}$ is derivable from the allocation fields. Two entries are new. The linear FLOP count $P_{\mathrm{lin,active}}$ prices the weight multiplications, and a layered attention-FLOP function prices the context work,

$$a_{\mathrm{attn}}(\bar L) = \alpha_{\mathrm{global}}\, \bar L + \alpha_{\mathrm{local}}\, \min(\bar L, w) + \alpha_{\mathrm{state}}, \qquad \bar L = \ell + \frac{c+1}{2},$$

with $\bar L$ the average attended context of the iteration. Each coefficient is $4d$ FLOPs per pair times the number of layers in that group, so this is the FLOP mirror of $K_{\mathrm{read}}(L) = \kappa_{\mathrm{global}} L + \kappa_{\mathrm{local}} \min(L, w) + K_{\mathrm{state}}$ above, and the same hybrid, sliding, and recurrent splits carry over unchanged. One decode caveat softens here. The MoE adapter had to distinguish the expected expert count from its worst-case floor, but prefill token counts saturate $m$ so quickly that the expectation and the worst case are the same number, all $E$ experts. The decode field with no prefill counterpart is $\rho$. The extended adapter is

$$\mathrm{Adapter}(M) = \big[\, W_{\mathrm{resident}},\ W_{\mathrm{batch}}(n),\ K_{\mathrm{alloc}},\ K_{\mathrm{read}},\ \rho,\ P_{\mathrm{lin,active}},\ a_{\mathrm{attn}}(\cdot) \,\big].$$

Calculator procedure

The bound is now ready to compute. Because the same computation runs for every hardware-and-model pair, it is worth stating once as a procedure.

The inputs are a hardware row $(C, R, F, O)$, with $F$ needed only for prefill, a model adapter $(W_{\mathrm{resident}}, W_{\mathrm{batch}}(\cdot), K_{\mathrm{alloc}}(\cdot), K_{\mathrm{read}}(\cdot), \rho, P_{\mathrm{lin,active}}, a_{\mathrm{attn}}(\cdot))$, and workload assumptions $(L_{\mathrm{alloc}}, L_{\mathrm{read}}, r_\star)$, plus a prompt length $L$ and an optional TTFT budget $\tau_\star$ for prefill.

1. Compute the resident margin $C - W_{\mathrm{resident}} - O$. If it is negative, the model does not fit, so stop.
2. Compute $K_{\mathrm{alloc}}(L_{\mathrm{alloc}})$ and the memory-fit batch $b_{\mathrm{mem}}$.
3. For each integer batch $1 \le b \le b_{\mathrm{mem}}$, compute $W_{\mathrm{batch}}(b\rho)$ and $q_{\mathrm{KV}}(b, L_{\mathrm{read}})$.
4. Compute the aggregate ceiling $R / q_{\mathrm{KV}}$ and the per-session ceiling $R / (b, q_{\mathrm{KV}})$ at each batch.
5. Keep the batches whose per-session ceiling is at least $r_\star$.
6. Among the kept batches, choose the one with the largest aggregate ceiling, and report it as the KV-aware result with its batch as $b^\star$.
7. Separately compute the memory-power ceiling for orientation.

The output is a stack of gates, and the right phrasing depends on which gate bound.

State	Meaning
Resident fit	The model plus overhead fits in memory
Session fit	At least one reserved-context session fits
Floor fit	Some fitting batch clears $r_\star$
No floor	Sessions fit, but no batch clears $r_\star$

The common invalid reading is that fitting in memory implies serving usefully. A model can pass resident fit and session fit and still have an empty usable batch set, because every fitting batch is below the floor. The honest report for that case is “fits, but no batch satisfies the floor”, a distinct verdict from a true fit failure. Keeping the two apart is the reason the floor gate exists.

The prefill computation runs the same way over batch and chunk settings. For each $(b, c)$ that passes the capacity gate, compute the totals $Q_{\mathrm{pf}}$ and $A_{\mathrm{pf}}$, take $T_{\mathrm{pf}}(b, c) = \min(R,bL/Q_{\mathrm{pf}},\ F,bL/A_{\mathrm{pf}})$ and $t_{\mathrm{pf},\min} = bL/T_{\mathrm{pf}}$, and report the largest $T_{\mathrm{pf}}$ with its batch, chunk, both resource ceilings, and the latency floor. In practice the search over $c$ collapses, since the optimum sits at $c = L$ whenever the workspace fits, and smaller chunks only need evaluating as policy lines for chunk-capped schedulers. The central formula is

$$T_{\mathrm{pf,max}}(L) \le \max_{\text{setting fits}} \min\left(\frac{R}{q_{\mathrm{pf}}},\ \frac{F}{a_{\mathrm{pf}}}\right),$$

the prefill counterpart of the KV-aware decode bound, with a supplied $\tau_\star$ restricting the settings inside the maximum.

Worked examples

Now check the theory against real hardware. Consider two 128 GB machines, one bandwidth-rich and one bandwidth-poor, which isolate the effect of $R$ at fixed $C$.

Two machines

NVIDIA’s DGX Spark, a small desktop AI machine, carries 128 GB of LPDDR5x unified memory at 273 GB/s. An Apple M5 Max with a 40-core GPU reaches 614 GB/s and is configurable to 128 GB of unified memory. At equal capacity their memory powers are

Hardware	$C$	$R$	$D = CR$
DGX Spark	128 GB	273 GB/s	34,944 $\mathrm{GB}^2/\mathrm{s}$
Apple M5 Max 128GB	128 GB	614 GB/s	78,592 $\mathrm{GB}^2/\mathrm{s}$

Both bandwidth numbers are catalog figures rather than measured sustained rates, so the earlier caveat applies. If one machine sustains a larger share of its spec than the other, the comparison will make the other machine look better than it really is.

Three models

Three MoE models, modeled from their published cards as adapter parameters, without re-measurement.

• Qwen3.6-35B-A3B, with 35B total / 3B active parameters, $E = 256$ routed experts, $k = 8$ routed (plus one shared) per token, and weights quantized NVFP4.
• Gemma 4 26B-A4B-it, with 26B total / 4B active, $E = 128$, top-8 routing, hybrid local/global attention, and NVFP4 weights.
• DeepSeek V4 Flash (DS4), with 284B total / 13B active, $E = 256$ routed plus one shared, $k = 6$ per token, million-token context via compressed/sparse attention, and weights at a Q2-style mixed quantization.

All rows use the Our Models defaults, namely reserved context $L_{\mathrm{alloc}} = 100{,}000$, active context $L_{\mathrm{read}} = 32{,}000$, per-session floor $r_\star = 20$ tok/s/session, ordinary decoding $\rho = 1$, runtime overhead $O = 8$ GB, and the memory roofline only. The numbers are memory-side upper bounds from the simplified adapters, to be read as ceilings for comparing hardware.

Results

Hardware	Model	Single-session	$b^\star$	KV-aware aggregate	Memory-power ceiling
DGX Spark	Qwen3.6-35B-A3B	$\le$ 149 tok/s	17	$\le$ 345 tok/s	$\le$ 18.2k tok/s
DGX Spark	Gemma 4 26B-A4B-it	$\le$ 120 tok/s	16	$\le$ 333 tok/s	$\le$ 23.7k tok/s
DGX Spark	DeepSeek V4 Flash (Q2)	$\le$ 83 tok/s	7	$\le$ 154 tok/s	$\le$ 13.1k tok/s
Apple M5 Max 128GB	Qwen3.6-35B-A3B	$\le$ 336 tok/s	50	$\le$ 1,006 tok/s	$\le$ 41.0k tok/s
Apple M5 Max 128GB	Gemma 4 26B-A4B-it	$\le$ 271 tok/s	66	$\le$ 1,326 tok/s	$\le$ 53.3k tok/s
Apple M5 Max 128GB	DeepSeek V4 Flash (Q2)	$\le$ 188 tok/s	22	$\le$ 446 tok/s	$\le$ 29.5k tok/s

Two things stand out. First, with capacity held equal, the higher M5 Max bandwidth lifts the single-session ceilings in proportion to $R$, and the batched ceilings by even more, because the extra bandwidth also lets more sessions clear the per-session floor. The bandwidth-rich machine wins exactly where the theory says it should, in batched throughput. Second, the memory-power column sits one to two orders of magnitude above the KV-aware column. That gap is the cost of private context traffic and expert diversity, and showing it is the point of the derivation.

Forced concurrency

What if concurrency is fixed by policy rather than chosen at the floor-satisfying optimum? On DGX Spark, pushing past $b^\star$ buys aggregate throughput at the cost of per-session rate.

Model	Batch $b$	Aggregate ceiling	Per-session ceiling
Qwen3.6-35B-A3B	32	$\le$ 394 tok/s	$\le$ 12.3 tok/s/session
Qwen3.6-35B-A3B	64	$\le$ 478 tok/s	$\le$ 7.5 tok/s/session
Gemma 4 26B-A4B-it	32	$\le$ 430 tok/s	$\le$ 13.4 tok/s/session
Gemma 4 26B-A4B-it	64	$\le$ 575 tok/s	$\le$ 9.0 tok/s/session

This is the serving tradeoff in numbers. For DGX Spark under these assumptions, 32 and 64 sessions are too high if the goal is around 20 tok/s/session, exactly the regime the usable-batch correction is built to reject, and the reason $b^\star$ for these models settles near 16.

A sanity check against a real run

A reported DGX Spark run served Gemma at concurrency 16 at roughly 16 to 18 tok/s/session, an aggregate of $16 \times 16 = 256$ to $16 \times 18 = 288$ tok/s. The KV-aware aggregate ceiling for Gemma at this batch is $\le 333$ tok/s, so observed throughput is

$$\frac{256}{333} \approx 77\% \quad\text{to}\quad \frac{288}{333} \approx 86\%$$

of the simplified ceiling. That is close enough to suggest the implementation is near the memory-side roofline. It does not prove the quantization is optimal. The bound omits compute, scheduler behavior, kernel details, and exact cache traffic, and proving optimality would require profiler evidence of bandwidth saturation with no compute, scheduler, or CPU stalls. A bound this close simply means there is little memory headroom left to capture.

Prefill on real hardware

The prefill example uses the DGX Spark again, paired with Poolside’s Laguna S 2.1 NVFP4 checkpoint, because a real prefill benchmark exists for the pair. For $F$, NVIDIA quotes the GB10 chip at 1 PFLOP of FP4 with sparsity, which reads as 500 TFLOP/s of dense FP4 and 125 TFLOP/s of dense BF16. The attention math runs on the BF16 path regardless of weight precision, so it is charged at 125 TFLOP/s throughout, and only the linear term ranges over the 125-to-500 bracket, since it is not obvious how much of that math runs on the fast path.

The bound needs three things from the model, its sweep size, its linear FLOP count, and its KV write coefficient. They come from the model card. Laguna S 2.1 has 117.6B total and 8.5B active parameters, 48 layers of which 12 use full attention and 36 use a 512-token sliding window, 48 query heads of dimension 128 (so $d = 6144$), 256 routed experts with 10 picked per token, and an FP8 KV cache. From those numbers, the 71.9 GB NVFP4 checkpoint carries a swept weight payload of 71.28 GB, subtracting the embedding and output head from the active parameters leaves $P_{\mathrm{lin,active}} \approx 7.9$B, and the FP8 cache takes 98,304 bytes of writes per prefilled token.

The expert formula decides the memory side. The expected distinct experts $m(c)$ already reach 254 of 256 at a 128-token chunk, so any realistic chunk sweeps the full 71.28 GB, and at 273 GB/s the machine manages at most 3.8 sweeps per second. A scheduler that caps chunks at 2048 tokens is therefore held near $3.8 \times 2048 \approx 7{,}800$ tokens per second whatever the prompt length. One-shot prefill sweeps once per prompt instead and lifts the memory ceiling to 31k tok/s at an 8k prompt, 120k at 32k, and 425k at 128k. The crossover confirms the regime, since $S^\star$ for this pair sits between about 2,100 and 8,300 tokens per sweep depending on the compute bracket, so a 2048-token chunk is still memory-bound at the FP4 bracket while a one-shot prompt is far past the crossover.

The compute side uses the layered attention form with 12 full and 36 sliding-window layers. The ceiling, at batch 1 with one-shot prefill:

Prompt $L$	$a_{\mathrm{pf}}$ per token	Compute ceiling	Memory ceiling	Combined ceiling	TTFT floor
8,192	17.5 GFLOP	7.2k to 22.3k tok/s	31k tok/s	7.2k to 22.3k tok/s	0.4 to 1.1 s
32,768	21.1 GFLOP	5.9k to 13.5k tok/s	120k tok/s	5.9k to 13.5k tok/s	2.4 to 5.5 s
131,072	35.6 GFLOP	3.5k to 5.3k tok/s	425k tok/s	3.5k to 5.3k tok/s	25 to 37 s

The 2048-token-chunk policy is the same formula with $c$ pinned:

Prompt $L$	Memory ceiling	Combined ceiling	TTFT floor
8,192	7.8k tok/s	7.2k to 7.8k tok/s	1.0 to 1.1 s
32,768	7.8k tok/s	5.9k to 7.8k tok/s	4.2 to 5.5 s
131,072	7.6k tok/s	3.5k to 5.3k tok/s	25 to 37 s

The weight multiplications, 15.8 GFLOP per token, dominate the compute cost until the context gets long. At 128k the attention work reaches 19.8 GFLOP per token and overtakes them, and because it runs on the BF16 path it pulls even the high bracket down to 5.3k, where compute is the limit under either policy.

A measured run checks the bound. A vLLM NVFP4 benchmark on the GB10 prefilled an 8,192-token prompt at 2,348 tokens per second, a 3.49-second wait for the first token. The benchmark does not report its chunk setting, so it has to be read against both ceilings, and it lands at about 30% of the 2048-chunk policy ceiling and 11% to 33% of the unconstrained one. Decode on the same stack reached 77% to 86% of its bound, so prefill leaves far more of this machine idle, and headroom of that size points at the software, either at kernels sustaining far less than the conservative 125 TFLOP/s bracket or at a weight sweep missing full bandwidth.

Omitted rooflines

The memory and compute rooflines are now both explicit, and real throughput is still the minimum over further limits,

$$T_{\max} \le \min\big(T_{\mathrm{memory}},\ T_{\mathrm{compute}},\ T_{\mathrm{kernel}},\ T_{\mathrm{scheduler}},\ T_{\mathrm{interconnect}}\big),$$

where the terms this post computes can be undercut by dequantization kernels, attention kernels and KV layout, prefill/decode phase mixing, scheduler overhead and request churn, CPU and PCIe involvement, multi-GPU communication, allocator fragmentation, thermal and power limits, tokenization and sampling, speculative rejection rates, and prefix-cache hit rates. Each belongs as its own limit term. The two derived rooflines remain useful because they make the first unavoidable ceilings explicit and cheap to compute.

The compute roofline also carries a weakness the memory one does not. Every simplification in the memory bound kept it a true ceiling, while a ceiling computed from spec-sheet peak FLOP/s is far too high, because real kernels on quantized weights often reach less than half of peak and the fraction changes with kernel and precision. For bandwidth this post could note the sustained-versus-spec gap and move on. For $F$ it decides whether the number means anything, and measuring believable sustained FLOP/s across consumer devices is the hard open problem of the prefill extension. Apple silicon illustrates it, since Apple publishes no comparable dense BF16 or FP16 GPU rate, and Neural Engine TOPS or estimates from GPU core counts are no substitute.

One recurring caution applies to models with recurrent or linear state. A model with tiny fixed state and tiny private read traffic produces an enormous memory-side aggregate at high concurrency, because almost nothing in the denominator grows with the batch. That is precisely the signal that compute, kernel, scheduler, and recurrent-state details must be added before the aggregate number is treated as realistic. The memory bound describes the best case the hardware allows, and reaching it is the implementation’s job.

Cheat sheet

This section collects the whole formulation in one place, so it can be read on its own. A machine is four numbers, a model adapter is five decode quantities plus two prefill ones, and the workload adds its assumptions.

Symbol	Meaning
$C$, $R$, $O$	Usable memory capacity, sustained memory bandwidth, runtime overhead
$W_{\mathrm{resident}}$	Full resident weight footprint, which must fit in memory
$W_{\mathrm{batch}}(n)$	Shared weight traffic for an iteration of $n$ token routings, $n = b\rho$ in decode; equal to $W_{\mathrm{resident}}$ for dense models and growing with $n$ for MoE
$W_{\mathrm{active}}$	Active-weight sweep size, the lower bound $W_{\mathrm{batch}}(1)$
$K_{\mathrm{alloc}}(L_{\mathrm{alloc}})$	KV memory reserved per session
$K_{\mathrm{read}}(L_{\mathrm{read}})$	Private context traffic per output token
$\rho$	Tokens emitted per session per iteration, one for ordinary decoding
$\gamma$, $W_{\mathrm{draft}}$, $K_{\mathrm{draft,read}}$, $K_{\mathrm{draft,alloc}}$	Draft length, draft weight sweep, draft per-step cache read, and draft cache allocation for speculative decoding
$L_{\mathrm{alloc}}$, $L_{\mathrm{read}}$	Reserved and average read context, $L_{\mathrm{read}} \le L_{\mathrm{alloc}}$
$r_\star$	Minimum useful tokens/s per session
$F$	Sustained compute throughput in FLOP/s, used only by prefill
$L$, $c$, $\ell$	Prompt length, chunk width, and cached prefix length in prefill
$\kappa_{\mathrm{KV,w}}$	KV bytes written per prompt token
$P_{\mathrm{lin,active}}$, $d = N_q d_h$	Active linear parameters, and the query-side hidden width
$N_{\mathrm{layers}}$	Transformer layer count
$\tau_\star$	Optional TTFT budget for the prefill floor

Everything descends from the two rooflines, $T_{\max} \le \min(R/q,\ F/a)$ for a token costing $q$ bytes of traffic and $a$ FLOPs. Decode is memory-bound, so its half of the sheet keeps only

$$T_{\max} \le \frac{R}{q} .$$

The first gate is whether sessions fit. Load the weights, reserve the overhead, and divide what is left by the per-session KV allocation.

$$b_{\mathrm{mem}}(L_{\mathrm{alloc}}) = \left\lfloor \frac{C - W_{\mathrm{resident}} - O}{K_{\mathrm{alloc}}(L_{\mathrm{alloc}})} \right\rfloor .$$

Per-token traffic is the shared weight sweep amortized over the batch plus the private context read, which no batch size amortizes.

$$q(b, L_{\mathrm{read}}) = \frac{W_{\mathrm{batch}}(b\rho)}{b\,\rho} + K_{\mathrm{read}}(L_{\mathrm{read}}) .$$

The roofline gives the aggregate ceiling at batch $b$, and dividing by the batch gives the per-session rate. The aggregate rises with $b$ while the per-session rate falls.

$$T(b) \le \frac{R}{q(b, L_{\mathrm{read}})}, \qquad r(b) = \frac{T(b)}{b} = \frac{\rho R}{W_{\mathrm{batch}}(b\rho) + b\,\rho\,K_{\mathrm{read}}(L_{\mathrm{read}})} .$$

Imposing the floor $r_\star$ rejects the batches where every session crawls, and the usable batch is whichever gate binds first.

$$b_{\mathrm{rate}}(L_{\mathrm{read}}, r_\star) = \left\lfloor \frac{\rho R / r_\star - W_{\mathrm{active}}}{\rho\,K_{\mathrm{read}}(L_{\mathrm{read}})} \right\rfloor, \qquad b_{\mathrm{usable}} = \min\big(b_{\mathrm{mem}},\ b_{\mathrm{rate}}\big) .$$

The usable batch set holds the batches that fit and clear the floor, and the KV-aware bound is the best aggregate over it.

$$\mathcal{B} = \left\{ b : 1 \le b \le b_{\mathrm{mem}},\ \frac{R}{b\, q(b, L_{\mathrm{read}})} \ge r_\star \right\}, \qquad T_{\max} \le \max_{b \in \mathcal{B}} \frac{R}{q(b, L_{\mathrm{read}})} .$$

Setting $b = 1$ in the same formula gives the single-session ceiling. Dropping the private context term and taking the largest fitting batch gives the looser memory-power ceiling,

$$T_{\max} \le \rho\,\frac{D}{K_{\mathrm{alloc}}(L_{\mathrm{alloc}})\,W_{\mathrm{active}}}\left(1 - \frac{W_{\mathrm{resident}} + O}{C}\right), \qquad D = CR ,$$

With speculation the private read divides by $\rho$, because one verification pass reads the cache for all accepted tokens, and the draft adds its own costs,

$$q_{\mathrm{spec}}(b) = \frac{W_{\mathrm{batch}}(b(\gamma+1)) + \gamma\, W_{\mathrm{draft}}}{b\,\rho} + \frac{K_{\mathrm{read}}(L_{\mathrm{read}}) + \gamma\, K_{\mathrm{draft,read}}}{\rho},$$

with the draft’s residency and cache joining the memory gate. For ordinary decoding the three levels always order the same way.

$$\text{actual throughput} \;\le\; \text{KV-aware bound} \;\le\; \text{memory-power bound} .$$

Prefill runs on the same adapter plus $F$. Per prompt token, an iteration of chunk width $c$ over a cached prefix $\ell$ moves

$$q_{\mathrm{pf}}(b, c, \ell) \approx \frac{W_{\mathrm{batch}}(bc)}{bc} + \frac{K_{\mathrm{read}}(\ell)}{c} + \kappa_{\mathrm{KV,w}}$$

bytes of memory traffic and spends

$$a_{\mathrm{pf}}(c, \ell) \approx 2 P_{\mathrm{lin,active}} + 4 N_{\mathrm{layers}}\, d \left(\ell + \frac{c+1}{2}\right)$$

FLOPs of compute, with attention FLOPs charged at the BF16 rate when the weight path runs faster. The ceiling at prompt length $L$ maximizes the lower of the two rooflines over settings that pass the same memory gate,

$$T_{\mathrm{pf,max}}(L) \le \max_{\text{setting fits}} \min\left(\frac{R}{q_{\mathrm{pf}}},\ \frac{F}{a_{\mathrm{pf}}}\right),$$

with the maximum at one-shot $c = L$, fixed-chunk evaluations serving as policy ceilings, and a supplied TTFT budget $\tau_\star$ restricting the settings through the wait $t_{\mathrm{pf}} = bL/T_{\mathrm{pf}}$. Decode is the $c = 1$ case of the same formulas, the rooflines cross near $S^\star = W_{\mathrm{batch}}(S^\star) F / (2 P_{\mathrm{lin,active}} R)$ tokens per sweep, and a range over prompt lengths reports $T_{\mathrm{pf,lo}}$ and $T_{\mathrm{pf,hi}}$ with the length at each extreme. Unlike decode, prefill’s loosened bounds form two chains hanging off the shared minimum, one per resource, so no single orientation line exists.

Every number these formulas produce is an upper bound built from memory capacity, memory bandwidth, and for prefill compute throughput. Real implementations land below it, and kernel quality, software overhead, and interconnects can only lower the ceiling further.

When reporting a result, always state the assumptions that move it, namely $L_{\mathrm{alloc}}$, $L_{\mathrm{read}}$, $\rho$, $r_\star$, the weight precision, and the KV-cache precision or attention adapter, and for prefill also the prompt length, the chunk mode, and which $F$ the compute side charged. Without them, a single tok/s number is not reproducible.

To see these bounds computed live for hundreds of audited model profiles against a catalog of local hardware, try Our Models. An independent implementation of the decode bound runs at the Localmaxxing decode calculator.