Pick a model, a quantization and a device — see whether it fits and how fast it runs.
Precision of the model weights. Lower uses less VRAM but costs quality.
KV cache precision. Dominates VRAM at long context.
Hardware
Hardware configuration
One machine, or GPUs spread over several nodes you lay out yourself.
Select your GPU or configure a custom device.
Number of devices1
Tensor-parallel replicas. Comms overhead is included.
12481632
Lets the model exceed VRAM — at host-bandwidth speed.
Workload
Batch size 1
Sequences processed per step. Raises throughput, costs KV cache.
Sequence length1,024
Tokens per sequence (prompt + generation). Drives KV cache.
Concurrent users 1
Simultaneous requests. Multiplies KV cache, splits per-user speed.
Inference simulation
Press play to watch this configuration generate at its estimated rate.
Comfortable
0 GB
of 0 GB usable
Generation speed–
Per-token latency–
Time to first token–
Total throughput–
Bottleneck–
Memory allocation