AI Fit

How the numbers are made

Everything here is a roofline estimate. It is meant to get the order of magnitude and the trade-offs right (memory versus bandwidth versus compute versus network) before you rent or buy anything. Measure on the real stack before you commit; the assumptions panel lets you dial the model to what you measure.

Memory

Each accelerator may use util of its memory (vLLM's gpu_memory_utilization). The weights of one replica are split evenly over its TP × PP accelerators; a fixed runtime overhead plus a share of the weight shard is reserved on each. What remains is the KV cache pool of the replica:

pool = TP·PP · (mem·util − overhead) − weights
weights = params · (0.98 · bytes(precision) + 0.02 · 2)

Bytes per parameter: BF16 2, FP8 and INT8 1, INT4 0.52, FP4 0.53 (group scales included). KV bytes per token per layer: 2 · kv_heads · head_dim · bytes for standard and grouped-query attention, (d_c + d_rope) · bytes for multi-head latent attention, nothing for linear-attention or state-space layers (they keep a small constant state per sequence). Sliding-window and chunked layers hold at most window tokens. A shared prefix that is prefix-cached is stored once per replica and subtracted from every session; the pool also loses a fragmentation share to paged blocks. When TP exceeds the number of KV heads (or the model uses MLA) the KV cache is replicated across ranks unless data-parallel attention is on.

Sessions resident. With "kept in GPU memory" every user's session must fit; otherwise only in-flight requests (users × activity) do. Idle sessions parked in host RAM count against 0.8 · host RAM · nodes; the remainder re-prefill from cold.

Decode

One decode step for a batch of b requests reads the weights that are touched (for mixture-of-experts: 1 − (1 − k/E)^tokens of the routed experts, plus the dense part) and the KV cache of every request, then runs 2 · active_params + attention FLOPs per token. Pipeline stages re-read the weights per micro-batch, which is why PP is slower than TP inside a node:

t_bw   = (weights_read + b · kv(ctx) / PP) / (TP · bandwidth · bw_eff)
t_comp = b · (2 · active + attn_flops(ctx)) / (TP · PP · peak · mfu)
t_comm = 2 · layers · (latency + bytes / link_bw)   for TP > 1
step   = (max(t_bw, t_comp) + t_comm) · (1 + bubble · (PP − 1) / PP)

Attention FLOPs per token per layer are 2 · heads · (d_qk + d_v) · context, capped at the window for local layers and at top-k (plus a light indexer) for sparse attention; MLA uses the absorbed form with d_c + d_rope and d_c. Peak compute follows the weight precision when the accelerator has native FP8 / FP4 tensor cores; weight-only formats run in BF16. Collectives cross the inter-node network when TP is wider than a node.

Prefill and turns

A turn prefills only the tokens that are not cached: the new prompt when the session's KV was retained, the whole conversation minus the shared prefix when it was evicted. Prefill runs on the TP accelerators of one replica at mfu of peak; attention costs 2 · heads · (d_qk + d_v) · Σ min(position, window) over the new positions. Restoring a parked session adds kv_session / (TP · restore_bw). Time to first token is the warm case for the retention policy you chose; the cold case is shown next to it.

Prefill and decode share the accelerators (chunked prefill): with λ requests per second per replica, prefill takes a share f = λ · prefill_flops / (TP · PP · peak · mfu) of the time and every decode step stretches by 1 / (1 − f). λ = b / (TTFT + output · ITL) is solved as a fixed point. Above 85% the replica is called prefill-saturated. With prefill/decode disaggregation that work moves to a separate pool sized λ · prefill_flops / (peak · mfu) accelerators per replica.

Speculative decoding

With k draft tokens accepted at rate α, a step produces (1 − α^(k+1)) / (1 − α) tokens while its compute and collective traffic grow by k + 1 and memory traffic barely changes, so the gain is largest when decode is memory-bound and fades once compute saturates.

Capacity and the reverse search

For a layout, the memory limit is the number of sessions that fit the pool; the speed limit is the largest batch per replica whose per-user tokens/s still clear the target without prefill saturation, provided the time to first token is within its limit (a binary search over the load model). Concurrency is the lower of the two times the replica count; users are concurrency divided by activity. The context-at-load figure inverts the memory equation for the sessions you asked to keep resident.

"Best for this workload" tries every TP in {1…64} that divides the head count (across nodes only if you allow it) and every PP in {1…16}, filling the accelerator count with replicas, and keeps the layout with the most users, then the fewest idle accelerators, then the highest per-user speed. The workload planner does the same per candidate accelerator with an open count: it sizes one replica, multiplies until the users fit, and keeps the smallest count, then the lowest hourly price. Cost per million output tokens is hourly price over aggregate decode throughput.

What is not modeled

• Scheduling detail: queueing, batching windows, chunk sizes, CUDA graph capture limits, CPU overhead of the serving process.
• Expert-parallel all-to-all traffic (treated like the tensor-parallel all-reduce), load imbalance across experts, and MoE kernels that run below the BF16 roofline.
• Quality effects of quantization or RoPE scaling; the tool only flags contexts beyond the native window.
• Vision encoders, tool-call overheads, and multi-LoRA serving.

Catalog figures come from vendor data sheets and model cards as of mid-2026; entries marked ~ are announced or derived. Correct them in the catalog view if you have better numbers.

Glossary

Every dotted term in the app has one of these behind it: hover, focus or tap the term for the short version.