Static versus continuous batching
Static batch
waits for the slowest request ↑
Continuous
With continuous (in-flight) batching, a new request joins at the next decode step as soon as any sequence finishes. Since decode is memory bound, ten users cost little more bandwidth per step than one.
Paged KV cache: virtual memory for attention
request A
request B
request C
physical KV blocks in VRAM
PagedAttention stores each sequence's KV cache in fixed-size blocks anywhere in a shared pool, like virtual memory pages. No sequence has to reserve its maximum context up front, so many more fit at once, and identical prefixes can share blocks.
Batching is cheap until memory fills
One weight read serves every user in the batch. With B sequences:
t_\text{step} \approx \frac{W + B\cdot \text{KV}_\text{read}}{\eta \cdot \text{BW}},\qquad \text{throughput} = \frac{B}{t_\text{step}}
While the weights W dominate, throughput grows almost linearly with B. Three limits end this:
- KV memory. Each sequence needs its own cache. When VRAM is full, requests wait in a queue.
- KV bandwidth. At long contexts, the KV reads rival the weight reads.
- Compute. Past the roofline ridge (Chapter 3), the tensor cores reach their limit.
Step 2 above shows free batching. Step 4 shows the KV memory limit.
Five techniques
Each technique removes waste.
- Continuous batching: the scheduler admits and retires requests at every decode step. vLLM, SGLang, TensorRT-LLM and llama-server (
--parallel) use it. - PagedAttention (vLLM): the KV cache sits in fixed-size blocks, like virtual memory. No sequence reserves worst-case space, and sequences share blocks.
- Prefix caching: requests that start with the same tokens reuse cached KV blocks. They skip that part of prefill.
- Chunked prefill: the server splits a long prompt into chunks between the decode steps of other users. One 30k-token prompt then does not freeze the other streams.
- Speculative decoding: a small draft model proposes tokens. The target model checks them in one pass, so each weight read gives more tokens.
Static or continuous
Continuous batching gives more throughput and a lower TTFT at the same load. Slots never wait for the longest request. Step 3 above shows the static penalty.
Past saturation, both policies queue. Then add KV capacity (FP8 KV, paged allocation), add a second GPU, or limit admissions.
Metrics
Quote four metrics, and state the test setup with them.
| Metric | Meaning | Driven by |
|---|---|---|
| TTFT | queueing + prefill until the first token | compute, prompt length, prefix cache hits, scheduler |
| TPOT / ITL | time per output token or between tokens | bandwidth, batch size, KV length |
| Throughput | total output tokens/s across users | batch size, KV capacity |
| Goodput | requests/s that meet the SLO (e.g. TTFT < 1 s, TPOT < 50 ms) | all of the above |
State the model, quant, context, concurrency, hardware, driver and runtime version. Without them, nobody can compare the numbers.
Explain it
- "What is PagedAttention?" The server stores KV caches in fixed blocks addressed through a table, like OS virtual memory. Sequences reserve no worst-case memory, and they share common prefixes.
- "TTFT is high at low load. What helps?" Prefix caching for the repeated system prompt and tool schemas. Chunked prefill and shorter tool outputs also help.
- "Throughput or latency?" Check the SLO. Batch until per-user TPOT reaches the SLO limit; that point is the goodput optimum.
Exercises
- Add prefix caching to the simulator: a shared 1,500-token system prompt costs prefill once per slot.
- Find the arrival rate where median TTFT crosses 2 s for each policy, at 8 and 32 slots.