64-VALVE FILLERWAITING0REPLIES0PER USERTOTAL0tok/s0tok/s1 weight read feeds 12 bottles

Bottle = one request’s replyGreen crates = KV cache in usePlugged pockets = no KV roomA bottle-filling carousel: each bottle is one request; every turn drops a token into all of them. Watch the green KV crates. When the memory is full, bottles wait.A decode step reads every active weight once, however many sequences ride in the batch, so a turn takes the same time with one bottle or forty: total tokens per second climb almost in step with users while each user's rate falls slowly, until the KV cache (the crates) runs out or the GPU's arithmetic becomes the limit. Static batching shuts the gate until the slowest bottle is full; continuous batching (vLLM, SGLang, TensorRT-LLM) drops a new request into any free pocket at the next step, and paged KV blocks with a shared 512-token system prompt (prefix caching) fit more pockets than reserving 1,472 tokens per slot. Each turn here stands for 64 decode steps, slowed down; the readout comes from a simulation with 1,024-token prompts and 64–448-token replies, step time = bytes read ÷ (bandwidth × 0.7) or FLOPs ÷ (dense FP16 × 0.5), whichever is longer, and TTFT includes queue time.