What Running a Model Actually Does

One request, end to end

A request passes through five steps.

1. Chat template. The app writes the messages in the model's exact prompt format. A wrong template is the most common cause of bad output.
2. Tokenize. Text becomes integer token IDs (about 3–4 English characters each).
3. Prefill. All prompt tokens go through the network in one pass. Each layer stores keys and values in the KV cache.
4. Decode. The model outputs logits (a score for every token). A sampler picks one token. The model reads the cache and repeats until a stop token.
5. Detokenize and stream. The text returns to the app.

Why the two phases differ

Prefill is limited by compute. Decode is limited by memory bandwidth. A dense model with N parameters does about 2N floating-point operations (FLOPs) per token.

• Prefill does about 2NP FLOPs for P prompt tokens. It reads the weights once for all of them, so tensor cores set the speed (TTFT).
• Decode reads every weight again for each token. That is only 2/b FLOPs per byte (b = bytes per weight): 1 at FP16, 4 at 4-bit. A GPU can do hundreds, so bandwidth is the limit:

Here \eta \approx 0.6\text{–}0.8 is the share of peak bandwidth that real kernels reach. Chapter 3 tests this formula on this machine.

The KV cache

The KV cache stores the keys and values of every earlier token. Without it, each new token would recompute all of them.

The 2 counts K and V. L is the number of layers. H_{kv} is the number of key/value heads. Grouped-query attention (GQA) uses fewer KV heads than query heads, which shrinks the cache.

Gemma 3 and gpt-oss use sliding-window layers. Those layers keep only the last window, so long context costs less.

At 128k tokens, Llama 3.1 8B's KV cache (≈17 GB at FP16) is about 3.5× its 4-bit weights. Long context is a memory problem first. Chapter 8 covers paged KV.

Sampling

The sampler turns probabilities into one token.

• Temperature sharpens or flattens the distribution.
• Top-k keeps the k most likely tokens.
• Top-p keeps the smallest set whose probabilities add up to p.

Lab: measure the two phases

Ollama reports prefill time and decode time for every request. These are TTFT and TPOT.

# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
import _local as local

model = local.need_chat_model()
if model:
    local.generate(model, "Hi", num_predict=4)  # warm-up: loads weights into memory
    for prompt_words in (20, 400, 1600):
        prompt = "Summarize this: " + " ".join(["The GPU streams weights from VRAM for every token."] * (prompt_words // 9))
        r = local.generate(model, prompt, num_predict=64)
        print(f"prompt {r['prompt_tokens']:>5} tok · prefill {r['prefill_s']*1000:7.1f} ms ({r['prefill_tps']:8.0f} tok/s) · "
              f"decode {r['output_tokens']} tok at {r['decode_tps']:6.1f} tok/s")

prompt    37 tok · prefill   124.3 ms (     298 tok/s) · decode 64 tok at   89.8 tok/s
prompt   499 tok · prefill   280.0 ms (    1782 tok/s) · decode 64 tok at   84.6 tok/s
prompt  1962 tok · prefill  1185.2 ms (    1655 tok/s) · decode 64 tok at   63.8 tok/s

What to notice: prefill tokens/s is 10–100× higher than decode tokens/s. Decode speed barely changes with prompt length until the KV cache gets large.