Why Memory Bandwidth Sets Your Tokens per Second

Where bandwidth comes from

Bandwidth is bus width times data rate.

Each 32-bit GDDR channel is one memory chip, so the chip count follows the bus width. Capacity and bandwidth are separate decisions. The RTX PRO 6000 Blackwell keeps the 5090's 512-bit, 1,792 GB/s bus, but denser chips on both sides of the board give it 96 GB.

Predicting tokens/s

Batch-1 decode reads the active weights and the KV cache for every token. The playground predicts speed for an 8B model at 4-bit across the NVIDIA lineup:

The roofline

Speed is capped by memory or by compute, whichever limit is lower.

• Arithmetic intensity: FLOPs performed per byte moved.
• Limit: \min(\text{peak compute},\ \text{intensity} \times \text{bandwidth}).
• Ridge point: peak FLOPs ÷ bandwidth. Below it the device is memory bound. Above it the device is compute bound.
• GPUs have ridges of hundreds of FLOPs per byte. Batch-1 decode sits at 2/b (1 FLOP/byte at FP16).
• Batching B sequences reuses each weight read B times. This moves the point right by B\times.

Lab: measure this machine's bandwidth

Time a copy to measure bandwidth, then predict and verify tokens/s. The copy reads and writes every byte once. The lab predicts decode speed for your chat model and compares it with Ollama's measured speed.

# /// script
# requires-python = ">=3.11"
# dependencies = ["numpy", "torch"]
# ///
import _local as local

import time, torch

def copy_bandwidth(device, gigabytes=1.0, repeats=10):
    n = int(gigabytes * 1e9 / 4)
    src = torch.empty(n, dtype=torch.float32, device=device).uniform_()
    dst = torch.empty_like(src)
    sync = {"cuda": torch.cuda.synchronize, "mps": torch.mps.synchronize}.get(device, lambda: None)
    dst.copy_(src); sync()
    start = time.perf_counter()
    for _ in range(repeats):
        dst.copy_(src)
    sync()
    return 2 * src.numel() * 4 * repeats / (time.perf_counter() - start) / 1e9  # read + write

device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
measured_bw = copy_bandwidth(device)
print(f"{device}: ≈ {measured_bw:.0f} GB/s achieved (spec for an M4 is 120 GB/s; NVIDIA GPUs list theirs on the chapter page)")

model = local.need_chat_model()
if model:
    info = next(m for m in local.models() if m["name"] == model)
    weights_gb = info["size"] / 1e9
    predicted = measured_bw / weights_gb
    local.generate(model, "Hi", num_predict=4)
    r = local.generate(model, "Explain memory bandwidth in one paragraph.", num_predict=128)
    print(f"{model}: {weights_gb:.2f} GB of weights")
    print(f"predicted ≈ {predicted:.0f} tok/s  (bandwidth ÷ bytes per token)")
    print(f"measured   {r['decode_tps']:.0f} tok/s  → {r['decode_tps']/predicted:.0%} of the bandwidth bound")

mps: ≈ 56 GB/s achieved (spec for an M4 is 120 GB/s; NVIDIA GPUs list theirs on the chapter page)
qwen3:0.6b: 0.52 GB of weights
predicted ≈ 107 tok/s  (bandwidth ÷ bytes per token)
measured   89 tok/s  → 83% of the bandwidth bound

What to notice:

• A copy reaches about 45–70% of the spec-sheet bandwidth (this M4 measured 56–71 of 120 GB/s). Predictions from the measured number are conservative.
• Real decode reaches roughly 50–85% of the bound at spec bandwidth. Launch overhead, dequantization, KV reads and sampling cause the gap, which is larger for tiny models.
• A result far below ~50% means a fault: the model spilled out of VRAM, the backend fell back to the CPU, or the build is a debug build.

Beating the wall without more bandwidth

Four methods raise tokens/s without more bandwidth.

• Quantization (Chapter 5): fewer bytes per weight give proportionally more tokens/s.
• Batching (Chapter 8): many users share each weight read.
• Speculative decoding: a small draft model (or extra heads, as in EAGLE or Medusa) proposes tokens. The big model verifies them all in one forward pass. When most guesses are accepted, you get 2–3× more tokens per weight read.
• MoE (Chapter 10): the model reads only the active experts' weights.