1. 0Loadonce: file → VRAM
  2. 1You typetext in the app
  3. 2TokenizeCPU: text → IDs
  4. 3To the GPUIDs cross PCIe
  5. 4Prefillwhole prompt, one pass
  6. 5Decoderead all weights → 1 token
  7. 6Show textID → word · repeat 5
LOCAL AI BENCH1000 WCLOUD$KV0Start here1Inference2Why a GPU3Bandwidth4Will it fit?5Quantization6Model files7Runtimes8Serving9Offload10Big models11Beyond chat12Measure13Build your box14Hybrid15Datacenter

Grey blocks = model weightsTiles = your prompt and reply wordsGreen = KV cache on a VRAM chipA PC on an open bench. The model is a file of grey blocks. It moves once, from SSD to RAM to VRAM. Watch the chips: each word re-reads them.Pick a GPU and the ring of VRAM chips changes: a 5090 has sixteen on a 512-bit bus, a 5070 Ti eight on a 256-bit bus. One block is what one chip holds (doubled for very large files, so the file is at most 16 blocks). Weights load from the SSD at about 7 GB/s and cross PCIe 5.0 (64 GB/s) into VRAM once. Then every token streams them from VRAM into the GPU die. Your prompt crosses PCIe as 22 token IDs × 4 bytes = 88 bytes. A model that does not fit leaves surplus blocks in system RAM. Decode time is then the VRAM read plus the slower RAM read (≈ 90 GB/s). Fit check: weights plus an 8,192-token KV cache plus 1 GB overhead must fit in 95% of VRAM; decode assumes 70% of peak bandwidth.