Master Local AI
  1. 0Load
  2. 1You type
  3. 2Tokenize
  4. 3To the GPU
  5. 4Prefill
  6. 5Decode
  7. 6Show text
Can it run?

Chapter 15 · Master

The Other End of the Ladder

Datacenter GPUs win on volume. Local wins on privacy and control.

  1. 0Loadonce: file → VRAM
  2. 1You typetext in the app
  3. 2TokenizeCPU: text → IDs
  4. 3To the GPUIDs cross PCIe
  5. 4Prefillwhole prompt, one pass
  6. 5Decoderead all weights → 1 token
  7. 6Show textID → word · repeat 5

1Watch

Now

Press a step on the right, or change any control under the machine.

Readout

Key to the machine

Two boards, one bench, one scale. HBM glass stores sit beside the die, and the rail to the next module is wide. Raise the batch: the cost per token falls.
How the numbers work

Both boards hold the same job: Llama 3.1 8B at FP8 (8.03 GB of weights) with an 8-bit KV cache (0.134 GB per user at 2,048 tokens). Decode is bandwidth-bound: one step reads the weights once plus every user's cache, so tokens per second = users × 70% of peak bandwidth ÷ bytes per step. Users beyond the free memory (95% of capacity minus weights and 1 GB) wait. Cost per token is a proxy, board power ÷ tokens per second (energy per token), shown relative to the RTX 5090 at batch 1. It ignores prices, compute limits, networking and idle time, so it is illustrative, not a bill. Specs come from NVIDIA datasheets in data/hardware.json; NVLink is the total for both directions, and a PCIe 5.0 x16 slot is 64 GB/s each way, 128 GB/s in total. One rail track in the figure stands for 128 GB/s.

  • Grey = model weights
  • Green = KV cache
  • Moving dashes = feed rate
Speed
Open full screen ↗

2Try this

Then change anything yourself.

3Read on ↓

HBM against GDDR

HBM is a glass store built beside the die, so it feeds the die much faster. GDDR chips sit in a ring on the card's board. HBM stacks sit on a silicon interposer (CoWoS) next to the die. The RTX 5090 feeds its die at 1,792 GB/s. The H100 feeds its die at 3,350 GB/s, and the B300 at 8,000 GB/s. Step 2 above shows both stores.

An SXM module screws onto a baseboard, and NVLink is Chapter 10's railway yard, scaled up. A GeForce card sits in a PCIe slot. That slot gives 128 GB/s in total, which is one track in the figure. The H100 has 900 GB/s of NVLink: seven tracks, joined by NVSwitch chips. NVL72 joins 72 GPUs into one big yard. Step 3 above shows the snap.

Batching economics

Chapter 8's batching, at scale, is why a small model can cost cents per million tokens. One read of the weights serves every user in the batch. A cloud server keeps many users in flight, so each token carries a small share of the read. In the figure, the energy per token falls about 20 times from batch 1 to batch 64. A local box serves one user, so it stays near batch 1. Step 4 above shows the fall.

The Transformer Engine

The Transformer Engine is Chapter 5's quantization built into the chip. Hopper GPUs run layers in FP8. Blackwell GPUs add FP4. Fewer bits per weight mean fewer bytes per read, so each read feeds more work.

A Grace CPU joined by NVLink-C2C is the reason DGX Spark and DGX Station exist. NVLink-C2C links the CPU and the GPU as one coherent memory pool. There is no PCIe copy between them. DGX Station keeps 748 GB in that pool (Chapter 10).

Power, cooling and ECC

Power sets the cooling. The RTX 5090 uses 575 W with air fans. A module uses 700 W to 1,400 W, and the GB300 needs liquid cold plates (step 5). ECC memory corrects flipped bits in long jobs, and datacenter parts and RTX PRO cards offer it.

Five generations

GPU Memory Bandwidth NVLink per GPU Board power
A100 SXM 80GB 80 GB 2,039 GB/s 600 GB/s 400 W
H100 SXM 80 GB 3,350 GB/s 900 GB/s 700 W
H200 SXM 141 GB 4,800 GB/s 900 GB/s 700 W
B200 180 GB 8,000 GB/s 1,800 GB/s 1,000 W
B300 (GB300) 288 GB 8,000 GB/s 1,800 GB/s 1,400 W

As of 2026-10, from the NVIDIA datasheets and pages named in data/hardware.json. NVLink counts both directions. The B200 and B300 per-GPU figures come from NVIDIA's eight-GPU system pages and its Blackwell Ultra post.

Explain it
  1. "Is a datacenter GPU much faster for one user?" No. One stream is bound by bandwidth, so the H100 is about twice as fast as the RTX 5090 (3,350 against 1,792 GB/s). The large gain comes from many users per weight read.
  2. "Why can't I link two RTX 5090 cards with NVLink?" GeForce RTX 50-series cards have no NVLink. Two cards talk over PCIe at 128 GB/s in total.
  3. "Why does local still make sense?" Your data stays on your machine, and one user rarely fills a batch of 64.

Exercises
  1. Raise the batch until the RTX 5090's red beacon lights, and name the batch and the memory that ran out.
  2. Compare the A100 and the H100 at batch 64, and say which cost falls more.
  3. Work out how many 128 GB/s tracks the B300 needs for its NVLink.