Master Local AI
  1. 0Load
  2. 1You type
  3. 2Tokenize
  4. 3To the GPU
  5. 4Prefill
  6. 5Decode
  7. 6Show text
Can it run?

Chapter 10 · Expert

Big Models Locally: Experts, Many GPUs and Unified Memory

Experts decode like small models but need big memory.

  1. 0Loadonce: file → VRAM
  2. 1You typetext in the app
  3. 2TokenizeCPU: text → IDs
  4. 3To the GPUIDs cross PCIe
  5. 4Prefillwhole prompt, one pass
  6. 5Decoderead all weights → 1 token
  7. 6Show textID → word · repeat 5

1Watch

Now

Press a step on the right, or change any control under the machine.

Readout

Key to the machine

Railway yard: each cart is a token, each shed holds weights. A mixture-of-experts model stores every shed but visits only a few. Compare STORED with READ PER TOKEN.
How the numbers work

A mixture-of-experts model stores many experts but routes each token to a few: Qwen3 30B-A3B keeps 128 experts per layer and uses 8 (30.5 B stored, 3.3 B read), gpt-oss-20b 21 B / 3.6 B, gpt-oss-120b 117 B / 5.1 B. Memory holds every shed, but a cart visits only the active ones, so decode speed follows the active weights: on a DGX Spark (128 GB at 273 GB/s) gpt-oss-120b in MXFP4 runs several times faster than a dense 32B. The drawing shows 8 sheds: for a dense model each stands for an eighth of its layers and all open for every token; for a mixture of experts it stands for one expert, and a token's share of them is rounded up to one. The readout uses the real ratios. The REPLY train keeps the last four tokens; the depot sign counts them all. Switch to two GPUs to join two yards by a bridge: a layer split crosses it once per token (fine over PCIe); with tensor parallelism each GPU holds half of every layer, so the two half-carts meet on the bridge after every attention and FFN block, two all-reduces per layer, which wants NVLink-class links once prompts are long; GeForce RTX 50 and RTX PRO 6000 cards have no NVLink. The CROSSINGS plate counts the real number per token. Decode tok/s = 0.7 × bandwidth ÷ bytes read per token, plus 18 µs per crossing.

  • Cart = one token · plates = GB read
  • Sheds = experts or layers
  • Green loads = KV cache
  • Stakes and string = no room
Speed
Open full screen ↗

2Try this

Then change anything yourself.

3Read on ↓

MoE in one equation

A mixture-of-experts (MoE) model needs memory for all its experts, but speed for only the active ones. A router picks the top k of E expert MLPs per token.

\text{memory} \propto N_\text{total},\qquad \text{tokens/s} \propto \frac{\text{bandwidth}}{N_\text{active}\cdot b}
Model total B active B Q4/MXFP4 GB RTX 5090 (32 GB) RTX PRO 6000 (96 GB) DGX Spark (128 GB)
Qwen3 32B 32.8 32.8 19.9 ≈61 tok/s ≈61 tok/s ≈9 tok/s
Llama 3.1 70B 70.6 70.6 42.8 doesn't fit ≈29 tok/s ≈4 tok/s
Qwen3 30B-A3B (MoE) 30.5 3.3 18.5 ≈570 tok/s ≈570 tok/s ≈87 tok/s
gpt-oss-20b (MoE) 21 3.6 11.2 ≈638 tok/s ≈638 tok/s ≈97 tok/s
gpt-oss-120b (MoE) 117 5.1 62.2 doesn't fit ≈450 tok/s ≈69 tok/s

The DGX Spark story in one row: gpt-oss-120b fits no single GeForce card. It fits Spark with room to spare. Only ~5B parameters are active per token, so its bandwidth bound is roughly 65 tok/s. A dense 70B at Q4_K_M reaches about 4.5 tok/s on the same box. Real runtimes land below the bound. The ~15× gap is the point.

Batching an MoE model

MoE batching is less free than dense batching. Qwen3 30B-A3B reads 8 of 128 experts per token, about 6%. With uniform routing, a batch of B tokens touches 1-(1-8/128)^B of them. That is 12% at 2 tokens, 40% at 8 and 87% at 32. Real routers favour popular experts, which helps a little.

Two GPUs, two ways

Layer split adds capacity. Tensor parallelism adds speed but needs a fast link.

  • Layer (pipeline) split: GPU 0 holds layers 1–24 and GPU 1 holds layers 25–48. One small hidden-state vector crosses the link per token. This is the llama.cpp default (-sm layer). At batch 1 the GPUs take turns, so it gives no speedup.
  • Tensor parallel: each GPU holds half of every layer, so both work on every token. Each layer ends with two all-reduces (sum-and-share steps), about 2L per token. PCIe latency limits them; NVLink handles them well. vLLM and SGLang use --tensor-parallel-size.

GeForce RTX 50-series cards have no NVLink, so a layer split is the safe consumer default. Step 5 above shows the crossing count.

Unified memory: the third way

Unified memory trades bandwidth for capacity. DGX Spark (GB10), DGX Station and Apple silicon give the CPU and GPU one shared memory pool. There is no PCIe copy and no offload cliff. Spark's LPDDR5X gives 273 GB/s, against 1,792 for GDDR7. Chapter 4's tank rack shows the same trade-off.

Explain it
  1. "Why do MoE models matter locally?" They give big-model quality at small-model speed. Expert offload also lets them run partly on a consumer GPU plus system RAM.
  2. "Two RTX 5090s: tensor parallel or layer split?" For one user on llama.cpp, use a layer split (64 GB). For many users on vLLM, measure tensor parallelism, because all-reduce latency reduces the gain.
  3. "Who should buy a DGX Spark over a 5090?" A developer who needs 70B–200B MoE models and accepts lower speed on smaller ones.

Exercises
  1. Add an RTX 5090 Laptop and a DGX Station to the table, and list the models that now fit.
  2. Compare the per-token all-reduce traffic of Llama 3.1 70B on 2 GPUs with PCIe 5.0 (hidden size 8,192, BF16, 2 per layer, 80 layers).