8 OF 128 SHOWNNEXTTOKENREPLY0 TOK STORED0 READ PER TOKEN0 8 OF 128 SHOWN8 OF 128 SHOWNCROSSINGS 0 / 1NEXTTOKENREPLY0 TOK STORED0 READ PER TOKEN0

Cart = one token · plates = GB readSheds = experts or layersGreen loads = KV cacheStakes and string = no roomRailway yard: each cart is a token, each shed holds weights. A mixture-of-experts model stores every shed but visits only a few. Compare STORED with READ PER TOKEN.A mixture-of-experts model stores many experts but routes each token to a few: Qwen3 30B-A3B keeps 128 experts per layer and uses 8 (30.5 B stored, 3.3 B read), gpt-oss-20b 21 B / 3.6 B, gpt-oss-120b 117 B / 5.1 B. Memory holds every shed, but a cart visits only the active ones, so decode speed follows the active weights: on a DGX Spark (128 GB at 273 GB/s) gpt-oss-120b in MXFP4 runs several times faster than a dense 32B. The drawing shows 8 sheds: for a dense model each stands for an eighth of its layers and all open for every token; for a mixture of experts it stands for one expert, and a token's share of them is rounded up to one. The readout uses the real ratios. The REPLY train keeps the last four tokens; the depot sign counts them all. Switch to two GPUs to join two yards by a bridge: a layer split crosses it once per token (fine over PCIe); with tensor parallelism each GPU holds half of every layer, so the two half-carts meet on the bridge after every attention and FFN block, two all-reduces per layer, which wants NVLink-class links once prompts are long; GeForce RTX 50 and RTX PRO 6000 cards have no NVLink. The CROSSINGS plate counts the real number per token. Decode tok/s = 0.7 × bandwidth ÷ bytes read per token, plus 18 µs per crossing.