Fig 15A datacenter GPU beside an RTX cardH100 vs RTX 5090 · 80 vs 32 GB · 3,350 vs 1,792 GB/s
Grey = model weightsGreen = KV cacheMoving dashes = feed rateTwo boards, one bench, one scale. HBM glass stores sit beside the die, and the rail to the next module is wide. Raise the batch: the cost per token falls.Both boards hold the same job: Llama 3.1 8B at FP8 (8.03 GB of weights) with an 8-bit KV cache (0.134 GB per user at 2,048 tokens). Decode is bandwidth-bound: one step reads the weights once plus every user's cache, so tokens per second = users × 70% of peak bandwidth ÷ bytes per step. Users beyond the free memory (95% of capacity minus weights and 1 GB) wait. Cost per token is a proxy, board power ÷ tokens per second (energy per token), shown relative to the RTX 5090 at batch 1. It ignores prices, compute limits, networking and idle time, so it is illustrative, not a bill. Specs come from NVIDIA datasheets in data/hardware.json; NVLink is the total for both directions, and a PCIe 5.0 x16 slot is 64 GB/s each way, 128 GB/s in total. One rail track in the figure stands for 128 GB/s.