OFFLOAD RIG LAYERS ONE TOKEN

Tanks = VRAM (big), RAM (small)Slab = one layer · orange = in RAMLever = layers on GPU (-ngl)Layers are slabs. The big tank is VRAM; the small tank is RAM. Every token reads every slab, so a few slabs behind the thin pipe slow the turbine.When a model does not fit in VRAM, llama.cpp keeps -ngl of its layers on the GPU and runs the rest from system RAM on the CPU. Every token still reads every layer, the GPU's share at about 1,792 GB/s on an RTX 5090 and the CPU's share at about 90 GB/s from dual-channel DDR5, one after the other, so the times add: tok/s = 1 ÷ (GPU GB ÷ GPU bandwidth + CPU GB ÷ RAM bandwidth), here at 70% of peak and with each layer's KV cache read where the layer lives. Moving just a few layers to RAM can cost most of the speed. Longer context grows the KV cache, which leaves room for fewer layers. DGX Spark and Macs keep one unified pool, so there is no split, only lower bandwidth. Both sight glasses stack one slab per layer (its weights plus its KV cache) at one scale (GB per unit of height), so a layer that leaves one glass lands in the other at the same thickness. The pipes are drawn about 3 to 1 so both stay readable; the real bandwidth ratio is about 20 to 1, printed on each pipe's tag. The flow meters show average GB/s on a log scale, and the dashes move at that rate divided by the pipe's cross-section.