Master Local AI
  1. 0Load
  2. 1You type
  3. 2Tokenize
  4. 3To the GPU
  5. 4Prefill
  6. 5Decode
  7. 6Show text
Can it run?

Chapter 9 · Builder

Configuration: Offload, Context and the Knobs That Matter

Layers in RAM cut speed. Use a smaller quant first.

  1. 0Loadonce: file → VRAM
  2. 1You typetext in the app
  3. 2TokenizeCPU: text → IDs
  4. 3To the GPUIDs cross PCIe
  5. 4Prefillwhole prompt, one pass
  6. 5Decoderead all weights → 1 token
  7. 6Show textID → word · repeat 5

1Watch

Now

Press a step on the right, or change any control under the machine.

Readout

Key to the machine

Layers are slabs. The big tank is VRAM; the small tank is RAM. Every token reads every slab, so a few slabs behind the thin pipe slow the turbine.
How the numbers work

When a model does not fit in VRAM, llama.cpp keeps -ngl of its layers on the GPU and runs the rest from system RAM on the CPU. Every token still reads every layer, the GPU's share at about 1,792 GB/s on an RTX 5090 and the CPU's share at about 90 GB/s from dual-channel DDR5, one after the other, so the times add: tok/s = 1 ÷ (GPU GB ÷ GPU bandwidth + CPU GB ÷ RAM bandwidth), here at 70% of peak and with each layer's KV cache read where the layer lives. Moving just a few layers to RAM can cost most of the speed. Longer context grows the KV cache, which leaves room for fewer layers. DGX Spark and Macs keep one unified pool, so there is no split, only lower bandwidth. Both sight glasses stack one slab per layer (its weights plus its KV cache) at one scale (GB per unit of height), so a layer that leaves one glass lands in the other at the same thickness. The pipes are drawn about 3 to 1 so both stay readable; the real bandwidth ratio is about 20 to 1, printed on each pipe's tag. The flow meters show average GB/s on a log scale, and the dashes move at that rate divided by the pipe's cross-section.

  • Tanks = VRAM (big), RAM (small)
  • Slab = one layer · orange = in RAM
  • Lever = layers on GPU (-ngl)
Speed
Open full screen ↗

2Try this

Then change anything yourself.

3Read on ↓

Offload math

Every layer in RAM slows every token. A fraction f of the active bytes B sits in VRAM (bandwidth \text{BW}_g). The CPU reads the rest from system RAM (\text{BW}_c):

t_\text{token} \approx \frac{f\,B}{\text{BW}_g} + \frac{(1-f)\,B}{\text{BW}_c}

With 1,792 GB/s against ~90 GB/s (dual-channel DDR5), moving 10% of the weights to the CPU makes each token about 3× slower. Step 2 above shows this.

Offload is a capacity escape hatch. The better fixes are a smaller quant (step 5), a smaller context, an MoE model, or more VRAM.

The exception is MoE expert offload. llama.cpp keeps attention and shared weights on the GPU. It puts only the expert FFN tensors in system RAM (--n-cpu-moe N, or override-tensor rules). Each token touches few experts, so the CPU reads few bytes. A 100B-class MoE model then runs on a 24–32 GB card with fast RAM.

The knobs

Each knob trades one resource for another.

Knob llama.cpp Ollama vLLM Trade-off
GPU layers -ngl 99 num_gpu (auto) — (all GPU) speed vs capacity
Context size -c 16384 num_ctx --max-model-len KV memory vs window
KV precision -ctk q8_0 -ctv q8_0 OLLAMA_KV_CACHE_TYPE=q8_0 --kv-cache-dtype fp8 ½ KV memory, small quality cost
FlashAttention -fa (often default on) OLLAMA_FLASH_ATTENTION=1 default less memory, faster long prompts
Parallel sequences --parallel N (server) OLLAMA_NUM_PARALLEL --max-num-seqs throughput vs KV per user
Batch sizes (prefill) -b 2048 -ub 512 — --max-num-batched-tokens TTFT vs memory spikes
CPU threads -t N num_thread — matters only for CPU or offloaded work
Memory mapping --no-mmap, --mlock — — startup and paging behaviour
VRAM headroom — — --gpu-memory-utilization 0.9 KV capacity vs OOM risk
Multi-GPU split -ts 3,1, -sm layer/row auto --tensor-parallel-size see Chapter 10
Keep loaded — OLLAMA_KEEP_ALIVE — cold-start TTFT vs freed VRAM

Set the sampling values from the model card. Qwen3, for example, publishes separate settings for thinking and non-thinking modes. Wrong values often make a local model look worse than the API.

Lab: benchmark with llama-bench

llama-bench is the standard way to compare settings. It measures prompt processing (pp) and token generation (tg), repeats each test, and reports mean ± standard deviation. The lab uses the GGUF file behind your smallest Ollama chat model.

Lab · runs on your own machine

uv run labs/09_llama_bench.pyDownload all labs (.zip)

Needs uv and, for the live parts, Ollama with a small chat model: ollama pull qwen3:0.6b.

Show the code
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
import _local as local

import shutil, subprocess

model = local.need_chat_model()
if model and shutil.which("llama-bench"):
    path = local.blob_path(model)
    cmd = ["llama-bench", "-m", str(path), "-p", "512", "-n", "128", "-ngl", "0,99", "-r", "3", "-o", "md"]
    print(" ".join(cmd))
    print(subprocess.run(cmd, capture_output=True, text=True, timeout=900).stdout)
elif model:
    print("llama-bench not found; install llama.cpp (brew install llama.cpp) to run this lab")
Sample output on this course's Mac (Apple M4, 32 GB)
llama-bench -m /Users/mpl/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa -p 512 -n 128 -ngl 0,99 -r 3 -o md
| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| qwen3 0.6B Q4_K - Medium       | 492.75 MiB |   751.63 M | BLAS,MTL   |       4 |           pp512 |        159.59 ± 3.23 |
| qwen3 0.6B Q4_K - Medium       | 492.75 MiB |   751.63 M | BLAS,MTL   |       4 |           tg128 |         76.09 ± 1.47 |
| qwen3 0.6B Q4_K - Medium       | 492.75 MiB |   751.63 M | BLAS,MTL   |       4 |           pp512 |      2225.41 ± 83.14 |
| qwen3 0.6B Q4_K - Medium       | 492.75 MiB |   751.63 M | BLAS,MTL   |       4 |           tg128 |         95.84 ± 2.33 |

build: 7fe450e19 (11146)

What to notice: compare -ngl 0 (all CPU) with -ngl 99 (all GPU). On a unified-memory Mac the gap is small, because both sides share one memory bus. On an RTX PC the tg gap is typically a factor of 10.

Explain it
  1. "A 12 GB card runs a 14B model at 4 tok/s. Why?" The Q4_K_M weights (about 8.5 GB) plus a large default context pushed layers to the CPU. Lower the context, use a q8_0 KV cache, or check ollama ps.
  2. "When is CPU offload a good idea?" For batch work that tolerates latency, and for MoE expert offload.
  3. "What makes a benchmark trustworthy?" A fixed model file and quant, a stated context and batch, warm-up, repeats with variance, and version numbers. Report both pp and tg.

Exercises
  1. Use the offload formula to find the GPU share where Qwen3 32B on an RTX 5070 Ti drops below 10 tok/s.
  2. Run llama-bench with -ctk q8_0 -ctv q8_0 -c 8192 and compare it with the default.