Mastery: Measure, Optimize, Ship

What mastery looks like

Each skill has an artifact that proves it.

Benchmarking honestly

A benchmark is honest when anyone can repeat it.

1. Fix the model file and quant (by hash), the prompt set, the context, the output length and the concurrency.
2. Record the hardware, driver, CUDA, runtime version and power mode. Laptop power limits change results a lot.
3. Warm up. Repeat at least 5 times. Report the median and p90 (the slowest 1 run in 10 is worse), not the best run.
4. Report TTFT (time to first token) and decode tok/s separately, with peak VRAM.
5. Compare with the bandwidth bound (Chapter 3). A result far below ~50% means a wrong setting.

# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
import _local as local

import json, platform, statistics, subprocess, time

PROMPTS = [
    "Explain why memory bandwidth limits token generation on GPUs.",
    "Write a Python function that parses a GGUF header.",
    "Summarize the trade-offs between DGX Spark and an RTX 5090 for local LLMs.",
]

def benchmark(model, prompts=PROMPTS, repeats=5, num_predict=128):
    local.generate(model, "warm up", num_predict=8)
    runs = []
    for _ in range(repeats):
        for p in prompts:
            r = local.generate(model, p, num_predict=num_predict, temperature=0, seed=1)
            runs.append(r)
    ttft = [r["prefill_s"] * 1000 for r in runs]
    tps = [r["decode_tps"] for r in runs]
    env = {"os": platform.platform(), "machine": platform.machine(), "ollama": local._get("/api/version")["version"], "model": model,
           "digest": next(m["digest"][:12] for m in local.models() if m["name"] == model)}
    q = lambda xs, p: sorted(xs)[min(len(xs) - 1, int(p * len(xs)))]
    report = (f"| metric | median | p90 |\n|---|---:|---:|\n"
              f"| prefill (TTFT excl. queue), ms | {statistics.median(ttft):.0f} | {q(ttft, .9):.0f} |\n"
              f"| decode tok/s | {statistics.median(tps):.1f} | {q(tps, .1):.1f} (p10) |\n")
    return env, report

model = local.need_chat_model()
if model:
    env, report = benchmark(model)
    print(json.dumps(env, indent=1))
    print((report))

{
 "os": "macOS-26.6.2-arm64-arm-64bit",
 "machine": "arm64",
 "ollama": "0.33.0",
 "model": "qwen3:0.6b",
 "digest": "7df6b6e09427"
}
| metric | median | p90 |
|---|---:|---:|
| prefill (TTFT excl. queue), ms | 26 | 39 |
| decode tok/s | 92.9 | 68.3 (p10) |

A ship plan (template)

A ship plan gives every target a number.

Project:            <app / tool / model>            Type: model | runtime | agent | app
Goal:               e.g. "ship offline mode for 2M Windows users by Q3"
Platform goal:      e.g. "use the GPU-optimized path by default; test on new drivers on day 0"
Hardware targets:   RTX 5060 Ti 16GB … RTX 5090 · RTX 5090 Laptop · RTX PRO 6000 · DGX Spark
Success criteria:   TTFT p50 < 400 ms @ 2k prompt · ≥ 60 tok/s on RTX 5070 Ti · < 10 GB VRAM · 99.5% install success
Milestones:         M1 integration (week 2) · M2 perf parity vs baseline (week 5) · M3 launch readiness review (week 8)
Owners:             eng lead · GPU/runtime expert · product · docs · legal (model licence)
Risks:              model license, driver minimums, laptop power states, Windows VRAM contention
Launch assets:      blog with honest benchmarks, sample repo, demo video, driver/runtime notes, support FAQ
Feedback loop:      bi-weekly sync; blockers → reproducible tickets with impact data

Triage checklist for "it's slow on my GPU"

Ask six questions, in this order.

1. Does the GPU run the model? Check nvidia-smi and the runtime logs for a CPU fallback or the wrong CUDA build.
2. Does everything fit? Compare weights + KV + overhead with VRAM (Chapter 4). Look for offload splits.
3. Is TTFT or tok/s slow? Slow TTFT means prefill and compute. Slow tok/s means decode and bandwidth (Chapters 1 and 3).
4. Is the quant right for the runtime and GPU? Use Q4_K_M or Q8_0 for GGUF, and FP8 or NVFP4 on Blackwell with TensorRT-LLM.
5. Check the configuration: context size, KV type, FlashAttention, parallel slots and power mode (Chapter 9).
6. Is the runtime right for the job? (Chapter 7.)

Your point of view on local AI

Say the case for local AI in 60 seconds.

"AI becomes a feature of every application. More of it will run on the user's own device, for privacy, latency, cost and offline use. Memory and bandwidth limit what runs there. The best ecosystem makes the right model, in the right format, on the right runtime easy to use. Hybrid is the default: small models run locally, big models run in the cloud."