A course in running AI on hardware you own

Master Local AI

Every chapter explains one part of this machine.

1Watch

Now

Point at a number on the machine, or at a chapter on the right.

Readout

Key to the machine

A PC on an open bench. The model is a file of grey blocks. It moves once, from SSD to RAM to VRAM. Watch the chips: each word re-reads them.

How the numbers work

Pick a GPU and the ring of VRAM chips changes: a 5090 has sixteen on a 512-bit bus, a 5070 Ti eight on a 256-bit bus. One block is what one chip holds (doubled for very large files, so the file is at most 16 blocks). Weights load from the SSD at about 7 GB/s and cross PCIe 5.0 (64 GB/s) into VRAM once. Then every token streams them from VRAM into the GPU die. Your prompt crosses PCIe as 22 token IDs × 4 bytes = 88 bytes. A model that does not fit leaves surplus blocks in system RAM. Decode time is then the VRAM read plus the slower RAM read (≈ 90 GB/s). Fit check: weights plus an 8,192-token KV cache plus 1 GB overhead must fit in 95% of VRAM; decode assumes 70% of peak bandwidth.

Grey blocks = model weights
Tiles = your prompt and reply words
Green = KV cache on a VRAM chip

Speed

Open full screen ↗

2The course

Chapter 0Start Here: The Local AI Stack →

Novice

Practitioner

Builder

Expert

Master

All 16 chapters ↓Can it run on my machine? →

From the first prompt you type to designing your own personal AI box. Every chapter has a working machine you can push to its limits: change the GPU, the model, the context or the number of users, and watch what breaks.

The whole course follows one question, Why does my GPU need so much VRAM?, from the keyboard to the GPU and back.

Novice

Chapter 0Start Here: The Local AI StackMemory decides if a model runs. Bandwidth decides how fast.Chapter 1What Running a Model Actually DoesPrefill reads the prompt at once. Decode writes one token per pass.

Practitioner

Chapter 2Why a GPUA GPU does thousands of multiply-adds at the same time.Chapter 3Why Memory Bandwidth Sets Your Tokens per SecondEvery token reads the whole model. Bandwidth sets your speed.Chapter 4Will It Fit? Sizing Models for VRAMWeights + KV cache + overhead must fit in VRAM.Chapter 5Quantization: Fewer Bits, Same Model (Mostly)Fewer bits: smaller, faster, a little more error. FP16 shows none.

Builder

Chapter 6Model Files: GGUF, Safetensors, ONNX and EnginesA model is more than weights. The format decides who can load it.Chapter 7Inference Runtimes: One Socket, Many EnginesEvery engine offers the same socket. Pick by who you serve.Chapter 8Serving: Batching, Paged KV and Prefix CachingBatching is nearly free. KV memory sets the limit.Chapter 9Configuration: Offload, Context and the Knobs That MatterLayers in RAM cut speed. Use a smaller quant first.

Expert

Chapter 10Big Models Locally: Experts, Many GPUs and Unified MemoryExperts decode like small models but need big memory.Chapter 11Beyond Chat: RAG, Agents, Vision, Speech and ImagesRetrieved text is prompt text: more cards, slower first token.

Master

Chapter 12Mastery: Measure, Optimize, ShipMeasure honestly, find the bottleneck, then ship.Chapter 13Capstone: Build Your Own Personal AI BoxSize the box from the spec. Then prove it with a benchmark.Chapter 14Hybrid: One Socket, Local FirstLocal first. The cloud takes the overflow, never private data.Chapter 15The Other End of the LadderDatacenter GPUs win on volume. Local wins on privacy and control.

How each chapter works

Look, then try. The left panel says what happens now. The right panel’s Try this steps drive the machine for you, one limit at a time.
Read the idea and the math. Short, with numbers you can check.
Run the lab. Optional. uv run labs/… measures the same thing on your own machine.
Explain it. Say the answers out loud. If you can explain it, you know it.