Master Local AI
  1. 0Load
  2. 1You type
  3. 2Tokenize
  4. 3To the GPU
  5. 4Prefill
  6. 5Decode
  7. 6Show text
Can it run?

Chapter 0 · Novice

Start Here: The Local AI Stack

Memory decides if a model runs. Bandwidth decides how fast.

  1. 0Loadonce: file → VRAM
  2. 1You typetext in the app
  3. 2TokenizeCPU: text → IDs
  4. 3To the GPUIDs cross PCIe
  5. 4Prefillwhole prompt, one pass
  6. 5Decoderead all weights → 1 token
  7. 6Show textID → word · repeat 5

1Watch

Now

Press a step on the right, or change any control under the machine.

Readout

Key to the machine

A PC on an open bench. The model is a file of grey blocks. It moves once, from SSD to RAM to VRAM. Watch the chips: each word re-reads them.
How the numbers work

Pick a GPU and the ring of VRAM chips changes: a 5090 has sixteen on a 512-bit bus, a 5070 Ti eight on a 256-bit bus. One block is what one chip holds (doubled for very large files, so the file is at most 16 blocks). Weights load from the SSD at about 7 GB/s and cross PCIe 5.0 (64 GB/s) into VRAM once. Then every token streams them from VRAM into the GPU die. Your prompt crosses PCIe as 22 token IDs × 4 bytes = 88 bytes. A model that does not fit leaves surplus blocks in system RAM. Decode time is then the VRAM read plus the slower RAM read (≈ 90 GB/s). Fit check: weights plus an 8,192-token KV cache plus 1 GB overhead must fit in 95% of VRAM; decode assumes 70% of peak bandwidth.

  • Grey blocks = model weights
  • Tiles = your prompt and reply words
  • Green = KV cache on a VRAM chip
Speed
Open full screen ↗

2Try this

Then change anything yourself.

3Read on ↓

What local AI means

Local AI means the model runs on hardware you control. That hardware can be a laptop, an RTX desktop, a workstation or a DGX Spark. Hybrid AI adds a cloud model for the hardest requests.

Why local What it means in practice
Privacy and data control Documents, code and voice never leave the device. This is often the deciding factor for enterprises and creators.
Latency No network round trip, so a speech or agent loop can respond in tens of milliseconds.
Cost Hardware is already paid for, with no per-token bill, which matters for agents that call a model thousands of times.
Availability Works offline, on planes, in factories and air-gapped labs.
Control Pin a model version, fine-tune it, swap the runtime, and inspect everything.

Your hardware sets the limit. For this reason the hardware chapters (2–5) come before the software chapters (6–11).

The four numbers that decide everything

Four hardware numbers answer almost every local AI question.

  1. Memory capacity (GB). Does the model fit, with its KV cache?
  2. Memory bandwidth (GB/s). This sets tokens per second when the model writes.
  3. Compute (TFLOPS). This sets time to first token.
  4. Interconnect (PCIe, NVLink). This sets the cost of moving data between memories.
Device Memory Bandwidth Kind Note
GeForce RTX 4060 8 GB 272 GB/s discrete previous generation
GeForce RTX 4090 24 GB 1,008 GB/s discrete previous generation
GeForce RTX 5060 Ti 16GB 16 GB 448 GB/s discrete 128-bit GDDR7
GeForce RTX 5070 12 GB 672 GB/s discrete 192-bit GDDR7
GeForce RTX 5070 Ti 16 GB 896 GB/s discrete 256-bit GDDR7
GeForce RTX 5080 16 GB 960 GB/s discrete 256-bit GDDR7
GeForce RTX 5090 32 GB 1,792 GB/s discrete 512-bit GDDR7
GeForce RTX 5090 Laptop 24 GB 896 GB/s discrete 256-bit GDDR7, the top AI laptop part
RTX PRO 4000 Blackwell SFF 24 GB 432 GB/s discrete low-profile, 70 W, 192-bit GDDR7 ECC; the GPU in Ghost Core
RTX PRO 6000 Blackwell 96 GB 1,792 GB/s discrete workstation, 512-bit GDDR7 ECC
DGX Spark (GB10) 128 GB 273 GB/s unified Grace Blackwell desktop, LPDDR5x, 1 PFLOP sparse FP4
DGX Station (GB300) 252 GB 7,100 GB/s unified 252 GB HBM3e at 7.1 TB/s + 496 GB LPDDR5X at 396 GB/s, 748 GB coherent
Apple M4 (base) 32 GB 120 GB/s unified the Mac this course was written on
Apple M4 Max 128 GB 546 GB/s unified
H100 SXM 80 GB 3,350 GB/s datacenter HBM3, for contrast; see Chapter 15
Dual-channel DDR5-5600 96 GB 90 GB/s host a typical desktop CPU's RAM

How to read this table. An RTX 5090 and an RTX PRO 6000 Blackwell have the same bandwidth, so they give the same tokens/s on a model that fits both. The PRO card fits 3× more. DGX Spark holds 4× the 5090's memory but only about 1/6.5 of its bandwidth. It runs bigger models, more slowly. Chapter 4's tank rack shows this.

Check your machine

One lab shows what your machine can run. It looks for Ollama, llama.cpp and LM Studio.

Lab · runs on your own machine

uv run labs/00_check_machine.pyDownload all labs (.zip)

Needs uv and, for the live parts, Ollama with a small chat model: ollama pull qwen3:0.6b.

Show the code
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
import _local as local

import platform, shutil, subprocess

print("OS:", platform.platform())
print("Python:", platform.python_version())
for tool in ["ollama", "llama-server", "llama-cli", "lms", "nvidia-smi"]:
    print(f"{tool:>13}:", shutil.which(tool) or "not found")
try:
    import torch
    print("PyTorch", torch.__version__, "| CUDA:", torch.cuda.is_available(), "| MPS:", torch.backends.mps.is_available())
except ImportError:
    print("PyTorch not installed")
print("Installed Ollama models:", [m["name"] for m in local.models()])
Sample output on this course's Mac (Apple M4, 32 GB)
OS: macOS-26.6.2-arm64-arm-64bit
Python: 3.11.15
       ollama: /opt/homebrew/bin/ollama
 llama-server: /opt/homebrew/bin/llama-server
    llama-cli: /opt/homebrew/bin/llama-cli
          lms: /Users/mpl/.lmstudio/bin/lms
   nvidia-smi: not found
PyTorch not installed
Installed Ollama models: ['qwen3:0.6b', 'nomic-embed-text:latest']

The journey

Every chapter has the same parts. Each one has a figure, the math, a lab, playgrounds, an "Explain it" drill and exercises.

Stage Chapters You will be able to…
Novice 0–1 explain what happens between pressing Enter and seeing tokens
Practitioner 2–5 size any model for any GPU from first principles, and choose a quantization
Builder 6–9 pick formats and runtimes, serve many users, and tune a configuration
Expert 10–11 reason about MoE, multi-GPU, unified memory, RAG and agent workloads
Master 12–15 benchmark honestly, build your own personal AI box, route between local and cloud through one socket (LiteLLM), and compare it with datacenter GPUs

Explain it
  1. What is local AI? The model runs on your own hardware. You gain privacy, low latency and no per-token cost.
  2. What is the stack on an RTX PC? App, agent layer, runtime (Ollama, llama.cpp), model file (GGUF), kernels, CUDA, GPU with VRAM.
  3. Which spec matters most? VRAM capacity decides if a model runs. Bandwidth decides how fast it writes (Chapter 3).

Exercises
  1. Pick three devices from the table and name the largest 4-bit model for each.
  2. Place five local AI apps on the stack map.