Master Local AI
  1. 0Load
  2. 1You type
  3. 2Tokenize
  4. 3To the GPU
  5. 4Prefill
  6. 5Decode
  7. 6Show text
Can it run?

Chapter 13 · Master

Capstone: Build Your Own Personal AI Box

Size the box from the spec. Then prove it with a benchmark.

  1. 0Loadonce: file → VRAM
  2. 1You typetext in the app
  3. 2TokenizeCPU: text → IDs
  4. 3To the GPUIDs cross PCIe
  5. 4Prefillwhole prompt, one pass
  6. 5Decoderead all weights → 1 token
  7. 6Show textID → word · repeat 5

1Watch

Now

Press a step on the right, or change any control under the machine.

Readout

Key to the machine

Home AI box assembly jig. The tank is GPU memory: grey blocks are the model, discs are helpers, green liquid is each person's memory. Overflow spills to the RAM tray.
How the numbers work

An always-on AI box is a resident model plan that must fit in GPU memory all at once: the chat or agent model, an embedding model for search, speech-to-text and text-to-speech for voice, maybe a vision model, plus KV cache for every session the household runs at once. Gemma 3 27B at about 4.5 bits is ≈ 15 GB of weights alone, so a 16 GB card holds it only at lower bits and short context; 24 GB is comfortable; a 128 GB DGX Spark holds a 120B MoE but reads memory at 273 GB/s, so it generates more slowly than a 5090 at 1,792 GB/s. What does not fit spills into system RAM (dual-channel DDR5, 89.6 GB/s) and every token waits on it. Then the box has to live in your home 24/7: idle watts dominate the bill, load watts become heat and fan noise. Decode speed assumes 70% of peak bandwidth; the wattmeter assumes 2 busy hours per session per day. Prices are approximate US street prices from Aug–Oct 2026, during the memory shortage; the Oki Home ($2,299) and Ghost Core ($3,499) presets copy those products' published specs. The weights are drawn as eight blocks, one per eighth of the layers; discs are the side models (embed, speech-in, voice-out, vision) and dark rings the OS and runtime; the green liquid is one KV layer per household session. When the plan overflows the tank, the top blocks (the last layers) are lifted out into the RAM tray, split at the exact number of GB that spills; when the plan won't load, all eight wait on the loading dock behind a closed gate and a red beacon, and the tank holds only the OS reserve, side models, runtime and KV. A stack too tall for its tray or dock is drawn squeezed.

  • Grey blocks = model weights
  • Discs = side models
  • Green liquid = KV cache
  • Red beacon = won't load
  • Needles = watts, fan, tok/s
Speed
Open full screen ↗

2Try this

Then change anything yourself.

3Read on ↓

What a personal AI box is

A personal AI box is a quiet, always-on computer that serves a language model to your home or office. Two products show the trade-offs:

Oki Home Ghost Core
GPU GeForce RTX 5060 Ti 16 GB (448 GB/s) RTX PRO 4000 Blackwell SFF 24 GB (432 GB/s, 70 W)
CPU · RAM 32 GB system RAM Ryzen 5 7600 · 64 GB DDR5
Storage 2 TB, expandable to 16 TB 1 TB NVMe
Network 10 GbE, Wi-Fi 7 not listed
Headline model a 27B-class model, always on Qwen and Gemma models preinstalled
Price at launch $2,299 $3,499

The GPU is the difference. Both cards have nearly the same bandwidth. Ghost Core costs about $1,200 more for 8 GB more VRAM and a 70 W card that suits 24/7 use. Oki Home bets that 16 GB plus a small quant can run a 27B model.

Step 1: write the spec

Answer four questions before you choose parts.

  1. Model: 7–9B for quick answers, 24–32B for coding, a 100B+ MoE for cloud-like quality.
  2. Context: 8k for chat, 32k or more for documents (Chapter 4).
  3. Users: each user adds a KV cache (Chapter 8).
  4. Speed: below ~10 tok/s feels slow. Above 30 feels instant.

Step 2: size it

One memory budget and one speed formula size the whole box.

\text{VRAM needed} = \underbrace{N \cdot \tfrac{\text{bits}}{8}}_{\text{weights}} + \underbrace{\text{KV}(\text{context}, \text{users})}_{\text{cache}} + \underbrace{\approx 1\ \text{GB}}_{\text{overhead}} \;\le\; 0.95 \times \text{VRAM}
\text{tokens/s} \approx 0.7 \times \frac{\text{memory bandwidth}}{\text{bytes read per token}}

The reference builds:

Build GPU Model · weights · context Needs Verdict Decode, 1 user
Starter · Oki Home-class RTX 5060 Ti 16GB · 16 GB · 448 GB/s Qwen3 8B · Q4_K_M · 32k 10.8 of 15.2 GB fits ≈ 42 tok/s
Starter · Oki Home-class RTX 5060 Ti 16GB · 16 GB · 448 GB/s gpt-oss-20b (MoE) · MXFP4 · 32k 13.0 of 15.2 GB fits ≈ 135 tok/s
Starter · Oki Home-class RTX 5060 Ti 16GB · 16 GB · 448 GB/s Gemma 3 27B · Q4_0 (QAT) · 8k 17.3 of 15.2 GB tight: 2.1 GB spills to RAM ≈ 13 tok/s
Quiet 24 GB · Ghost Core-class RTX PRO 4000 Blackwell SFF · 24 GB · 432 GB/s Gemma 3 27B · Q4_K_M · 32k 20.5 of 22.8 GB fits ≈ 17 tok/s
Quiet 24 GB · Ghost Core-class RTX PRO 4000 Blackwell SFF · 24 GB · 432 GB/s Qwen3 32B · Q4_K_M · 8k 23.0 of 22.8 GB tight: 0.2 GB spills to RAM ≈ 15 tok/s
Fast · single 5090 RTX 5090 · 32 GB · 1,792 GB/s Qwen3 32B · Q4_K_M · 32k 29.5 of 30.4 GB fits ≈ 52 tok/s
Big memory · DGX Spark DGX Spark (GB10) · 128 GB · 273 GB/s gpt-oss-120b (MoE) · MXFP4 · 64k 65.6 of 121.6 GB fits ≈ 49 tok/s
Big memory · DGX Spark DGX Spark (GB10) · 128 GB · 273 GB/s Llama 3.1 70B · Q4_K_M · 32k 54.5 of 121.6 GB fits ≈ 4 tok/s
Workstation · RTX PRO 6000 RTX PRO 6000 Blackwell · 96 GB · 1,792 GB/s Llama 3.1 70B · Q8_0 · 32k 86.7 of 91.2 GB fits ≈ 16 tok/s

Estimates: 70% of peak bandwidth, KV cache half full, 1 GB overhead. Spilled weights move at about 90 GB/s (DDR5), not VRAM speed.

Row 3 shows the 16 GB trade-off. Gemma 3 27B in Q4_0 needs about 15.2 GB for weights alone, so some spills to RAM. Speed falls to about a third of the 24 GB card. A 3-bit quant fits, at some quality cost.

Step 3: choose the parts

Buy the smallest GPU that fits the budget, then the most bandwidth you can afford.

Part What it decides How to choose
GPU what fits (VRAM) and how fast it talks (bandwidth) the smallest card where the budget above fits with 5–10% headroom; then the most bandwidth you can afford
System RAM loading, offload, RAG indexes, other services at least the model file size, ideally 2× VRAM; 32 GB minimum, 64 GB if you will ever offload
CPU tokenizing, sampling, the web UI, embeddings for RAG a modern 6-core is plenty; the CPU is not on the hot path when the model fits in VRAM
Storage how long a cold load takes and how many models you keep NVMe Gen 4 (≈ 7 GB/s loads a 20 GB model in ~3 s); 1–2 TB holds a healthy model library
Power supply stability under load spikes GPU board power + ~150 W, from a quality unit; Blackwell cards spike above their rating
Cooling and case noise and lifetime when running 24/7 low-profile 70 W cards and large slow fans; avoid blower cards in a living room
Network how many devices it can serve gigabit is plenty for text: a busy reply is a few KB/s; 10 GbE matters for moving models and files

Energy is part of the price. Yearly kWh = average watts × 8.76. A 60 W average is about 525 kWh a year: roughly $160 at $0.30/kWh.

What the next step up buys you

Bandwidth buys speed. Memory buys fit.

GPUVRAM (GB)Band­width (GB/s)Largest model that fitsGemma 3 27B (tok/s)Users at 16kLaunch figuresFine-tuningWhat it unlocks
Price ($)Power (W)
Consumer
RTX 40608272Qwen3 8B—0299115nofits 8B
RTX 5060 Ti16448gpt-oss-20b (MoE)spills0429180small LoRAfits 20B MoE
RTX 5070 Ti16896gpt-oss-20b (MoE)spills0749300small LoRAsame fit, faster
RTX 5090321,792Qwen3 32B7371,999575LoRAfits 27B–32B
Workstation
RTX PRO 6000961,792gpt-oss-120b (MoE)73418,565600fullfits 70B, 120B MoE
DGX
DGX Spark128 unified273gpt-oss-120b (MoE)11583,999240fullfits 120B MoE, many users
DGX Station252 HBM3e7,100gpt-oss-120b (MoE)290124—1,600fullfits 120B MoE, fast

Q4_K_M weights · FP16 KV · 70% of peak bandwidth · KV half full · 1 GB overhead · 5% headroom. Largest model: the biggest in the course data at 8k context, 1 user. Users: 16k context beside one Gemma 3 27B. Price and power are launch figures. The PRO 6000 price is a reported reseller price. DGX power is for the whole system. DGX Station has no published price.

DGX Spark loses on speed but wins on fit. Its 273 GB/s gives 11 tok/s on Gemma 3 27B, against 73 on the RTX 5090. Its 128 GB holds a 120B MoE and 58 users. To compare Macs, DGX and GPUs for this year's open models, use the Can it run? tool.

Blackwell's FP4 is a format gain (Chapter 5), not a bandwidth gain.

Step 4: the software stack

Choose each layer for an appliance.

  1. OS and driver: long-term-support Linux and the current NVIDIA driver.
  2. Runtime: Ollama or llama-server for a few users (Chapter 7), vLLM or SGLang for many (Chapter 8). Keep the model loaded: OLLAMA_KEEP_ALIVE=-1.
  3. API: expose the OpenAI-compatible endpoint on the LAN only. Apps connect when you change one URL.
  4. Apps: a chat UI and a RAG index (Chapter 11).
  5. Privacy: never expose the port to the internet. Use a VPN and require logins.

Step 5: prove it

A box is done when it meets its spec in an honest benchmark (Chapter 12). Far below ~50% of the bandwidth bound means a spill or CPU fallback.

Acceptance checklist

  • ☐ Model and context fit in VRAM with headroom (no spill).
  • ☐ Decode speed meets the spec for the expected users.
  • ☐ TTFT (time to first token) stays under 1–2 s.
  • ☐ Idle power is measured, and the box is quiet enough.
  • ☐ The API is reachable on the LAN only.

Explain it
  1. "Why buy a costlier GPU?" What fits matters more than peak speed. Spilled weights read about 5× slower.
  2. "Do I need a fast CPU?" No. If the model fits in VRAM, the CPU only tokenizes and samples.
  3. "How many people share one box?" Divide the VRAM left after the weights by the cache per user.

Exercises
  1. Family of four, $2,000: write the spec and choose the parts.
  2. Office of 15: size the KV cache for 15 chats at 16k.
  3. Improve Oki Home: change one part, within 10% of the price.
  4. Build it and publish your benchmark.