The budget
Memory needed is weights plus KV cache plus overhead.
\text{memory} \approx \underbrace{N\cdot\frac{\text{bits}}{8}}_{\text{weights}} \;+\; \underbrace{2\,L\,H_{kv}\,d\cdot\frac{\text{kv bits}}{8}\cdot \text{context}\cdot\text{users}}_{\text{KV cache}} \;+\; \underbrace{0.5\text{–}1.5\ \text{GB}}_{\text{runtime overhead}}
- Weights use the total parameter count, MoE included. Every expert must be in memory.
- KV cache (the model's short-term memory) scales with context × concurrent users.
- A runtime allocates the KV cache at start for the full context. A 128k setting costs 128k even for short chats. The settings are
--ctx-size(llama.cpp),num_ctx(Ollama) andmax_model_len(vLLM). - Overhead is the CUDA context, prefill scratch buffers and allocator slack.
- On Windows, leave 5–10% free for the desktop.
| Model · weights | Context · KV | Device | Weights | KV cache | Overhead | Total | Verdict |
|---|---|---|---|---|---|---|---|
| Llama 3.1 8B · Q4_K_M | 8k · 16-bit | RTX 5090 (32 GB) | 4.9 GB | 1.1 GB | 1.0 GB | 6.9 GB | fits |
| Qwen3 32B · Q4_K_M | 8k · 16-bit | RTX 5090 (32 GB) | 19.9 GB | 2.1 GB | 1.0 GB | 23.0 GB | fits |
| Qwen3 32B · Q4_K_M | 32k · 8-bit | RTX 5090 (32 GB) | 19.9 GB | 4.3 GB | 1.0 GB | 25.2 GB | fits |
| Gemma 3 27B · Q4_K_M | 8k · 16-bit | RTX 5060 Ti 16GB (16 GB) | 16.4 GB | 1.1 GB | 1.0 GB | 18.5 GB | does not fit |
| Gemma 3 27B · Q4_K_M | 8k · 16-bit | RTX PRO 4000 Blackwell SFF (24 GB) | 16.4 GB | 1.1 GB | 1.0 GB | 18.5 GB | fits |
The fit map
Find your device and model; the cell is the largest context that fits. It assumes Q4_K_M, one user, an FP16 KV cache and 1 GB of overhead. A dash means the weights alone don't fit.
| Model | RTX 5060 Ti 16GB | RTX 5070 | RTX 5080 | RTX 5090 Laptop | RTX 5090 | RTX PRO 4000 Blackwell SFF | RTX PRO 6000 Blackwell | DGX Spark (GB10) | Apple M4 (this Mac) |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3 0.6B | 121k | 88k | 121k | 187k | 253k | 187k | 783k | >1M | 253k |
| Llama 3.2 3B | 107k | 74k | 107k | 173k | 239k | 173k | 770k | >1M | 239k |
| Llama 3.1 8B | 71k | 42k | 71k | 129k | 187k | 129k | 651k | 883k | 187k |
| Qwen3 8B | 63k | 37k | 63k | 114k | 166k | 114k | 578k | 784k | 166k |
| Mistral Small 3.1 24B | — | — | — | 44k | 91k | 44k | 462k | 647k | 91k |
| Gemma 3 27B | — | — | — | 61k | 154k | 61k | 896k | >1M | 154k |
| Qwen3 32B | — | — | — | 7k | 36k | 7k | 268k | 384k | 36k |
| Llama 3.1 70B | — | — | — | — | — | — | 145k | 237k | — |
| Qwen3 30B-A3B (MoE) | — | — | — | 34k | 111k | 34k | 729k | >1M | 111k |
| gpt-oss-20b (MoE) | 60k | — | 60k | 369k | 678k | 369k | >1M | >1M | 678k |
| gpt-oss-120b (MoE) | — | — | — | — | — | — | 523k | >1M | — |
To test Macs and this year's open models, use the Can it run? tool.
A model's trained context limit can bind first (128k for Llama 3.1, 32k native for Qwen3 8B).
Sliding-window models (Gemma 3, gpt-oss) cache only the last window on most layers. Their numbers are high.
Choosing hardware
Capacity decides what loads; bandwidth decides how fast it runs.
| If the workload is… | Prefer | Because |
|---|---|---|
| A ≤14B chat or coding assistant, single user, speed matters | GeForce RTX 5070 Ti / 5080 / 5090 | everything fits; bandwidth gives 100–250 tok/s |
| 24–32B dense models, long context | RTX 5090 (32 GB) | fits Q4 weights plus a usable KV cache at 1.8 TB/s |
| 70B dense, multi-user, or fine-tuning | RTX PRO 6000 Blackwell (96 GB) | the 5090's bandwidth with 3× the capacity, plus ECC |
| 100B+ MoE, big-memory prototyping, the NVIDIA software stack on a desk | DGX Spark (128 GB unified) | capacity first; MoE keeps decode tolerable at 273 GB/s |
| Frontier-scale local work | DGX Station (GB300) | 252 GB of HBM3e at 7.1 TB/s plus 496 GB of LPDDR5X |
| Laptops | RTX 5090 Laptop (24 GB) / 5080 Laptop | portable; plan for 8–14B models with room for context |
Explain it
- "Can a 16 GB RTX 5080 run Qwen3 32B?" Not on the GPU alone: Q4_K_M weights are about 20 GB. Use a 3-bit quant, offload part of it (Chapter 9), or get a 32 GB GPU. Even the MoE Qwen3 30B-A3B (about 18.5 GB at Q4_K_M) is slightly too big.
- "I want 128k context on a laptop." An 8B GQA model needs about 17 GB of KV at FP16 for 128k. Use an FP8 or Q8 KV cache, a smaller context, or retrieval.
- "Why does Ollama use more VRAM than the formula?" Its defaults allocate KV for several parallel sequences plus compute buffers. The server log prints the exact allocation.
Exercises
- Recompute the fit map with an FP8 KV cache and Q8_0 weights; which pairs change category?
- Write
recommend(model, context, users)returning the cheapest device that fits with 10% headroom.