Master Local AI
  1. 0Load
  2. 1You type
  3. 2Tokenize
  4. 3To the GPU
  5. 4Prefill
  6. 5Decode
  7. 6Show text
Can it run?

Chapter 5 · Practitioner

Quantization: Fewer Bits, Same Model (Mostly)

Fewer bits: smaller, faster, a little more error. FP16 shows none.

  1. 0Loadonce: file → VRAM
  2. 1You typetext in the app
  3. 2TokenizeCPU: text → IDs
  4. 3To the GPUIDs cross PCIe
  5. 4Prefillwhole prompt, one pass
  6. 5Decoderead all weights → 1 token
  7. 6Show textID → word · repeat 5

1Watch

Now

Press a step on the right, or change any control under the machine.

Readout

Key to the machine

A ball sorter: each ball is a weight, each slot a value the format can store. Fewer bits mean wider slots, more error. One outlier widens its block.
How the numbers work

Each weight is stored as a small code plus a shared scale. Each lane is one block: its micrometer sets the scale so the comb spans the block's largest ball, and every weight rounds to the nearest bin; the dashed ring marks its true value and the whisker the rounding error, and the gauge reads the RMS of those errors over all 4,096 weights. All lanes share one value axis in σ, linear to ±4σ and squeezed beyond, so a wider scale shows as wider, fewer slots. The number on each lane's post is the block size; for one scale per row the gang bar rises against all four micrometers. FP32 stores each weight in 4 bytes and FP16 in 2: the 8B example falls from 32.1 GB to 16.1 GB, and because decode reads every weight once per token it runs about twice as fast, while the rounding error stays below 0.1% of σ. Fewer bits means fewer, coarser bins (16 for 4-bit, 256 for 8-bit), a smaller model and fewer bytes to read per token, so faster decode, at the price of rounding error. The bit count includes the scales: INT4 with one scale per 16 weights reads 6.00 bits, not 4, because a 16-bit scale and a 16-bit zero point are stored for every 16 weights (4 + 32 ÷ 16 = 6); with 128-weight blocks the same cost is 4.25 bits. Turn on the outlier and its lane's comb must stretch to reach it, crowding the other weights of that block into the few middle bins; one scale for a whole 4,096-weight row (FP8, INT8 per-channel) lets one outlier coarsen every lane, while NVFP4's 16-weight blocks confine the damage. The error is computed live on a fixed seeded sample of 4,096 Gaussian weights with each format's real arithmetic: symmetric absmax for INT8 and INT4, the FP4 E2M1 grid {0, 0.5, 1, 1.5, 2, 3, 4, 6} with an E4M3 scale per 16 (NVFP4) or a power-of-two scale per 32 (MXFP4), asymmetric 4-bit with 6-bit sub-scales and mins (the Q4_K tensors of Q4_K_M, with plain min/max where llama.cpp searches for slightly better scales), signed 3-bit with 6-bit sub-scales (Q3_K). IQ2_XS really uses an E8-lattice codebook over groups of 8; it is drawn as a plain 2-bit grid, so its error here is an upper bound. Speed assumes decode reads every weight once per token at 70% of an RTX 5090's 1,792 GB/s.

  • Comb slot = storable value
  • Ball = one weight
  • Dashed ring = true value
  • Whisker = rounding error
  • Highlighted ball = outlier
  • Needle = average error
  • Post number = weights per scale
  • Gang bar up = row-wide scale
  • Top panel = bytes, GB vs 5090
Speed
Open full screen ↗

2Try this

Then change anything yourself.

3Read on ↓

Symmetric block quantization

Each block of weights shares one scale. The scale turns each weight into a small integer. For a block of g weights and a signed k-bit grid with q_{\max} = 2^{k-1}-1:

s = \frac{\max_i |w_i|}{q_{\max}},\qquad q_i = \operatorname{round}(w_i / s),\qquad \hat w_i = s\, q_i
  • Storage is g\cdot k bits plus one scale per block.
  • Q4_0 stores 32 weights with an FP16 scale: 4 + 16/32 = 4.5 bits per weight.
  • Kernels dequantize tiles inside the matmul. Memory traffic falls by roughly 16/bits.
  • Decode is bandwidth bound. Tokens/s rises by nearly the same factor.

The formats you'll meet

Pick the format your runtime supports; the bits per weight tell you the size.

Format Layout Bits/weight Where
FP32 4 bytes per weight 32 training format; too big for most inference
FP16 / BF16 2 bytes per weight 16 the inference default: half of FP32, no visible error
GGUF Q8_0 32 × int8 + FP16 scale 8.5 near-lossless llama.cpp baseline
GGUF Q4_K_M 256-weight super-blocks of 8 sub-blocks with 6-bit sub-scales and mins; some tensors kept at Q6_K ≈4.8–4.9 the default "good" llama.cpp/Ollama quant
GGUF IQ2/IQ3 + imatrix lattice codebooks, importance-weighted 2–3.5 extreme compression; quality drops
AWQ / GPTQ INT4 (g128) int4 + FP16 scale (and zero point) per 128; calibration on real activations ≈4.1–4.25 vLLM, SGLang, TensorRT-LLM
FP8 E4M3 8-bit float, per-tensor or per-channel scale 8 Ada, Hopper and Blackwell tensor cores; weights and KV cache
NVFP4 E2M1 values, blocks of 16, FP8 E4M3 scale per block + FP32 per-tensor scale ≈4.5 Blackwell tensor cores (RTX 50, RTX PRO 6000, DGX Spark)
MXFP4 E2M1 values, blocks of 32, 8-bit power-of-two (E8M0) scale 4.25 OCP standard; gpt-oss ships its MoE weights in MXFP4

E2M1 can represent only ±{0, 0.5, 1, 1.5, 2, 3, 4, 6}. This non-uniform grid packs more resolution near zero, where most weights live, so FP4 with small blocks beats naive INT4.

Lab: quantize real weights

Quantize a real matrix five ways and compare the output error. The lab reads the MLP up-projection of the first block of nomic-embed-text from the GGUF blob Ollama downloaded. It measures the weight error and the error of x @ Wᵀ on random activations.

Lab · runs on your own machine

uv run labs/05_quantize.pyDownload all labs (.zip)

Needs uv and, for the live parts, Ollama with a small chat model: ollama pull qwen3:0.6b.

Show the code
# /// script
# requires-python = ">=3.11"
# dependencies = ["numpy", "torch"]
# ///
import _local as local

import numpy as np, torch

path = local.blob_path("nomic-embed-text:latest")
W = local.read_gguf_tensor(path, "blk.0.ffn_up.weight").astype(np.float32)
print("real weight matrix:", W.shape, "from", path.name[:20], "…")

E2M1 = np.array([0, 0.5, 1, 1.5, 2, 3, 4, 6], dtype=np.float32)

def to_grid(v, grid):
    idx = np.abs(np.abs(v)[..., None] - grid).argmin(-1)
    return np.sign(v) * grid[idx]

def blocks(w, g):
    return w.reshape(-1, g)

def int_block(w, bits, g):
    b = blocks(w, g); qmax = 2 ** (bits - 1) - 1
    s = np.abs(b).max(1, keepdims=True) / qmax; s[s == 0] = 1
    s = s.astype(np.float16).astype(np.float32)
    return (np.clip(np.round(b / s), -qmax - 1, qmax) * s).reshape(w.shape)

def nvfp4(w):
    b = blocks(w, 16)
    tensor_scale = np.abs(w).max() / (6 * 448)
    s = np.abs(b).max(1, keepdims=True) / 6 / tensor_scale
    s = torch.tensor(s).to(torch.float8_e4m3fn).to(torch.float32).numpy() * tensor_scale
    s[s == 0] = 1
    return (to_grid(b / s, E2M1) * s).reshape(w.shape)

def mxfp4(w):
    b = blocks(w, 32)
    amax = np.abs(b).max(1, keepdims=True); amax[amax == 0] = 1
    s = 2.0 ** (np.floor(np.log2(amax)) - 2)
    return (to_grid(np.clip(b / s, -6, 6), E2M1) * s).reshape(w.shape)

def fp8(w):
    s = np.abs(w).max() / 448
    return torch.tensor(w / s).to(torch.float8_e4m3fn).to(torch.float32).numpy() * s

schemes = {
    "Q8_0  (int8, g32)": (lambda w: int_block(w, 8, 32), 8.5),
    "FP8 E4M3 (per-tensor)": (fp8, 8.0),
    "Q4_0  (int4, g32)": (lambda w: int_block(w, 4, 32), 4.5),
    "NVFP4 (E2M1, g16, FP8 scale)": (nvfp4, 4.5),
    "MXFP4 (E2M1, g32, pow2 scale)": (mxfp4, 4.25),
    "INT4 per-tensor (no blocks)": (lambda w: int_block(w.reshape(1, -1), 4, w.size).reshape(w.shape), 4.0),
}
x = np.random.default_rng(0).normal(size=(64, W.shape[1])).astype(np.float32)
ref = x @ W.T
rows = ["| scheme | bits/weight | weight rel. error | output rel. error |", "|---|---:|---:|---:|"]
for name, (fn, bpw) in schemes.items():
    Wq = fn(W)
    we = np.linalg.norm(Wq - W) / np.linalg.norm(W)
    oe = np.linalg.norm(x @ Wq.T - ref) / np.linalg.norm(ref)
    rows.append(f"| {name} | {bpw} | {we:.2%} | {oe:.2%} |")
print(("\n".join(rows)))
Sample output on this course's Mac (Apple M4, 32 GB)
real weight matrix: (3072, 768) from sha256-970aa74c0a90e …
| scheme | bits/weight | weight rel. error | output rel. error |
|---|---:|---:|---:|
| Q8_0  (int8, g32) | 8.5 | 0.54% | 0.54% |
| FP8 E4M3 (per-tensor) | 8.0 | 2.65% | 2.63% |
| Q4_0  (int4, g32) | 4.5 | 9.79% | 9.78% |
| NVFP4 (E2M1, g16, FP8 scale) | 4.5 | 9.51% | 9.46% |
| MXFP4 (E2M1, g32, pow2 scale) | 4.25 | 11.52% | 11.47% |
| INT4 per-tensor (no blocks) | 4.0 | 46.07% | 45.87% |

Relative error on this matrix:

  • Q8_0 ≈ 0.5%.

  • FP8 ≈ 2.6%.

  • Blocked 4-bit ≈ 10%.

  • Per-tensor INT4 ≈ 46%.

  • 8-bit is close to free. Q8_0's small blocks beat per-tensor FP8.

  • Blocked 4-bit costs about 10% per matrix. Q4_K_M and NVFP4 still score close to BF16, because errors partly average out across layers.

  • NVFP4's 16-wide blocks edge out Q4_0. Both use 4.5 bits.

  • Per-tensor INT4 is catastrophic. One outlier sets the scale for all weights.

  • Good 4-bit formats use small blocks, non-uniform grids or calibration (AWQ, GPTQ, imatrix).

Measuring quality

Weight error is only a proxy; measure the model's outputs.

  • Measure the KL divergence between the quantized and BF16 next-token distributions (llama-perplexity --kl-divergence). Perplexity on held-out text also works.
  • Run task checks that match your use: coding pass@1, retrieval QA, tool-call validity.
  • Q8_0 and FP8 are near-lossless.
  • Q5_K_M and Q4_K_M are the sweet spot.
  • Below 4 bits, quality falls fast, especially in small models.

The KV cache can be quantized too

An FP8 or Q8_0 KV cache halves its memory at small quality cost. Use --cache-type-k q8_0 --cache-type-v q8_0 in llama.cpp or --kv-cache-dtype fp8 in vLLM. This is often the cheapest way to fit long context.

Explain it
  1. "What is NVFP4?" Blackwell's 4-bit float: E2M1 values in blocks of 16 with FP8 scales, run natively by tensor cores. Near-FP8 accuracy at about half the bytes.
  2. "Q4_K_M or AWQ?" Choose by runtime. Q4_K_M is GGUF (llama.cpp, Ollama, LM Studio). AWQ and GPTQ are INT4 safetensors (vLLM, SGLang, TensorRT-LLM).
  3. "How do you prove a quant is good enough?" Compare KL divergence or perplexity with BF16 plus your own task evaluation, and report speed too.

Exercises
  1. In the lab, vary the INT4 block size from 16 to 1024; where is the knee in error versus bits?
  2. Multiply 0.1% of the weights by 20 and repeat; which formats stay robust?