Master Local AI
  1. 0Load
  2. 1You type
  3. 2Tokenize
  4. 3To the GPU
  5. 4Prefill
  6. 5Decode
  7. 6Show text
Can it run?

Chapter 6 · Builder

Model Files: GGUF, Safetensors, ONNX and Engines

A model is more than weights. The format decides who can load it.

  1. 0Loadonce: file → VRAM
  2. 1You typetext in the app
  3. 2TokenizeCPU: text → IDs
  4. 3To the GPUIDs cross PCIe
  5. 4Prefillwhole prompt, one pass
  6. 5Decoderead all weights → 1 token
  7. 6Show textID → word · repeat 5

1Watch

Now

Press a step on the right, or change any control under the machine.

Readout

Key to the machine

A filing cabinet on a loading dock. The cabinet is the model file. Each drawer is one section. Watch the weight blocks travel to the GPU bay.
How the numbers work

A runnable model is weights plus a tokenizer, a chat template and an architecture config. GGUF packs all four into one self-describing file: the magic GGUF and version 3, a tensor count, metadata keys such as general.architecture, the context length, tokenizer.ggml.* and tokenizer.chat_template, then one index entry per tensor (name, shape, quant type, offset), zero padding to the alignment (32 bytes by default), then the tensor data. safetensors holds only tensors (an 8-byte little-endian header length, a JSON header of dtype, shape and offsets, then raw bytes); the tokenizer and config sit beside it as JSON files. ONNX is an operator graph plus weights (big ones in an external data file) for ONNX Runtime and Windows ML, and a TensorRT engine is a compiled plan for one GPU type. GGUF and safetensors data is aligned and contiguous, so a loader can memory-map it and the OS pages blocks in from disk (NVMe about 7 GB/s) as they are touched, or from RAM if they are still in the page cache. The hoist lifts pallets from the cabinet (disk) onto the RAM shelf (the page cache); the conveyor (PCIe) carries them to the card, where each block hops onto a memory site. Empty pallets stack on the floor. If the model is too big for the GPU, the bay fills: the blocks that do not fit wait on the RAM shelf (the CPU runs those layers), or go back into the cabinet when RAM cannot hold them either. Drag RAM below the file size and the gauge pegs: there is no warm load.

  • Lit block = weight in transit
  • Drawers = file sections, disk order
  • Bare board = no memory site
Speed
Open full screen ↗

2Try this

Then change anything yourself.

3Read on ↓

What makes a model runnable

A runnable model needs five pieces, and each format stores them differently.

Piece Hugging Face repo GGUF
Weights model-0000x-of-0000N.safetensors + model.safetensors.index.json tensor data section
Architecture config.json (layers, heads, rope, vocab…) <arch>.* metadata keys
Tokenizer tokenizer.json / tokenizer.model tokenizer.ggml.* keys
Chat template tokenizer_config.json → chat_template (Jinja) tokenizer.chat_template
Generation defaults generation_config.json runtime parameters (Ollama Modelfile PARAMETER)
  • GGUF (llama.cpp, Ollama, LM Studio): one self-describing file that the OS can memory-map. It stores the quantized types (Q4_K, Q8_0, IQ…, MXFP4) natively.
  • GGUF layout: the magic GGUF, a version, the tensor and metadata counts, the metadata, the tensor infos (name, shape, type, offset), padding, then the data.
  • Safetensors (Hugging Face, PyTorch, vLLM, SGLang, TensorRT-LLM checkpoints): an 8-byte header length, a JSON header (dtype, shape, byte offsets), then raw bytes. It is safe because loading it cannot run code, unlike pickle .bin files.
  • ONNX (ONNX Runtime, Windows ML): an operator graph plus weights. Execution providers (CUDA, TensorRT, TensorRT for RTX, DirectML) make it portable across vendors.
  • TensorRT engine (TensorRT, TensorRT-LLM): a graph compiled for one GPU architecture and TensorRT version. It is the fastest format, but you rebuild it for each target.
  • TensorRT for RTX builds the engine on the device, just in time. An app then ships one portable file to the RTX installed base.
  • MLX (Apple silicon): safetensors with MLX-specific quantization.

Lab: open a real GGUF

Ollama stores every model as a GGUF blob, and the header holds everything a runtime needs. The lab reads the header of the blob on this machine and skips the weights.

Lab · runs on your own machine

uv run labs/06_gguf.pyDownload all labs (.zip)

Needs uv and, for the live parts, Ollama with a small chat model: ollama pull qwen3:0.6b.

Show the code
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
import _local as local

from collections import Counter

name = (local.chat_models() or local.embed_models() or [None])[0]
if name:
    path = local.blob_path(name)
    g = local.read_gguf(path)
    print(f"{name} → {path}")
    print(f"GGUF v{g['version']} · {len(g['tensors'])} tensors · {len(g['metadata'])} metadata keys · header {g['header_bytes']/1024:.0f} KiB of {g['file_bytes']/1e9:.2f} GB")
    for key, value in g["metadata"].items():
        if key.startswith("tokenizer.ggml.") and isinstance(value, dict):
            value = f"array of {value['array_of']:,}: {value['first'][:4]}…"
        print(f"  {key:45} {str(value)[:90]}")
    print("tensor types:", dict(Counter(t["type"] for t in g["tensors"])))
    for t in g["tensors"][:6]:
        print(f"  {t['name']:32} {str(t['shape']):>16} {t['type']:>6} @ {t['offset']:,}")

model = local.need_chat_model()
if model:
    print(local.show(model).get("template", "")[:1200])
Sample output on this course's Mac (Apple M4, 32 GB)
qwen3:0.6b → /Users/mpl/.ollama/models/blobs/sha256-7f4030143c1c477224c5434f8272c662a8b042079a0a584f0a27a1684fe2e1fa
GGUF v3 · 311 tensors · 28 metadata keys · header 5812 KiB of 0.52 GB
  general.architecture                          qwen3
  general.type                                  model
  general.name                                  Qwen3 0.6B
  general.basename                              Qwen3
  general.size_label                            0.6B
  general.license                               apache-2.0
  qwen3.block_count                             28
  qwen3.context_length                          40960
  qwen3.embedding_length                        1024
  qwen3.feed_forward_length                     3072
  qwen3.attention.head_count                    16
  qwen3.attention.head_count_kv                 8
  qwen3.rope.freq_base                          1000000.0
  qwen3.attention.layer_norm_rms_epsilon        9.999999974752427e-07
  qwen3.attention.key_length                    128
  qwen3.attention.value_length                  128
  tokenizer.ggml.model                          gpt2
  tokenizer.ggml.pre                            qwen2
  tokenizer.ggml.tokens                         array of 151,936: ['!', '"', '#', '$']…
  tokenizer.ggml.token_type                     array of 151,936: [1, 1, 1, 1]…
  tokenizer.ggml.merges                         array of 151,387: ['Ġ Ġ', 'ĠĠ ĠĠ', 'i n', 'Ġ t']…
  tokenizer.ggml.eos_token_id                   151645
  tokenizer.ggml.padding_token_id               151643
  tokenizer.ggml.bos_token_id                   151643
  tokenizer.ggml.add_bos_token                  False
  tokenizer.chat_template                       {%- if tools %}
    {{- '<|im_start|>system\n' }}
    {%- if messages[0].role == 'system' 
  general.quantization_version                  2
  general.file_type                             15
tensor types: {'Q6_K': 15, 'F32': 113, 'Q4_K': 155, 'F16': 28}
  output.weight                      [1024, 151936]   Q6_K @ 0
  output_norm.weight                         [1024]    F32 @ 127,626,240
  token_embd.weight                  [1024, 151936]   Q4_K @ 127,630,336
  blk.0.attn_k.weight                  [1024, 1024]   Q4_K @ 215,145,472
  blk.0.attn_k_norm.weight                    [128]    F32 @ 215,735,296
  blk.0.attn_norm.weight                     [1024]    F32 @ 215,735,808

{{- $lastUserIdx := -1 -}}
{{- range $idx, $msg := .Messages -}}
{{- if eq $msg.Role "user" }}{{ $lastUserIdx = $idx }}{{ end -}}
{{- end }}
{{- if or .System .Tools }}<|im_start|>system
{{ if .System }}
{{ .System }}
{{- end }}
{{- if .Tools }}

# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{{- range .Tools }}
{"type": "function", "function": {{ .Function }}}
{{- end }}
</tools>

For each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:
<tool_call>
{"name": <function-name>, "arguments": <args-json-object>}
</tool_call>
{{- end -}}
<|im_end|>
{{ end }}
{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 -}}
{{- if eq .Role "user" }}<|im_start|>user
{{ .Content }}
{{- if and $.IsThinkSet (eq $i $lastUserIdx) }}
   {{- if $.Think -}}
      {{- " "}}/think
   {{- else -}}
      {{- " "}}/no_think
   {{- end -}}
{{- end }}<|im_end|>
{{ else if eq .Role "assistant" }}<|im_start|>assistant
{{ if (and $.IsThinkSet (and .Thinking (or $last (gt $i $lastUserIdx)))) -}}
<think>{{ .Thinking }}</

What to notice: the tokenizer vocabulary, the special-token IDs and the architecture settings all live in the file. The header is a few hundred KB. The OS pages in the weights on demand, so the second load takes milliseconds: the pages are still in the file cache.

Lab: safetensors from scratch

A safetensors file is an 8-byte length, a JSON header and raw bytes. The script builds one and reads it back.

import json, struct, torch
from safetensors.torch import save_file

tensors = {"layer0.weight": torch.randn(256, 128, dtype=torch.float16), "layer0.bias": torch.zeros(256, dtype=torch.float16)}
save_file(tensors, "tiny.safetensors", metadata={"format": "pt", "course": "local-ai-builder"})
raw = open("tiny.safetensors", "rb").read()
header_len = struct.unpack("<Q", raw[:8])[0]
print("header length:", header_len, "bytes")
print(json.dumps(json.loads(raw[8:8 + header_len]), indent=1))
print("data bytes:", len(raw) - 8 - header_len, "= 256·128·2 + 256·2 =", 256 * 128 * 2 + 256 * 2)

Chat templates: the silent integration bug

The wrong chat template gives rambling output, broken tool calls or ignored system prompts, even with the right weights. Always check the template that the runtime applies.

Conversion paths

Two conversion paths cover nearly every local runtime.

# Hugging Face → GGUF (llama.cpp tools)
python convert_hf_to_gguf.py ./Qwen3-8B --outtype bf16 --outfile qwen3-8b-bf16.gguf
llama-quantize qwen3-8b-bf16.gguf qwen3-8b-Q4_K_M.gguf Q4_K_M        # add --imatrix for low-bit quants
ollama create my-qwen -f Modelfile                                   # FROM ./qwen3-8b-Q4_K_M.gguf

# Hugging Face → NVIDIA-optimized
# TensorRT Model Optimizer quantizes (FP8, NVFP4, INT4 AWQ) → TensorRT-LLM builds or loads → serve

To cover every local runtime on day one, publish BF16 safetensors, an official GGUF set (Q8_0, Q6_K, Q4_K_M) and an FP8 or NVFP4 checkpoint.

Explain it
  1. "Why did the ecosystem move from .bin to safetensors?" Pickle can run code on load. Safetensors is pure data, and it loads faster (zero-copy, mmap).
  2. "A model works in Transformers but outputs garbage in Ollama. First suspects?" Check the chat template, the stop tokens, the tokenizer conversion, and the rope or context settings. Check the quantization last.
  3. "What is the trade-off with TensorRT engines?" Compiling for the exact GPU gives peak speed but costs portability. TensorRT for RTX and Windows ML reduce that cost.

Exercises
  1. Extend read_gguf to compute each tensor's byte size, and check that the sizes add up to the file.
  2. Compare the metadata of two quantizations of one model: what changes besides the tensor types?