Master Local AI
  1. 0Load
  2. 1You type
  3. 2Tokenize
  4. 3To the GPU
  5. 4Prefill
  6. 5Decode
  7. 6Show text
Can it run?

Chapter 7 · Builder

Inference Runtimes: One Socket, Many Engines

Every engine offers the same socket. Pick by who you serve.

  1. 0Loadonce: file → VRAM
  2. 1You typetext in the app
  3. 2TokenizeCPU: text → IDs
  4. 3To the GPUIDs cross PCIe
  5. 4Prefillwhole prompt, one pass
  6. 5Decoderead all weights → 1 token
  7. 6Show textID → word · repeat 5

1Watch

Now

Press a step on the right, or change any control under the machine.

Readout

Key to the machine

A telephone switchboard wired to a rack of engines. Every app plugs into the same socket. Watch the trunk cord move when you change OS, audience or format.
How the numbers work

Apps reach a local model through a runtime, and almost every runtime exposes the same OpenAI-compatible HTTP API (POST /v1/chat/completions), so one app can swap engines without changing code. Desktop-first runtimes put ease first: llama.cpp (GGUF; CUDA, Vulkan, Metal and CPU backends; llama-server), Ollama (a model library and Modelfile on port 11434, GGUF on a llama.cpp-derived engine), LM Studio (a GUI over llama.cpp and MLX engines) and Windows ML, which ships in Windows on ONNX Runtime and picks an execution provider per PC, TensorRT for RTX on NVIDIA. Server-first engines put throughput first: vLLM (PagedAttention, continuous batching), SGLang (RadixAttention prefix cache, structured output), TensorRT-LLM (compiled engines, in-flight batching, FP8 and NVFP4 on Blackwell) and NVIDIA NIM (packaged, optimized microservices). Pick an OS, an audience, a GPU and a model format, and the bay patches the engine that fits. Raise users or context and watch the request bottles queue behind the gate (no free slot, or no memory left for another user's KV cache) or the memory bar spill loose segments into the overflow tray (the model does not fit). The bar is the card's memory in tenths: dark segments are weights plus about 1 GB of runtime overhead, green segments the KV cache of the users being served. Above 32 users each bottle stands for several users; the counters give the exact numbers.

  • Lit route = live request path
  • Rack = the eight runtimes
  • Green bar = KV cache
  • Red lamp = NO FIT
Speed
Open full screen ↗

2Try this

Then change anything yourself.

3Read on ↓

The landscape

Desktop runtimes optimize for "runs on any PC", and server runtimes optimize for throughput across many requests.

Runtime Model format GPU backends Optimized for Notes
llama.cpp GGUF CUDA, Vulkan, Metal, SYCL, CPU running anything, anywhere llama-server speaks the OpenAI API; the reference for quantized local inference
Ollama GGUF (pulls from its library or Hugging Face) CUDA, Metal, ROCm (+ Vulkan) developer ergonomics ollama run, REST on :11434, OpenAI-compatible /v1, Modelfile
LM Studio GGUF, MLX llama.cpp and MLX engines a GUI for end users and developers lms CLI; local server with an OpenAI-compatible API
Windows ML / ONNX Runtime ONNX execution providers incl. TensorRT for RTX, DirectML shipping AI inside Windows apps the OS-level path to the RTX installed base
vLLM safetensors (HF), AWQ/GPTQ, FP8 CUDA (also ROCm and others) throughput for many users PagedAttention, continuous batching, prefix caching
SGLang safetensors, FP8, AWQ CUDA structured, agentic workloads RadixAttention prefix cache, fast constrained decoding
TensorRT-LLM HF → optimized checkpoints/engines CUDA (NVIDIA only) peak NVIDIA performance FP8 and NVFP4, in-flight batching, speculative decoding
NIM packaged CUDA enterprise deployment containers with optimized engines and standard APIs

NVIDIA supports both groups. CUDA paths in llama.cpp and Ollama serve the broad installed base. TensorRT-LLM, TensorRT for RTX and NIM serve cases that need peak performance.

Lab: one client, any engine

The OpenAI-compatible API is the standard socket, so the client code does not change between engines. Point base_url at Ollama (:11434/v1), llama-server (:8080/v1), LM Studio (:1234/v1) or vLLM (:8000/v1). A gateway such as LiteLLM offers the same socket in front of several engines and the cloud (Chapter 14).

import json, time, urllib.request

def chat(base_url, model, prompt, max_tokens=64):
    body = {"model": model, "messages": [{"role": "user", "content": prompt}], "max_tokens": max_tokens, "stream": False}
    req = urllib.request.Request(base_url + "/chat/completions", data=json.dumps(body).encode(), headers={"Content-Type": "application/json"})
    start = time.perf_counter()
    with urllib.request.urlopen(req, timeout=300) as r:
        data = json.loads(r.read())
    return data["choices"][0]["message"]["content"], data.get("usage", {}), time.perf_counter() - start

ENGINES = {"Ollama": "http://127.0.0.1:11434/v1", "llama-server": "http://127.0.0.1:8080/v1", "LM Studio": "http://127.0.0.1:1234/v1", "vLLM": "http://127.0.0.1:8000/v1"}
for engine, url in ENGINES.items():
    try:
        with urllib.request.urlopen(url + "/models", timeout=1) as r:
            listed = [m["id"] for m in json.loads(r.read())["data"]]
        print(f"{engine:>12}: up · models {listed[:4]}")
    except OSError:
        print(f"{engine:>12}: not running")

model = local.need_chat_model()
if model:
    text, usage, seconds = chat(ENGINES["Ollama"], model, "In one sentence: why does VRAM bandwidth matter?")
    print(f"\n{text}\n{usage} · {seconds:.2f} s")

Starting each engine (reference)

Each engine starts with one command.

# llama.cpp, all layers on the GPU, 16k context, OpenAI API on :8080
llama-server -m qwen3-8b-Q4_K_M.gguf -ngl 99 -c 16384 --port 8080

# Ollama
ollama serve                      # API on :11434
ollama run qwen3:8b               # pulls the GGUF, then chats

# LM Studio
lms server start                  # API on :1234

# vLLM (Linux + NVIDIA)
vllm serve Qwen/Qwen3-8B --max-model-len 16384 --gpu-memory-utilization 0.9

# SGLang
python -m sglang.launch_server --model-path Qwen/Qwen3-8B --port 30000

# TensorRT-LLM (OpenAI-compatible server)
trtllm-serve Qwen/Qwen3-8B

Choosing, by scenario

Choose the runtime by who you serve.

Scenario Recommendation Why
Indie dev adds local chat to a desktop app on Windows, macOS and Linux Ollama or embedded llama.cpp one GGUF everywhere; excellent CUDA path on RTX
ISV ships an AI feature to millions of Windows RTX PCs Windows ML (ONNX) with the TensorRT for RTX execution provider OS-managed, small footprint, vendor-optimized
Team serves an internal coding model to 30 engineers from one RTX PRO 6000 vLLM or SGLang (or TensorRT-LLM for peak) continuous batching and prefix caching
Agent platform with long, shared system prompts SGLang or vLLM with prefix caching reuses KV across steps
Model developer launch publish GGUF + safetensors (+ FP8/NVFP4); day-0 Ollama and llama.cpp support meet every runtime where it is

Explain it
  1. "Ollama versus vLLM?" Ollama serves one user on broad hardware. vLLM serves many users with PagedAttention and continuous batching on Linux GPUs.
  2. "How does NVIDIA reach the Windows app ecosystem?" Windows ML with the TensorRT for RTX provider reaches app developers, and CUDA paths in llama.cpp and Ollama reach open-source desktops. NIM and TensorRT-LLM cover workstations and servers.
  3. "An app hardcodes an OpenAI endpoint. How hard is local?" Often only the base_url and the model name change. The hard parts are the model choice, its chat template and tool-call format, and the hardware.

Exercises
  1. Start llama-server with a GGUF, point the lab client at it, and compare tokens/s with Ollama on the same file.
  2. Write a one-page decision tree that picks a runtime in five minutes.