Master Local AI
  1. 0Load
  2. 1You type
  3. 2Tokenize
  4. 3To the GPU
  5. 4Prefill
  6. 5Decode
  7. 6Show text
Can it run?

Chapter 14 · Master

Hybrid: One Socket, Local First

Local first. The cloud takes the overflow, never private data.

  1. 0Loadonce: file → VRAM
  2. 1You typetext in the app
  3. 2TokenizeCPU: text → IDs
  4. 3To the GPUIDs cross PCIe
  5. 4Prefillwhole prompt, one pass
  6. 5Decoderead all weights → 1 token
  7. 6Show textID → word · repeat 5

1Watch

Now

Press a step on the right, or change any control under the machine.

Readout

Key to the machine

A sorting yard before your AI box. Each crate is one request. The gate is the router, such as LiteLLM. It picks the local box or the cloud dock.
How the numbers work

The local box is an RTX 5090 with 32 GB of VRAM that holds Qwen3 32B at GGUF Q4_K_M (19.9 GB of weights, 1 GB of overhead). Each slot reserves a 4,096-token KV cache (1.07 GB), so 7 slots fit inside 90% of VRAM. One decode step reads the weights once plus the cache of every busy slot, and gives each busy slot one token, so each slot gets 0.7 × 1,792 GB/s ÷ (19.9 GB + busy × 0.54 GB): 61 tok/s alone, 53 tok/s with all 7 slots busy. The gate sends a request to the dock only when fallback is on, the dock is open, and the request is not private; hard requests (20% of the mix, 900 tokens instead of 300) always go to the dock, easy ones go when every slot is taken. Users think for about 2 s after a reply. The cloud numbers are illustrative: Provider A is $4, B $1.50 and C $0.50 per million tokens, with 0.6 s, 0.9 s and 1.4 s to the first token at 140, 90 and 55 tok/s. A busy day is 4 hours and a month is 30 days. When the day's cost passes the monthly budget ÷ 30, the dock shuts for the rest of the day and the figure plays the day in 50 seconds: watch the TODAY needle reach the red tick. Latency is the wait in the queue plus the time in a slot or at the dock, without the belt rides. KV slices are drawn about twice as tall as their share of VRAM.

  • Tall crate = hard request
  • Dark crate, padlock = private
  • Green slice = KV cache in use
  • Grey slabs = model weights
  • Red beacon = dock shut
Speed
Open full screen ↗

2Try this

Then change anything yourself.

3Read on ↓

What hybrid means

A hybrid setup answers on your own box first and sends only the overflow to the cloud. The app keeps one socket, the OpenAI-compatible API from Chapter 7. A router sits behind that socket. For each request, the router picks the engine.

Chapter 13 built the box. This chapter answers two questions. What happens when the box is full? What happens when the question is too hard for the local model?

The routing rules

Four rules decide where a request goes.

  • Fallback. When every slot is busy, the gate sends the request to the cloud dock. Step 2 above shows the queue clear while the median stays near 6 s.
  • Privacy. A private request never crosses to the dock, even when the box is busy. It waits in the queue. Step 3 shows the price: private requests wait longer.
  • Cost and latency. Cloud providers differ in price, first-token delay and speed. Step 5 swaps the provider. The routes stay the same, and only the meter rate changes.
  • Budget. Cloud spend has a monthly limit. When the day's share is spent, the dock shuts and requests wait for the box again (Step 4).

The prices in the figure are illustrative. Check the current prices before you plan a real budget.

One socket, one config

One small file turns these rules into settings. LiteLLM is one router that reads such a file. It serves the same OpenAI-compatible socket and maps each alias to an engine.

model_list:
  - model_name: private            # local only: no fallback is listed for it
    litellm_params: {model: ollama/qwen3:32b, api_base: "http://localhost:11434"}
  - model_name: fast               # local first
    litellm_params: {model: ollama/qwen3:32b, api_base: "http://localhost:11434"}
  - model_name: strong             # the cloud dock
    litellm_params: {model: provider/cloud-model, api_key: os.environ/CLOUD_API_KEY}
router_settings:
  timeout: 20                      # a full box counts as a failed call
  fallbacks: [{fast: [strong]}]
litellm_settings: {max_budget: 150, budget_duration: 30d}

The app names only an alias: private, fast or strong. A fallback fires when the local call fails or times out, so a short timeout turns a full box into a fallback. Other routers do the same job: OpenRouter, Ollama's cloud models, or a small Python shim.

Explain it
  1. "Why not send everything to the cloud?" Private data would leave your box, and the bill grows with every request. The local box answers most requests at no cost per request.
  2. "What changes in the app when the router changes?" Nothing. The app talks to one socket and names an alias.
  3. "Why do private requests wait longer?" They cannot use the dock, so they queue for the same few slots.

Exercises
  1. Set 16 users and find the lowest budget that keeps the dock open all day.
  2. Add a second cloud alias to the YAML that falls back from strong.
  3. Write a Python shim that sends prompts with a [private] tag to the local socket only.