Beyond Chat: RAG, Agents, Vision, Speech and Images

# /// script
# requires-python = ">=3.11"
# dependencies = ["numpy"]
# ///
import _local as local

import numpy as np

notes = [
    "Decode is memory-bandwidth bound: every token re-reads all active weights from VRAM.",
    "Prefill processes the whole prompt in one parallel pass and is compute bound; it sets time to first token.",
    "The KV cache stores keys and values per layer and token; grouped-query attention shrinks it.",
    "Q4_K_M is a GGUF k-quant of about 4.85 bits per weight and the usual quality/size sweet spot.",
    "NVFP4 uses E2M1 values in blocks of 16 with an FP8 scale and runs natively on Blackwell tensor cores.",
    "DGX Spark has 128 GB of unified LPDDR5X at 273 GB/s: huge capacity, modest bandwidth.",
    "The RTX 5090 has 32 GB of GDDR7 on a 512-bit bus at 1,792 GB/s.",
    "Continuous batching admits new requests at every decode step; PagedAttention stores KV in blocks.",
    "Partial offload to system RAM is slow because per-token time is the sum of the GPU and CPU reads.",
    "Mixture-of-experts models need memory for all experts but only read the active ones per token.",
    "Windows ML runs ONNX models and can use the TensorRT for RTX execution provider on NVIDIA GPUs.",
    "llama-server, Ollama, LM Studio and vLLM all expose an OpenAI-compatible chat completions API.",
]
emb_model = (local.embed_models() or [None])[0]
if emb_model:
    doc_vecs = np.array(local.embed(emb_model, [f"search_document: {n}" for n in notes]))
    doc_vecs /= np.linalg.norm(doc_vecs, axis=1, keepdims=True)

    def retrieve(question, k=3):
        q = np.array(local.embed(emb_model, [f"search_query: {question}"])[0]); q /= np.linalg.norm(q)
        scores = doc_vecs @ q
        top = np.argsort(-scores)[:k]
        return [(float(scores[i]), notes[i]) for i in top]

    for question in ["Why is my laptop slow when the model doesn't fit in VRAM?", "Which machine should run a 120B model?", "What format should I ship for Blackwell?"]:
        print("Q:", question)
        for score, text in retrieve(question):
            print(f"   {score:.3f}  {text}")
else:
    print("Run: ollama pull nomic-embed-text")

model = local.need_chat_model()
if model and emb_model:
    question = "Why is my laptop slow when the model doesn't fit in VRAM?"
    context = "\n".join(f"- {t}" for _, t in retrieve(question, 3))
    r = local.generate(model, f"Answer using only these notes:\n{context}\n\nQuestion: {question}\nAnswer:", num_predict=120, think=False)
    print(r["text"].strip())
    print(f"\nprompt {r['prompt_tokens']} tokens · prefill {r['prefill_s']*1000:.0f} ms · decode {r['decode_tps']:.0f} tok/s")

Q: Why is my laptop slow when the model doesn't fit in VRAM?
   0.665  Partial offload to system RAM is slow because per-token time is the sum of the GPU and CPU reads.
   0.616  Decode is memory-bandwidth bound: every token re-reads all active weights from VRAM.
   0.593  Windows ML runs ONNX models and can use the TensorRT for RTX execution provider on NVIDIA GPUs.
Q: Which machine should run a 120B model?
   0.566  The RTX 5090 has 32 GB of GDDR7 on a 512-bit bus at 1,792 GB/s.
   0.562  NVFP4 uses E2M1 values in blocks of 16 with an FP8 scale and runs natively on Blackwell tensor cores.
   0.555  Windows ML runs ONNX models and can use the TensorRT for RTX execution provider on NVIDIA GPUs.
Q: What format should I ship for Blackwell?
   0.552  NVFP4 uses E2M1 values in blocks of 16 with an FP8 scale and runs natively on Blackwell tensor cores.
   0.539  Continuous batching admits new requests at every decode step; PagedAttention stores KV in blocks.
   0.505  llama-server, Ollama, LM Studio and vLLM all expose an OpenAI-compatible chat completions API.
Your laptop is slow because the model does not fit in VRAM, which limits the amount of data that can be processed at once. Here's why:

1. **VRAM Limitation**: If the model's size exceeds VRAM capacity, each token would require reading all active weights from VRAM, which is memory-intensive. This causes the system to slow down as it needs to perform operations like memory reads and compute.

2. **Partial Offload to System RAM is Slow**: Even if partial offloading is done, the time required for GPU and CPU reads still adds up, making the process

prompt 106 tokens · prefill 28 ms · decode 102 tok/s