Why a GPU

The workload is a matmul

A neural network is mostly matrix multiplication, and each output is independent. A linear layer multiplies activations X \in \mathbb{R}^{M\times K} by weights W \in \mathbb{R}^{K\times N}. Each of the M \cdot N outputs is a K-long dot product, for 2MNK FLOPs in total. The GPU can compute all outputs in parallel. A transformer layer holds a few such matrices (Q, K, V and output projections, and the MLP). Attention adds two batched matmuls.

• CPU: 8–24 fast cores for branchy, serial code, with wide SIMD units. A good desktop CPU reaches a few TFLOPS at FP32.
• GPU: dozens to ~200 streaming multiprocessors (SMs). Each SM runs threads in lockstep groups of 32 (warps). Tensor cores multiply a small tile of numbers in one instruction. A GeForce RTX 5090 has 170 SMs.

Precision is a throughput dial

Fewer bits per number give more operations per cycle on tensor cores. Each GPU generation adds a smaller format:

Vendors quote "AI TOPS" at the lowest precision with 2:4 structured sparsity. This number is roughly 2× a dense number and 4× an FP8 dense number. Compare devices at the same precision and the same sparsity setting.

Lab: matrix multiply on CPU vs GPU

Time large square matmuls on the CPU and the GPU, and report TFLOPS. PyTorch runs on the CPU and on this Mac's GPU (Metal Performance Shaders, mps). On an NVIDIA machine, the same code uses cuda.

# /// script
# requires-python = ">=3.11"
# dependencies = ["numpy", "torch"]
# ///
import _local as local

import time, torch

def bench(device, n=4096, dtype=torch.float32, repeats=10):
    a = torch.randn(n, n, device=device, dtype=dtype)
    b = torch.randn(n, n, device=device, dtype=dtype)
    sync = {"cuda": torch.cuda.synchronize, "mps": torch.mps.synchronize}.get(device, lambda: None)
    for _ in range(3):
        a @ b
    sync()
    start = time.perf_counter()
    for _ in range(repeats):
        a @ b
    sync()
    seconds = (time.perf_counter() - start) / repeats
    return 2 * n**3 / seconds / 1e12

devices = ["cpu"] + (["mps"] if torch.backends.mps.is_available() else []) + (["cuda"] if torch.cuda.is_available() else [])
results = {}
for device in devices:
    for dtype in (torch.float32, torch.float16):
        if device == "cpu" and dtype == torch.float16:
            continue
        results[(device, str(dtype).split(".")[-1])] = bench(device, dtype=dtype)
        print(f"{device:>4} {str(dtype).split('.')[-1]:>8}: {results[(device, str(dtype).split('.')[-1])]:6.2f} TFLOPS")

cpu  float32:   1.10 TFLOPS
 mps  float32:   1.96 TFLOPS
 mps  float16:   2.74 TFLOPS

What to notice:

• Even this Mac's integrated GPU beats its CPU cores on large matmuls, and FP16 beats FP32.
• A discrete RTX GPU adds one to two orders of magnitude of tensor throughput.
• Chapter 3 shows that, for chat, this compute mostly sits idle.

Why NVIDIA

The software matters as much as the silicon.

• CUDA and its libraries (cuBLAS, cuDNN, CUTLASS, NCCL), plus kernels such as FlashAttention.
• TensorRT, TensorRT-LLM and TensorRT for RTX (inference compilers), and NIM (packaged model microservices).
• llama.cpp, Ollama, vLLM, SGLang, PyTorch and ONNX Runtime all treat NVIDIA GPUs as a first-class target. Windows ML can route models through TensorRT for RTX.

A model built on this stack runs well on the largest installed base of AI-capable GPUs.