Why does AI need a GPU?
GPUs: thousands of cores and fast memory for matrix multiplies · Beginner · 5 min · llm, transformer · after Model memory: parameters × bytes, plus the KV cache
A GPU runs AI fast because a model is mostly matrix multiplication: billions of multiply-adds that don’t wait for each other. It has thousands of cores to do them at once and memory that feeds weights far faster.
1 Guess first
2 Look
Same sum, many hands
1 / 3Multiply two 4,096 × 4,096 grids of numbers
68,719,476,736 multiply-adds · every one of the 16,777,216 answer cells is a row times a column
- CPU, 1 core1.43 s
- CPU, all 4 cores421 ms
- T4, 32-bit peak17 ms
- T4, 16-bit peak2.1 ms
The T4 at its 16-bit peak: about 199× the whole laptop CPU.
CPU: Intel Core i7-6820HQ, numpy + OpenBLAS, float32, measured 2026-10-23 · T4: NVIDIA datasheet peak (a ceiling, not a measurement)
The laptop takes 0.42 s for 4,096 × 4,096. At its peak, a T4 would take about 2 ms: some 200 times less.
Every answer cell is a row times a column that waits for nothing, so thousands of simple cores beat a few fast ones.
Almost all of a model’s work is matrix multiplication (rows times columns, added up). Each new token costs about 2 operations per weight: 15 billion for a 7.6B model. No answer cell waits for another, so the work splits across any number of cores. A CPU has a few clever cores. A GPU has thousands of simple ones.
4 You are the model
You are one core. Work out the four cells of a tiny matrix multiply, one at a time, the way a single core would.
you are one core · each answer cell = a row of A times a column of B, added up
A = [1 2 / 3 4] · B = [5 6 / 7 8]
C = [? ? / ? ?]
Row 1 of A × column 1 of B = ?
A 7B model needs about 7.6 billion multiply-adds per token.
You are one core. Work out each cell of A × B.5 Check
Speed has two limits. Reading your prompt is compute: all its tokens go through at once. Writing the answer is memory: each new token reads every weight again. This laptop moves 15 GB a second, a T4 300. So quantization (fewer bits per weight) speeds up chat. VRAM decides what fits; LoRA makes training fit.
7 Say it back
8 Mission · 5 min
In a free Kaggle notebook (phone-verified), set Accelerator to “GPU T4 x2” and run: import torch, time; a = torch.randn(4096, 4096); t = time.time(); a @ a; print(time.time() - t). Then g = a.cuda().half(); g @ g; torch.cuda.synchronize(); t = time.time(); g @ g; torch.cuda.synchronize(); print(time.time() - t). How many times faster was the GPU?
Three quick ones
Drawn at random from this lesson's pool; different next time.
Done?
Go deeper
Sources
- CPU times: scripts/explain-data/gpus.py on 2026-10-23, Intel Core i7-6820HQ (4 cores, 8 threads, 2.7 GHz), Python 3.15.0, numpy 2.5.4 with scipy-openblas 0.3.34; float32 N × N matmul of standard-normal matrices (seed 0), one warm-up then the median of 5 runs (time.perf_counter), all cores (OpenBLAS default) and one core (OPENBLAS_NUM_THREADS=1), GFLOP/s = 2N³ / seconds. N = 4096: 0.421 s (326 GFLOP/s) all cores, 1.425 s (96.5) one core; N = 256 to 2048 in app/components/explain/lessons/gpus-data.ts
- Same script: plain Python triple loop, N = 64, median of 3: 0.0203 s (0.026 GFLOP/s); numpy.copyto of 1 GiB float32, median of 5: 15.0 GB/s (2 GiB read + written)
- NVIDIA T4 Tensor Core GPU datasheet (© 2019, PDF dated 2019-03; https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/tesla-t4/t4-tensor-core-datasheet-951643.pdf, checked 2026-10-23): 320 Turing Tensor Cores, 2,560 CUDA cores, 8.1 TFLOPS single precision, 65 TFLOPS mixed precision (FP16/FP32), 16 GB GDDR6, 300 GB/sec, 70 W. T4 times are 2N³ divided by these peaks: ceilings, not measurements
- Operations per token ≈ 2 × parameters for the forward pass (Kaplan et al. 2020, https://arxiv.org/abs/2001.08361, C_forward ≈ 2N plus an attention term ignored here); Qwen2.5 7B 7,615,616,512 parameters (lora lesson, safetensors headers) → 15.2 GFLOP per token, 15.23 GB at 16 bits; q4_K_M 4.7 GB (Ollama qwen2.5:7b-instruct, as in the vram lesson)
- Chat arithmetic: prompt of 1,000 tokens = 15.2 TFLOP → 47 s at 326 GFLOP/s, 0.23 s at 65 TFLOPS; one written token reads every weight once → 15.23 GB ÷ 15.0 GB/s = 1.0 s and ÷ 300 GB/s = 51 ms (4-bit: 0.31 s, 16 ms). Upper bounds; KV cache and overheads not counted
- aifree.dev catalog, kaggle-free-gpu-tpu (verified 2026-10-12): up to 30 hours/week GPU; “Accelerators: NVIDIA Tesla P100 (16GB), 2x NVIDIA T4, or TPU v3-8”; phone verification required
Published Oct 23, 2026 · updated Oct 23, 2026