Every phase of this glossary so far has explained the mathematics and architecture of AI models. This final phase turns to the physical hardware and software those models actually run on, starting with the chip that made the entire deep learning era possible: the GPU.
The simple definition
A GPU, or Graphics Processing Unit, is a chip originally designed to render video game graphics, built around doing thousands of simple calculations simultaneously rather than a few complex ones quickly. Recall from the Neural Networks phase that a model’s core operation — a weighted sum across many inputs — is fundamentally the same kind of repeated, simple arithmetic used to calculate the color of every pixel on a screen.
This shared mathematical shape is why a chip built for rendering video games turned out to be almost perfectly suited for training neural networks, decades before anyone designed a chip specifically for AI.
Why “many simple calculations at once” matters so much for AI
Recall from the Matrix and Vector articles, back in the Data Representation phase, that neural network computation is fundamentally matrix multiplication — the same operation, repeated across millions of numbers. A regular CPU is built to handle a few complex, varied tasks very quickly, one after another, in strict sequence — genuinely excellent for running an operating system or a web browser, since those tasks are unpredictable and interdependent.
A GPU instead has thousands of smaller, simpler processing cores, all doing the exact same simple operation on different pieces of data at the same time — precisely the structure matrix multiplication needs, since every entry in the result can be calculated independently and simultaneously.
flowchart LR
A[CPU: few powerful cores, sequential] --> B[Good for varied, unpredictable tasks]
C[GPU: thousands of simple cores, parallel] --> D[Good for the same simple math, repeated millions of times]ANALOGY vs. TECHNICAL REALITY
Analogy: Think of the difference between one master chef preparing an elaborate, multi-step dish entirely alone, versus a thousand line cooks each chopping one single vegetable at the same time. The master chef handles complexity brilliantly but works through steps one at a time. The thousand line cooks can’t handle a complex, varied task individually, but for a job that’s genuinely the same simple action repeated a million times, they finish overwhelmingly faster together.
Where this breaks down: Real line cooks would need to communicate and coordinate. A GPU’s thousands of cores execute the exact same instruction on different data completely independently, with the coordination handled by the chip’s own architecture and the CUDA software layer covered in the next article — a far more mechanical, rigid form of parallelism than any real kitchen team could achieve.
The real, genuinely pivotal moment this hardware made possible
Recall directly from the Computer Vision article’s AlexNet story, back in the Multimodal AI phase. AlexNet, the 2012 model that triggered the modern deep learning revolution with its 10-percentage-point ImageNet accuracy jump, was trained using two consumer GPUs — not specialized AI hardware, which didn’t yet exist, but the same kind of graphics cards sold for playing video games. This wasn’t a coincidence; it was the first widely celebrated proof that GPU parallelism, originally built for rendering, could make training genuinely large neural networks practical within days rather than months.
A concrete example, layered
For a simple beginner example: training even a small neural network to recognize handwritten digits runs dramatically faster on a GPU than a CPU, since the network’s core weighted-sum calculations, covered throughout the Neural Networks phase, can be split across thousands of GPU cores simultaneously. For a production example: Nvidia’s H100 and A100 GPUs, real, widely deployed data-center chips, are the standard hardware behind training and running most of the frontier language models referenced throughout this glossary — GPT, Claude, Llama — with a single H100 containing tens of thousands of smaller processing cores working in parallel.
Why GPUs aren’t automatically the right choice for everything
It’s worth being honest about a genuine trade-off here, not presenting GPUs as universally optimal. A GPU’s advantage is entirely about parallel, repetitive computation — for a task that’s genuinely sequential, or too small to meaningfully parallelize, a GPU’s individual cores are actually weaker and simpler than a CPU’s, and the overhead of coordinating thousands of them can make a GPU slower than a CPU for the wrong kind of job. This is exactly why every computer still has a CPU alongside any GPU, rather than GPUs having replaced CPUs entirely.
A real, disruptive lesson in what else wants this same parallel hardware
It’s worth knowing this story, because it shows AI isn’t the only field that discovered GPUs’ parallel math was valuable, with real, disruptive consequences for ordinary consumers.
Cryptocurrency mining, particularly for Ethereum, turns out to rely on the exact same kind of repetitive, parallel arithmetic covered throughout this article — and by 2021, miners buying up GPUs at scale had driven secondary-market prices as much as 300% above retail, making it genuinely difficult for ordinary gamers to buy a graphics card at all.
Nvidia responded in May 2021 by releasing “Lite Hash Rate” (LHR) cards, deliberately crippled in software specifically to reduce their mining efficiency while leaving gaming performance untouched — an unusual, direct intervention into its own product line. Miners cracked the restriction within months using tools like NiceHash’s QuickMiner, and Nvidia later faced a class-action lawsuit alleging it had misled investors about how much of its revenue actually depended on mining demand.
The whole episode is a genuine, real illustration that the parallel hardware this article describes is valuable well beyond AI, and that value can create real, disruptive shortages for everyone else who needs the same chips.
Common misconception
Training and token generation use the GPU differently
During training, the GPU processes many tokens and examples together. This creates large blocks of parallel matrix work and usually keeps the GPU’s arithmetic units busy.
During LLM generation, the next token depends on tokens already produced:
token 1 -> token 2 -> token 3 -> token 4The operations inside each step still run in parallel, but the steps themselves happen in order. The GPU may spend substantial time moving model weights and KV-cache data from VRAM rather than performing arithmetic. This is called being memory-bandwidth bound.
Plain meaning: a GPU needs both fast calculators and a very fast road carrying numbers to those calculators.
Follow one AI calculation through a GPU
Suppose a layer receives 1,000 numbers and must combine them with 1,000 learned weights for many nodes. A CPU can do this work, but a GPU can divide thousands of similar multiply-and-add operations among many processing units.
Input vectors + weight matrix
|
v
Thousands of parallel multiply-and-add operations
|
v
Output vectors for the next layerThe GPU does not decide what the answer means. The model architecture and weights define the calculation; the GPU performs the calculation quickly.
GPU, VRAM, and CUDA are different pieces
| Piece | Job |
|---|---|
| GPU | Performs large amounts of parallel arithmetic |
| VRAM | Holds weights, activations, and the KV cache close to the GPU |
| CUDA | Lets software use supported NVIDIA GPU capabilities |
A current production example
NVIDIA’s H200 provides 141 GB of HBM3e memory with 4.8 TB/s of memory bandwidth. NVIDIA reports nearly twice the memory capacity and 1.4 times the memory bandwidth of H100 for this comparison. Larger, faster memory helps an inference server hold more model data and active requests close to the processors.
This does not mean H200 is always faster by the same amount. Performance changes with the model, numerical format, batch size, prompt length, serving software, and number of connected GPUs.
Verified sources
A frequent beginner assumption: that a GPU is simply a “faster CPU,” a general performance upgrade with no real trade-offs. As this article has explained, a GPU is architecturally specialized for a genuinely different kind of workload — massive, uniform parallelism — and trades away the CPU’s flexibility for varied, sequential tasks to gain that speed, making it a complement to a CPU, not a strict replacement.
Where this fits in what comes next
You now understand the hardware foundation nearly all modern AI training and inference runs on. The next article, TPU, covers Google’s own, purpose-built alternative — a chip designed from scratch specifically for AI, rather than repurposed from graphics rendering, born from a genuine, real crisis inside Google’s own data centers.
In one sentence
A GPU’s thousands of simple, parallel processing cores happen to match the exact mathematical shape of neural network computation, and this originally accidental fit — proven decisively when AlexNet trained on two consumer graphics cards in 2012 — is the specific hardware foundation the entire modern deep learning era was built on.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed