TechByteByByte

VRAM

The specific number that determines whether a model runs at all — and the practical wall that gave birth to the entire quantization movement, letting billion-parameter models run on ordinary laptops.

#vram#gpu#memory#infrastructure-serving-phase

The CUDA article covered the software layer that lets a GPU actually run AI computation. This article covers the specific, practical resource that most often determines whether that computation can happen at all: VRAM.

The simple definition

VRAM, or Video RAM, is the memory built directly onto a GPU, used to store everything the GPU is actively working with — including, for AI, a model’s entire set of parameters. Recall from the Parameters article’s discussion of scale — a model like GPT-3 has 175 billion individual weight values. Every one of those values has to physically live somewhere the GPU’s cores can access instantly while computing, and that “somewhere” is VRAM, not a computer’s regular system memory, which is far too slow for a GPU’s cores to use directly during computation.

Why this specific number becomes the practical bottleneck for everything else

Recall from the GPU article that a GPU’s power comes from thousands of cores computing in parallel. All of those cores need fast, immediate access to the data they’re working on — recall from the Weights article’s earlier coverage of stored, real numeric values. If a model’s full set of parameters, plus the intermediate calculations happening during a forward pass, doesn’t fit within a GPU’s VRAM, the model simply cannot run on that GPU at all, regardless of how fast or capable its cores otherwise are.

flowchart LR
    A[Model's parameters + intermediate calculations] --> B{Does it fit in available VRAM?}
    B -->|Yes| C[Model runs]
    B -->|No| D["Out of memory error — model cannot run on this hardware at all"]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a chef’s countertop workspace — no matter how skilled or fast the chef is, if a recipe’s ingredients and tools genuinely don’t fit on the available counter space, the chef physically cannot prepare that dish on that counter, no matter how talented they are. A bigger, faster chef doesn’t help if the counter itself is too small.

Where this breaks down: A chef could, in principle, use a slightly smaller counter with some clever rearranging. A GPU’s VRAM limit is a hard, physical constraint, with no flexible workaround beyond either using a bigger GPU, splitting the model across multiple GPUs, or reducing how much space the model itself actually needs, which is precisely the motivation behind quantization, covered later in this phase.

Why VRAM, not raw processing speed, is often the real limiting factor

This is worth being direct about, since it’s a genuinely common, practical surprise for anyone actually working with these models. A consumer GPU might have plenty of raw computational power but only 8 or 12 gigabytes of VRAM — nowhere near enough to hold a 70-billion-parameter model’s weights, which, stored at standard precision, would require roughly 140 gigabytes just for the parameters alone, before accounting for any of the additional memory needed during actual computation.

This mismatch between available compute and available memory is exactly why so much practical AI engineering effort, covered throughout the rest of this phase, focuses specifically on reducing memory footprint rather than simply seeking more raw processing power.

A concrete example, layered

For a simple beginner example: a hobbyist trying to run a 7-billion-parameter open-weight model on a laptop with a modest GPU quickly discovers that the model simply won’t load at all if the laptop’s VRAM is too small — not a slowdown, but a hard failure, regardless of how patient the user is willing to be.

For a production example: Nvidia’s H100 GPU, referenced throughout this phase, ships with 80 gigabytes of VRAM specifically because modern frontier models require enormous memory just to hold their parameters, and real production deployments of the largest models often need to split a single model’s weights across many GPUs’ VRAM simultaneously, precisely because no single chip’s memory is large enough to hold the entire thing alone.

Why this exact constraint gave birth to an entire, real movement

It’s worth being direct about the genuine, practical consequence this specific limitation produced. The gap between “how much VRAM ordinary people actually have” and “how much VRAM a full-precision large model needs” is precisely the real, documented motivation behind the quantization techniques covered later in this phase — real, published tools like llama.cpp were built explicitly to shrink a model’s memory footprint enough to fit within the VRAM of an ordinary consumer laptop, turning a hard, physical wall into a solvable engineering problem.

Common misconception

Add the KV cache to the estimate

Imagine that FP16 weights consume 14 GB on a 24 GB GPU. The remaining 10 GB is not all free for conversation text:

24 GB total VRAM
-14 GB model weights
- 2 GB runtime workspace and temporary tensors   (illustrative)
= 8 GB left for KV cache and active requests

The 2 GB number is only an example; actual workspace changes by model and serving engine. The important idea is that the model file is only one occupant of VRAM.

Longer prompts, longer generated answers, larger batches, and more simultaneous users grow the KV cache. When memory becomes full, the server may reject requests, shorten allowed context, move some data to slower memory, or split work across additional GPUs.

What actually occupies VRAM

VRAM
├── Model weights
├── Temporary activations
├── KV cache for active token sequences
├── Input and output tensors
└── Framework workspace and memory overhead

During training, VRAM must additionally hold gradients and optimizer state. Training therefore usually needs much more memory than inference for the same model.

Calculate a first estimate

A model with 7 billion parameters needs approximately this much memory for weights alone:

Weight formatApproximate calculationWeight memory
FP327 billion × 4 bytes28 GB
FP16 or BF167 billion × 2 bytes14 GB
INT87 billion × 1 byte7 GB
INT47 billion × 0.5 byte3.5 GB

Real usage is higher because the KV cache, temporary calculations, quantization metadata, and software also require memory.

A current hardware comparison

One NVIDIA H200 has 141 GB of HBM3e. That does not mean a 140 GB model safely fits with room for real traffic. The inference server needs spare memory for active requests, and longer contexts make the KV cache grow.

When one GPU is insufficient, serving systems can split a model across several GPUs. This increases total available memory but introduces communication overhead between devices.

Verified sources

A frequent beginner assumption: that running a large AI model on a given computer is primarily a question of processing speed — “will it be slow” rather than “will it work at all.” As this article’s out-of-memory example demonstrated, VRAM is frequently the actual, binary gatekeeper — a model with insufficient available memory doesn’t run slowly, it typically fails to load at all, making VRAM capacity often the single most important hardware specification for anyone planning to run a specific model locally.

Where this fits in what comes next

You now understand the specific, physical memory constraint underlying nearly every practical decision covered throughout the rest of this phase. The next article, Inference Server, covers the software system responsible for actually managing this constraint efficiently in a real, production environment serving many users at once.

In one sentence

VRAM is the memory built directly onto a GPU, and because a model’s entire set of parameters has to fit within it to run at all, VRAM capacity is frequently the actual, hard limiting factor in AI hardware — a genuine, physical constraint that directly motivated the entire quantization movement covered later in this phase.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed