KV Cache

The Context Length article closed on a promise: a real engineering technique that makes processing long contexts faster and cheaper. That technique is the KV cache.

The simple definition

A KV cache stores the key and value vectors — calculated during self-attention, as covered in the Attention article — for every token already processed, so they don’t have to be recalculated from scratch every time a new token is generated. Recall from the Next-Token Prediction article that generating a full response means repeating the entire prediction process over and over, once per new token. Without a KV cache, each one of those repeated steps would have to recompute the keys and values for every single token already in the sequence — including tokens that were already fully processed several steps earlier — a massive, unnecessary amount of repeated work.

Why this repeated recalculation would be so wasteful

Recall from the Attention article’s query-key-value mechanism: every token has its own key and value vectors, calculated once from that token’s representation. Here’s the crucial insight the KV cache exploits — once a token’s key and value have been calculated, they never change again for the rest of that generation, since a decoder only looks backward, as covered in the Decoder article, and earlier tokens never get re-processed based on later ones. Recalculating the same, unchanging key and value vectors again and again at every single new step would be pure, avoidable waste — exactly the kind of redundant computation the KV cache is specifically built to eliminate.

flowchart LR
    A[Without KV Cache: recalculate keys/values for every token, every step] --> B[Massive redundant computation]
    C[With KV Cache: calculate each token's key/value once, reuse forever] --> D[Only the newest token needs fresh calculation]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a student solving a long, multi-part math problem, where each new part builds on earlier results. A wasteful approach would mean re-deriving every earlier intermediate result completely from scratch before tackling each new part. A KV cache is like keeping a running notebook of intermediate results already worked out — when a new part of the problem comes up, the student just looks up the previously calculated values instead of redoing that work, and only does fresh calculation for the genuinely new part.

Where this breaks down: A student’s notebook is a simple, static record. A KV cache is an actively growing structure in GPU memory, appended to with every single new token generated, and its size directly scales with how long the conversation gets — a real, physical memory cost, not just a metaphorical notebook, and one that genuinely competes for the same limited GPU memory the model’s own parameters need, as covered in the Parameters article’s discussion of memory and precision.

The real, genuine cost: KV cache memory grows with context length

This is worth being precise about, since it directly connects to the Context Length article’s cost discussion, and it’s a real, well-documented engineering challenge, not a minor footnote. The KV cache’s memory footprint grows linearly with context length — every additional token in the conversation adds its own key and value vectors to the cache, for every layer and every attention head, as covered in the Attention article’s discussion of GPT-3’s 96 layers and 96 heads. Published research on this exact problem has documented that for long contexts — 32,000 or 128,000 tokens — the KV cache alone can consume tens of gigabytes of GPU memory, in some cases growing large enough to become a bigger memory burden than the model’s own parameters. This is precisely why context window size, covered in the Context Window article, isn’t just a computational-cost trade-off — it’s also a genuine, hard memory-capacity constraint on the actual GPU hardware serving the model.

A concrete example, layered

For a simple beginner example: generating the sentence “The sky is blue today” one token at a time, a model calculates and caches the key and value for “The” once; when generating “sky,” it reuses “The“‘s cached key and value rather than recalculating them, and only computes fresh key and value vectors for “sky” itself — repeating this pattern, only ever doing fresh work for the newest token, all the way through the sentence. For a production example: published research on serving large language models has documented that, for a 7-billion-parameter model at a 128,000-token context length, the KV cache alone can require tens of gigabytes of memory under standard precision — a genuinely enormous figure that can exceed the memory capacity of even a high-end production GPU, which is exactly why real inference systems, as covered throughout current AI infrastructure research, actively invest in KV cache compression and management techniques to keep long-context serving practical and affordable.

Why this connects directly back to inference cost and speed

Recall from the Inference article’s discussion of prefill and decode: the KV cache is specifically what makes the decode stage fast. Without it, every single token generated during decode would require redoing the full attention calculation across the entire conversation so far — with it, decode only ever needs to calculate attention for the one newest token, reusing everything else already stored. This is a real, measured, substantial speedup, and it’s exactly why every serious production LLM serving system uses some form of KV caching as a standard, essential optimization, not an optional extra.

Watch the cache grow, one token at a time

Suppose the prompt is “The sky is” and the model generates “blue” and then “today.”

sequenceDiagram
    participant P as Prompt / new token
    participant M as Decoder layers
    participant C as KV cache
    P->>M: Prefill “The sky is”
    M->>C: Store K and V for The, sky, is at every layer
    P->>M: Decode blue
    C-->>M: Reuse earlier K and V
    M->>C: Append K and V for blue
    P->>M: Decode today
    C-->>M: Reuse The, sky, is, blue
    M->>C: Append K and V for today

Prefill: the model processes the supplied prompt and fills the cache. Many prompt positions can be calculated in parallel.
Decode: the model generates one new token at a time. It calculates the new token’s query, key, and value, attends to cached keys and values, and appends the new key and value.

The cache contains numeric tensors for each relevant decoder layer. It does not contain readable sentences, labels, model weights, or permanent memories.

A simplified memory estimate using GPT-3’s published dimensions

OpenAI’s GPT-3 paper lists 96 layers and a hidden size of 12,288 for the 175-billion-parameter model. Under a simplified standard multi-head estimate using 16-bit values:

KV bytes per token
= 2              key and value
× 96              layers
× 12,288          values per key or value across all heads
× 2 bytes         16-bit number
= 4,718,592 bytes ≈ 4.5 MiB per token

At 2,048 tokens: roughly 9 GiB for one sequence

This is a teaching estimate, not a claim about one serving system’s exact allocation. Real memory changes with data type, batch size, cache layout, quantization, and whether the model uses multi-head, grouped-query, or multi-query attention.

How Gemini reduces KV work

The Gemini 1.0 technical report says the family uses efficient attention mechanisms including multi-query attention. In ordinary multi-head attention, different query heads can keep separate key/value heads. Multi-query attention lets multiple query heads share key/value information, which can reduce KV-cache size and memory bandwidth during generation.

Many query heads + many separate K/V heads  → larger KV cache
Many query heads + shared K/V heads          → smaller KV cache

This optimization changes how much is stored, but not the learner’s core mental model: previously calculated keys and values are retained so the decoder does not repeat unchanged work.

KV cache versus API prompt caching

These two ideas sound similar but answer different questions:

Technique	Main purpose	Typical lifetime
Internal KV cache	Avoid recomputing earlier keys and values during token-by-token decoding.	At least the active generation; exact server handling varies.
API prompt or context caching	Reuse a repeated prompt prefix or uploaded context across separate API requests to reduce repeated processing, latency, or billed input cost.	Provider- and API-specific; may have retention rules.

During one answer, the decoder’s KV cache helps generate token 101 without recalculating unchanged key/value tensors for tokens 1–100. Across several separate questions about the same large document, provider context caching may let the service reuse processing associated with that repeated document.

Google’s current Gemini long-context guide recommends context caching when the same large context is reused many times and explains that it can reduce cost. OpenAI’s current model guidance describes explicit prompt caching for GPT-5.6. Those are user-facing API capabilities; they should not be confused with the decoder-level mechanism explained throughout this article.

When a KV cache helps—and when it does not

It greatly helps autoregressive decoding because earlier key/value tensors remain reusable.
It consumes more memory as context and batch size grow.
It may be invalidated or partly recomputed if the earlier prompt changes.
It does not correct a wrong answer, add new knowledge, or enlarge the model’s official context window.
It does not remove the sequential nature of generating the next token, then the next.

Long prompt + short answer → large prefill, relatively few decode steps
Short prompt + long answer → small prefill, many sequential decode steps
Many simultaneous users   → many caches competing for accelerator memory

This is why production inference systems manage KV-cache memory as a shared resource, not merely as a speed switch that can be turned on without cost.

Common misconception

A frequent beginner assumption: that the KV cache is some kind of memory of past conversations, similar to genuine long-term memory. As the Context Window article already cautioned, this isn’t the case — the KV cache is a purely computational optimization, scoped entirely to the current generation session and the current context window; it speeds up processing within one active conversation, but it has nothing to do with remembering information across separate, distinct conversations, and it gets discarded once that specific generation session ends.

Closing out this phase

This article completes the Transformers phase, and it’s worth tracing the full architecture it assembled: the Transformer replaced sequential processing with parallel, attention-based processing, split into an Encoder (full bidirectional understanding) and a Decoder (backward-looking, autoregressive generation, the half most modern LLMs actually use). Attention, specifically Self-Attention run in parallel as Multi-Head Attention, explained exactly how tokens weigh their relevance to each other through queries, keys, and values, while the Feed-Forward Network processed each token’s enriched representation individually. Residual Connections and Layer Normalization kept that stack of layers actually trainable, and Positional Encoding restored the sense of order that parallel processing would otherwise lose. Context Window and Context Length explained the fixed ceiling and current usage of how much a model can process at once, and the KV cache closes the loop by explaining the practical optimization that makes generating long responses within that context genuinely fast and feasible. From here, the glossary is positioned to move into the specialized techniques — fine-tuning, RAG, and agentic systems — built on top of this complete architectural foundation.

In one sentence

A KV cache stores each token’s already-calculated key and value vectors so they never need to be recomputed, turning what would otherwise be massive, repeated computation into a fast, incremental process — the essential, standard optimization that makes real-time, long-context text generation practically affordable.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026