The Sequence and Positional Encoding articles both mentioned “the context window” without stopping to define it fully. It’s time to make that precise: the context window.
The simple definition
The context window is the maximum number of tokens a model can process and attend to at once — the entire span of text it can actually “see” when generating a response. Recall from the Self-Attention article that every token attends to every other token in the sequence; the context window is the outer boundary of that sequence — everything within it is visible to the model’s attention mechanism, and anything beyond it simply doesn’t exist as far as the model’s current calculation is concerned.
Why this limit exists at all, and why it’s not arbitrary
Recall from the Attention article that self-attention compares every token’s query against every other token’s key — meaning the amount of computation required grows rapidly as the sequence gets longer, since every added token has to be compared against every other token already present. This isn’t a minor inconvenience — it’s a genuine, well-documented computational constraint of the self-attention mechanism itself, which is exactly why context windows have real, fixed limits rather than being unlimited by default. A model’s context window size is set as part of its architecture and training, balancing how much context is genuinely useful against the real computational cost of processing that much text at once.
flowchart LR
A[Tokens within the context window] --> B[Visible to self-attention, fully considered]
C[Tokens beyond the context window] --> D[Invisible to the model, effectively don't exist]ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a whiteboard used during a long meeting. No matter how much gets discussed, only what’s currently written on the board is visible to everyone in the room right now — older points, once erased to make room for new ones, are simply gone from view, even if someone technically said them earlier in the meeting.
Where this breaks down: A whiteboard’s limit is a simple matter of physical writing space. A context window’s limit comes from the real computational cost of self-attention, described above — a precise, engineered trade-off between capability and compute cost, not a simple space constraint, and unlike an erased whiteboard, what happens to information beyond a model’s context window (whether it’s dropped, summarized, or handled some other way) depends entirely on how the specific application built around the model chooses to manage it.
What actually happens when a conversation exceeds the context window
This is a genuinely practical detail worth understanding, since it explains something many people notice using chatbots directly. Once a conversation’s total token count — recall from the Token article that pricing and limits are measured in tokens, not words — approaches or exceeds a model’s context window, the application built around the model has to make a choice: drop the oldest messages, summarize earlier parts of the conversation, or refuse to continue without the user trimming the input. This is exactly why a very long conversation with a chatbot can sometimes seem to “forget” something discussed much earlier — that earlier content may have genuinely fallen outside the current context window.
A concrete example, layered
For a simple beginner example: a model with a context window of 100 tokens simply cannot process a 500-token document all at once — the application has to break that document into smaller chunks (echoing the retrieval-based chunk approach covered in the Vector Database article’s RAG discussion) or otherwise manage what fits. For a production example: GPT-4 is documented as supporting context windows of up to 128,000 tokens in some of its variants, exactly as noted in the Sequence article — roughly the length of a lengthy book chapter — meaning a user could paste in a substantial document and have the model consider the entire thing simultaneously, something a much smaller context window simply couldn’t accommodate.
Why a bigger context window isn’t automatically a free win
It’s worth being honest about a genuine trade-off here, not presenting bigger context windows as an unambiguous improvement. A larger context window means more computation per token generated, since self-attention has to compare against a longer history — directly connecting to the computational cost concern raised earlier in this article. Research has also documented that models don’t always use every part of a very long context equally well — information in the middle of a very long input can sometimes get less effective attention than information near the beginning or end, a real, actively studied phenomenon sometimes described informally as models being better at “remembering” the start and end of a long context than its middle.
What can occupy the window?
The user’s visible message is only one part of the material a real AI application may send to a model. Depending on the application, the context can also contain instructions, earlier messages, retrieved documents, tool results, images represented in the model’s internal format, and room for the answer.
flowchart LR
A[System and developer instructions] --> W[Available context]
B[Conversation history] --> W
C[Current user input] --> W
D[Retrieved documents and tool results] --> W
W --> E[Model generates output]The exact accounting is model- and API-specific. The important mental model is that everything supplied for the current calculation consumes capacity; a long retrieved document does not become free just because the user did not type it.
Context window, memory, and learned knowledge are different
| Idea | What it means |
|---|---|
| Context window | The maximum working space available for one model calculation. |
| Conversation memory | Information an application saves and may place back into a later context. |
| Model knowledge | Patterns stored in learned parameters during training, not a searchable copy of the current chat. |
A chatbot can appear to remember an old preference because the application saved it and inserted it into today’s context. That does not make the context window itself permanent memory.
Real GPT and Gemini context examples
Context limits have increased rapidly. The older GPT-4 API model lists an 8,192-token context window. OpenAI’s current model catalog lists much larger windows for newer models, showing why the exact model name and version matter rather than saying only “GPT.”
Google’s Gemini 1.5 technical report studied context lengths extending into millions of tokens and reported greater than 99% retrieval in controlled tests up to at least 10 million tokens. That result demonstrates long-context capability under a particular test; it does not mean every fact in every ten-million-token real document will always be used perfectly.
Larger window
├── can hold more conversation, code, audio, or documents
├── can reduce how often an application must split input
└── can require more memory, computation, time, and careful retrievalCurrent real-world examples and what fits
As of August 2026, OpenAI’s model catalog lists a 1.05-million-token context window and 128,000 maximum output tokens for GPT-5.6 Sol, Terra, and Luna. Google’s current long-context guide says many Gemini models provide one million or more tokens of context.
Google gives useful intuition for one million tokens: approximately 50,000 lines of code, eight average-length English novels, or transcripts from more than 200 average-length podcast episodes. These are rough comparisons because tokenization varies with language and content.
Example: ask questions about a software repository
Repository files 620,000 tokens
Architecture documents 110,000 tokens
Instructions and question 10,000 tokens
Reserved answer space 20,000 tokens
-------
Planned total 760,000 tokensThis may fit within a one-million-token window, but “fits” does not guarantee “works well.” Irrelevant files add cost, increase time to the first generated token, and may distract the model. Retrieval and filtering still matter even when the entire repository technically fits.
Google’s guide reports that one-million-token context enabled an in-context translation experiment using a 500-page grammar, a dictionary, and about 400 example translations for Kalamang, a language with fewer than 200 speakers. This is a concrete example of information being supplied in the context for a task rather than added to model weights through training.
Common misconception
A frequent beginner assumption: that a large context window means a model has genuine long-term memory, remembering past conversations the way a person would. As this article has explained, a context window only covers the current sequence being processed — once a conversation ends, or falls outside the current window, that information is gone from the model’s active consideration entirely, unless a separate system (like a saved chat history, reloaded as part of a new context window) is specifically built to reintroduce it.
Where this fits in what comes next
You now understand the context window as the outer boundary of what a model can actually attend to at once. The next article, Context Length, covers a closely related term worth distinguishing carefully — the actual, current size of a specific piece of text being processed, as opposed to the model’s maximum architectural limit.
In one sentence
The context window is the maximum number of tokens a model can attend to at once — a real, engineered limit rooted in the computational cost of self-attention, not an arbitrary restriction, and it directly determines how much of a conversation or document a model can genuinely consider at any given moment.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed