TechByteByByte

Context Window

Understand what a context window actually is, why LLMs have hard limits on it, what happens when a conversation grows too large, and the direct relationship between context windows and RAG — with a verified token-budget and truncation example.

#LLM#AI#Context Window#Context Management#RAG

Before you continue: three tools for this module

  • Context: all tokens visible to the current model call.
  • Context window: the maximum token budget the model can process.
  • Truncation: removing content when it does not fit.

You do not need to memorize these yet. Use this map when the terms reappear.

Begin with the central question

What hidden problem does Context Window solve inside a real language-model system?

Keep that central question about Context Window in mind. The definitions, numbers, diagrams, and code examples below answer it one piece at a time.

instructions + conversation + retrieved text + output tokens → limited context budget

1. What You Will Learn

Learning outcomes

  • Define the context window and identify everything that consumes its token budget.
  • Explain what happens when content is truncated or omitted.
  • Compare truncation, summarization, retrieval, and memory strategies.
  • Reason about context length, attention cost, KV-cache memory, and answer quality.

In one sentence

💡 Big picture

The context window is the model’s working desk: only the tokens that fit on the desk are visible during the current request.

2. Why This Module Exists

The problem this module solves

  • Long conversations, documents, instructions, and generated answers all compete for limited space.
  • If important information falls off the desk, the model cannot use it unless the application adds it again.

3. Intuition

the context window is the model’s entire field of view for a single request — everything it can “see” and reason about at once: system instructions, conversation history, retrieved documents, and the current message. Nothing outside this window exists to the model at all, for that specific forward pass — not “forgotten,” simply never provided.

Analogy: The Desk Workspace Limit & Truncation Folders Think of managing your inputs within the context window in terms of office desk space:

  • The Desk (The Context Window): You are writing an essay at a tiny office desk that can only fit 50 index cards (the token limit). Everything you need to write the next sentence must be laid out on this desk: the instructions manual (system prompt), notes from yesterday (history), and research documents (RAG).
  • The Spill (Over limit): If a colleague dumps a massive new folder of documents (a long user prompt) on your desk, and it doesn’t fit, you are physically blocked.
  • The Truncation Strategy: To make room, you take the oldest index cards from yesterday, pack them into a box under the desk (evicting old turns), and leave only the most recent conversations on the desk. You can no longer see what was on those early cards — they are out of your field of view.

📊 Visual Flowchart: Context Window Allocation & Truncation Sliding Window

Here is how history is managed to preserve system rules and recent turns under tight budgets:

graph TD
    classDef system fill:#e74c3c,stroke:#333,stroke-width:1px,color:#fff;
    classDef recent fill:#2ecc71,stroke:#333,stroke-width:1px,color:#fff;
    classDef dropped fill:#7f8c8d,stroke:#333,stroke-width:1px,color:#fff;

    subgraph ContextLimit ["Total Context Window (e.g., 50 tokens max)"]
        Sys["1. System Prompt Rules (9 tokens)<br>(Always preserved at index 0)"]:::system

        subgraph Evicted ["Dropped Context"]
            Turn1["Turn 1: User Password query (6 tokens)"]:::dropped
            Turn2["Turn 2: Assistant Password steps (8 tokens)"]:::dropped
        end

        subgraph Kept ["Active Context Workspace"]
            Turn3["Turn 3: User SMS query (9 tokens)"]:::recent
            Turn4["Turn 4: Assistant SMS steps (9 tokens)"]:::recent
            Turn5["Turn 5: User Phone change (6 tokens)"]:::recent
            NewTurn["New Prompt: Policy details query (22 tokens)"]:::recent
        end
    end

    Evicted -.->|Exceeds 50 tokens limit| Dropped["Discarded from request payload"]
    Kept --> SendModel["Forward pass executed successfully"]

4. Core Concept

Context window = the MAXIMUM number of tokens (input + output
                  combined, or sometimes input-only depending on
                  the provider's specific accounting) a model can
                  process in a single request
TermDefinition
Input tokensTokens in the prompt sent to the model
Output tokensTokens the model generates in response
Context lengthThe total token capacity — input + output must fit within this
Long-context modelsModels specifically trained/engineered to support much larger context windows (hundreds of thousands of tokens)

5. Why Context Windows Have Limits

You already covered this precisely in the Transformers course: attention computation scales quadratically with sequence length — doubling context length roughly quadruples the core attention computation cost, and the KV cache (Transformers course) grows with context length too, consuming real GPU memory.

A context window isn’t an arbitrary business restriction — it’s a direct consequence of these compute and memory costs, set at a level the serving infrastructure can actually support.

6. How It Works — Step by Step

1. Every request's TOTAL tokens (system prompt + conversation
   history + any retrieved context + the current message, PLUS
   however many tokens the model is asked to generate) must fit
   within the context window
2. If assembled input alone exceeds the limit, the request FAILS
   or must be TRUNCATED before sending
3. As a conversation grows turn by turn, previously-fine requests
   can eventually exceed the limit -- requiring a context
   management strategy: truncate the oldest turns, summarize
   older history, or retrieve only the most relevant prior
   context (directly connecting to RAG)

7. Mathematical Intuition

Read the mathematics as a story

instructions + conversation + retrieved text + output tokens → limited context budget

First locate the input, operation, and output. Then treat the formula as a compact description of that journey rather than a collection of symbols to memorize.

The core trade-off, precisely: context_used = tokens(system prompt) + tokens(conversation history) + tokens(retrieved context) + tokens(current message). This total must stay <= context_limit - tokens_reserved_for_generation. As any component grows, something else must shrink or be dropped — verified directly below with a real tracked example.

8. Small Worked Example

Walk through the example

  1. Name what each input represents.
  2. Follow one transformation at a time.
  3. Translate the result back into ordinary language.

The purpose is to reveal the mechanism, not merely display an answer.

A customer support conversation growing turn by turn will eventually approach a context limit. If a new user message would push the total over the limit, something has to give: dropping the oldest turns, summarizing them, or retrieving only the most relevant prior context rather than keeping the full history verbatim.

9. Python Example

What the code will demonstrate

The code builds a tiny version of the mechanism, prints values you can inspect, and connects them to the worked example. Predict the direction of the result before running it.

Python symbols used below

  • NumPy (np) stores numeric vectors and matrices.
  • np.array(...) creates a numeric collection.
  • Library calls perform the same conceptual steps shown above at a larger scale.
# Build a small, inspectable example of Context Window.
# Follow the inputs, transformations, and output in order.
CONTEXT_LIMIT = 50  # tiny illustrative limit (real models: e.g. 128,000+)

def rough_token_count(text):
    # illustrative approximation only -- Module 2: real counts need
    # the actual target model's tokenizer
    return max(1, len(text.split()))

conversation = [
    ("system", "You are a helpful assistant for a software company."),
    ("user", "How do I reset my password?"),
    ("assistant", "Go to Settings, click Security, then Reset Password."),
    ("user", "What if I don't have access to my email?"),
    ("assistant", "You can use SMS verification instead, under Security settings."),
    ("user", "My phone number changed recently though."),
]

running_total = 0
print("Token budget as conversation grows:")
for role, text in conversation:
    tokens = rough_token_count(text)
    running_total += tokens
    status = "OK" if running_total <= CONTEXT_LIMIT else "OVER LIMIT"
    print(f"  [{role:9s}] +{tokens:2d} tokens -> running total: {running_total:3d}/{CONTEXT_LIMIT}  [{status}]")

print(f"\nTotal tokens used: {running_total} / {CONTEXT_LIMIT}")

new_turn = ("user", "Also can you explain your entire password policy in detail, including all edge cases and exceptions for enterprise accounts across every region?")
new_tokens = rough_token_count(new_turn[1])
print(f"\nNew turn requires {new_tokens} tokens. Running total would become: {running_total + new_tokens}")

if running_total + new_tokens > CONTEXT_LIMIT:
    print("EXCEEDS CONTEXT LIMIT -- must truncate older turns before sending")
    kept = [conversation[0]]  # always keep system prompt
    remaining_budget = CONTEXT_LIMIT - new_tokens - rough_token_count(conversation[0][1])
    for role, text in reversed(conversation[1:]):
        t = rough_token_count(text)
        if t <= remaining_budget:
            kept.insert(1, (role, text))
            remaining_budget -= t
        else:
            break
    print("\nTruncated conversation kept (most recent turns prioritized):")
    for role, text in kept:
        print(f"  [{role}] {text}")
    print(f"  [user] {new_turn[1]}")

Expected Output:

Token budget as conversation grows:
  [system   ] + 9 tokens -> running total:   9/50  [OK]
  [user     ] + 6 tokens -> running total:  15/50  [OK]
  [assistant] + 8 tokens -> running total:  23/50  [OK]
  [user     ] + 9 tokens -> running total:  32/50  [OK]
  [assistant] + 9 tokens -> running total:  41/50  [OK]
  [user     ] + 6 tokens -> running total:  47/50  [OK]

Total tokens used: 47 / 50

New turn requires 22 tokens. Running total would become: 69
EXCEEDS CONTEXT LIMIT -- must truncate older turns before sending

Truncated conversation kept (most recent turns prioritized):
  [system] You are a helpful assistant for a software company.
  [assistant] You can use SMS verification instead, under Security settings.
  [user] My phone number changed recently though.
  [user] Also can you explain your entire password policy in detail, including all edge cases and exceptions for enterprise accounts across every region?

10. How It Works

  • The conversation genuinely fits comfortably within budget until turn 6 (47/50) — then the new, longer user message (22 tokens) would push the total to 69, clearly over the 50-token limit.
  • The truncation strategy kept the system prompt (always preserved) and the most recent turns, working backward from the newest until the remaining budget was exhausted — dropping the earliest exchange about password reset entirely. This is a genuine, common context management strategy: prioritize recency when budget is tight.
  • Notice a real, practical consequence: if the dropped early turn contained information still relevant to the new question, the model would have no access to it at all in this request — not “forgotten” in any cognitive sense, simply never included in what was sent.

11. What Happens When Context Becomes Too Large?

Option 1: TRUNCATE           drop oldest turns (as demonstrated) --
                              simple, but can lose relevant early
                              information

Option 2: SUMMARIZE            periodically compress older turns
                              into a shorter summary, preserving
                              key information while freeing budget

Option 3: RETRIEVE (RAG)         instead of keeping full history,
                              store it externally and retrieve
                              only what's relevant to the CURRENT
                              turn -- directly connects to your
                              RAG knowledge

Option 4: USE A LONG-CONTEXT       choose a model variant
MODEL                           specifically engineered for much
                                larger context windows, if the
                                use case genuinely requires it

12. How Is This Used in Modern AI?

Trace it through a real model call

user message → assembled context → LLM computation → decoded output → application checks

This topic affects one stage of that path; it is not the complete product. Hosted GPT- and Gemini-style applications also add instructions, safety systems, retrieval, tools, serving infrastructure, and evaluation around the model.

🤖 How Is This Used in Modern AI?

Every production LLM application with multi-turn conversations or large documents needs an explicit context management strategy — this isn’t optional infrastructure, it’s a direct, mandatory consequence of context windows being finite.

ApproachBest suited for
TruncationSimple chat apps where only recent context matters most
SummarizationLong-running conversations where early context still has some ongoing relevance
RAGLarge knowledge bases where only a small, relevant subset is needed per query
Long-context modelsGenuinely needing to reason over very large documents in full

13. How Is This Used in Agentic AI?

Separate the model from the runtime

goal + state + tool results → LLM proposal → runtime validation → execution or response

The LLM proposes text or a structured action. Ordinary application code controls permissions, tools, retries, memory, and execution.

Direct relevance to Agentic AI: Very High. Long-running agent conversations — with extensive system prompts, growing conversation history, and repeated tool call results — directly stress-test context window limits.

This is precisely why agent frameworks need deliberate context management (exactly this module’s truncation/summarization/ retrieval strategies) rather than naively appending every piece of history forever.

14. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: assuming context windows are unlimited or “large enough” by default.

Why it is incorrect: Even large context windows (hundreds of thousands of tokens) are finite, and — per the Transformers course’s quadratic cost analysis — using more of it costs meaningfully more in latency and compute, not just approaching a hard cutoff.

⚠️ Mistake

Incorrect idea: believing the model “remembers” dropped context in any sense.

Why it is incorrect: As demonstrated directly, once a turn is truncated out, the model has zero access to it for that request — nothing is retained beyond what’s explicitly included in the current input.

⚠️ Mistake

Incorrect idea: treating RAG as unrelated to context window management.

Why it is incorrect: As shown directly, RAG is precisely one of the standard strategies for handling context that wouldn’t otherwise fit — retrieve only what’s relevant, rather than trying to include everything.

15. Important Distinctions

Input TokensOutput Tokens
Tokens in the prompt sent TO the modelTokens the model GENERATES in response
Both count toward the same context window budget
TruncationRAG
Drops oldest content, chosen by recencyRetrieves specifically RELEVANT content, chosen by relevance
Simple, but can lose important early informationMore targeted, but requires a retrieval infrastructure

16. When to Use

Use simple truncation for short-lived, low-stakes conversations. Use summarization for longer conversations where general context (not specific details) matters. Use RAG (covered at basic level already, deepened in the upcoming Agentic AI course) whenever the total relevant knowledge base far exceeds what could ever fit in a single context window.

17. When Not to Use

Don’t rely purely on ever-growing conversation history for long-running applications — this guarantees eventually hitting the context limit, exactly as demonstrated directly, with an increasingly poor (and increasingly expensive, given quadratic cost) user experience as the conversation grows.

18. Production Considerations

  • Reserve budget for generation — the context limit typically covers input AND output together (provider-dependent); leaving no room for a substantial response is a common, avoidable mistake.
  • Longer context costs more — both in raw token pricing and in latency, given the quadratic attention cost (Transformers course) — a genuine reason to manage context deliberately rather than simply maximizing what’s included.
  • Context management strategy should match the application — a simple support chatbot’s needs differ meaningfully from a document-analysis agent’s.

19. What You Should Remember

  • A context window is the maximum number of tokens (input + output) a model can process in one request — a hard, finite limit, not a soft guideline.
  • Context limits exist because of real compute and memory costs (quadratic attention scaling, KV cache growth — Transformers course), not arbitrary restriction.
  • Truncation, summarization, and RAG are the standard strategies for managing context that wouldn’t otherwise fit — verified directly with a real truncation example, prioritizing recency.

20. Interview Questions

Beginner

Q: What is a context window?

Ans: The maximum number of tokens — combining input (prompt, conversation history) and output (the model’s generated response) — that a model can process in a single request. Anything beyond this limit either causes the request to fail or must be managed through truncation, summarization, or retrieval before sending.

Intermediate

Q: Why do context windows have hard limits rather than being unlimited?

Ans: Attention computation (Transformers course) scales quadratically with sequence length — doubling context roughly quadruples the core computation cost — and the KV cache used during inference grows with context length too, consuming real GPU memory.

Context limits are set at a level the serving infrastructure can actually support, given these real, unavoidable compute and memory costs.

Advanced

Q: What happens, mechanically, to information from early in a conversation once it’s truncated out of context due to a growing token budget?

Ans: It’s not “forgotten” in any cognitive sense — the model has zero access to any information not explicitly included in the current request’s input tokens.

Truncation is a decision made by the surrounding application (or the developer’s context management logic) about what to include in each request; verified directly in this module, a truncation strategy dropped the earliest conversation turn entirely once the token budget was exceeded, and that content simply wasn’t part of what the model processed for the next request at all.

Scenario

**Q: A team’s chatbot application seems to “lose track” of information mentioned early in very long conversations.

Diagnose this using what you’ve learned.** A: This is very likely a context management issue, not a model reasoning failure — as demonstrated directly, once a conversation’s total token count approaches or exceeds the context limit, some content must be dropped (via truncation) or otherwise managed (summarization, retrieval).

If the application is naively truncating oldest-first without preserving genuinely important early details, exactly the “losing track” symptom described would occur — the fix would be a more deliberate context management strategy: summarizing key facts from earlier in the conversation, or using retrieval to bring back specifically relevant earlier context when needed, rather than relying purely on recency-based truncation.

AI Engineering

Q: How does the choice between truncation, summarization, and RAG for context management depend on the specific application?

Ans: It depends on what kind of context actually matters for the application’s task. Truncation (keeping only recent turns) suits applications where only the most recent exchanges are genuinely relevant, like a simple support chat.

Summarization suits longer conversations where the general arc and key facts matter but exact wording of early turns doesn’t.

RAG suits applications with a large knowledge base where only a small, specifically relevant subset is needed per query — rather than trying to fit an entire knowledge base (or entire conversation history) into context at once, retrieval selects just what’s relevant to the current request, directly avoiding the token budget problem demonstrated in this module.

21. Next Step

Next: Module 4 — Embeddings and Representations — the LLM-specific view of how token IDs become vectors, and how those vectors change as they flow through Transformer layers.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed