TechByteByByte

Why RAG Exists

The realistic problem RAG was built to solve: what happens when an LLM needs to answer questions about information it was never trained on — private, current, or enterprise-specific knowledge.

#RAG#AI#Foundations#Level 1

Begin with the problem

A language model can write fluent answers while knowing nothing about your private documents or today’s changes. RAG exists to find useful evidence before asking the model to answer.

question → retrieve evidence → build context → model → answer

What you will learn

  • Explain Why RAG Exists in simple language before using its technical details.
  • Follow the mechanism step by step through a small RAG example.
  • Connect this topic to the modules before and after it.
  • Decide when to use it, when not to use it, and what to measure in production.

Current real-system grounding: Google’s Gemini File Search guide documents a managed RAG flow that imports, chunks, embeds, indexes, retrieves, and grounds model responses.

The product example proves that the pattern is used in a real system. It does not mean every provider uses the same hidden algorithm, defaults, limits, or pricing.

1. The problem this module solves

You’ve completed courses on LLMs, Prompt Engineering, and Generative AI — you understand how these models generate text. This course answers a different question: what do you do when the model needs to answer using information it was never trained on? This module starts with the real, concrete problem, before ever mentioning the term “RAG.”

2. The Problem

Imagine a company with:

- 50,000 internal documents
- HR policies
- Product documentation
- Technical manuals
- Customer information
- Engineering documentation

An employee asks a chatbot built on a powerful LLM:

“What is our company’s reimbursement policy for international travel?”

The LLM is really capable — it can write code, summarize articles, reason through problems. But it has never seen this specific company’s internal HR policy document. It simply cannot know:

- This company's specific policy
- The latest version of that policy
- Internal exceptions (e.g., a special London hotel limit)
- Employee-specific rules

3. Why the LLM Alone Really Struggles

It’s worth being precise about why this happens, not just that it happens:

The knowledge stored in a base LLM's learned parameters comes from its
training process and is not updated simply because a new event happens.
An AI application can still supply newer information through tools, web
search, memory, or retrieved context; RAG is one way to do that.

Without an external source, the model cannot reliably know:
   - Information that didn't exist publicly when it was trained
   - PRIVATE, internal, or proprietary information that was never
     part of any public training corpus
   - Information that has CHANGED since training (a policy updated
     last month)

If you ask the LLM directly, it has exactly two honest options: confidently guess (risking hallucination, your Generative AI course), or admit it doesn’t know. Neither actually answers the employee’s question.

4. The Naive “Just Paste It In” Approach — And Why It Breaks

A first instinct might be: “just paste the whole HR policy document into the prompt.” This really works for a single small document. But scale it up:

50,000 documents
   ↓
Far larger than any model's CONTEXT WINDOW (your LLM course)
   ↓
Even if it fit: massive TOKEN COST for every single question
   ↓
Even if affordable: the model has to search through THOUSANDS of
irrelevant pages to find the ONE relevant paragraph -- really
noisy, unreliable

This naive approach doesn’t scale — which is precisely the gap RAG was built to close.

5. The Intuition — Retrieve, Then Generate

Here’s the core idea, in plain language, before any formal terminology:

Instead of asking the LLM to know everything, or forcing the entire knowledge base into every prompt, find just the relevant pieces of information first — then hand only those pieces to the LLM as context for answering this specific question.

Employee's question
   ↓
Search through the 50,000 documents for the FEW paragraphs that are
actually relevant to THIS question
   ↓
Hand ONLY those relevant paragraphs to the LLM, along with the
question
   ↓
LLM generates an answer USING that provided information

This is really the entire idea. Everything else in this course is about doing each of these steps well, reliably, and at scale.

6. Naming It — Retrieval-Augmented Generation

Now that the intuition is established, the formal term:

Retrieval-Augmented Generation (RAG): an application pattern where relevant information is retrieved from an external knowledge source and provided to a language model as context, augmenting its generation with information it wouldn’t otherwise have.

RETRIEVAL:      finding the relevant information

AUGMENTED:         the model's input is ENRICHED with that
                 information

GENERATION:            the model produces an answer USING that
                     enriched input

Notice: nothing about the model itself changed. The model’s weights, training, and capabilities are exactly the same as before — what changed is what information it has access to for this specific question.

7. A Real Developer Example

Building a customer support chatbot for TechCorp:

WITHOUT RAG:      the chatbot is a general-purpose LLM. Ask it "what's
                 TechCorp's return policy?" and it either says "I
                 don't know" or, worse, GUESSES based on generic
                 e-commerce norms -- really wrong for this specific
                 company.

WITH RAG:            the chatbot first searches TechCorp's actual
                   policy documents for the relevant section, then
                   gives that ACTUAL policy text to the LLM as
                   context. The LLM now answers using TechCorp's
                   real, current policy -- not a generic guess.

8. A Simple Agentic AI Connection

RAG is one of the most common tools given to an AI agent (your Generative AI course covered agents in depth). An agent equipped with a “search knowledge base” tool performs exactly this retrieval process on demand — deciding when retrieval is really needed for a given user request, rather than retrieving unconditionally on every turn. This course builds toward exactly that kind of system.

9. How Is This Used in AI?

🤖 How Is This Used in AI?

RAG is one of the most widely deployed patterns in production AI systems — customer support tools grounded in company knowledge bases, internal documentation assistants, legal and research tools grounded in specific document collections, and any application where current, specific, verifiable information really matters more than what a model’s frozen training data alone can provide.

10. Real-World Applications

  • Internal company knowledge assistants (HR, IT, legal)
  • Customer support grounded in current product documentation
  • Research and legal document analysis
  • Technical documentation assistants for developers

11. When to Use RAG

Use RAG when the application really needs:

- CURRENT information that changes over time
- PRIVATE or proprietary information never in public training data
- LARGE knowledge bases too big to fit in a single prompt
- VERIFIABLE, source-grounded answers

12. When NOT to Use RAG (Preview)

Not every problem needs RAG — this is covered in full in a later module, but worth flagging early: if the knowledge base is small enough to fit entirely in a prompt, or the task is really about consistent style rather than facts, other tools (careful prompting, fine-tuning) may be a better fit. More on this soon.

13. Common Mistakes

Incorrect idea: Assuming a bigger, smarter LLM alone solves this problem.

Why it is incorrect: As shown directly in Section 3, no amount of raw capability lets a model know information it was never trained on.

Incorrect idea: Trying to paste an entire knowledge base into every prompt.

Why it is incorrect: As shown directly in Section 4, this really breaks down at real scale — both in context window size and in cost.

Incorrect idea: Believing RAG changes the model itself.

Why it is incorrect: As emphasized directly in Section 6, RAG changes what information the model has access to — the model’s weights and capabilities are completely unchanged.

14. Limitations

  • RAG is only as good as the retrieval step — if the right information isn’t found, the LLM can’t compensate for that gap (a principle this course returns to repeatedly)
  • RAG adds real architectural complexity compared to a simple, direct LLM call — this complexity is only worth it when the problem really requires it

15. Quick Reference — The Whole Idea in One Diagram

WITHOUT RAG:

User Question
   ↓
LLM (relies only on frozen training knowledge)
   ↓
Answer (may be wrong, outdated, or "I don't know")


WITH RAG:

User Question
   ↓
Retrieve Relevant Information (from an external knowledge source)
   ↓
Relevant Context
   ↓
LLM (now has BOTH the question AND relevant, current information)
   ↓
Answer (grounded in actual, retrieved information)

16. Code — Demonstrating the Core Problem and Solution

🎯 Target of this example: make Section 2-6’s entire argument directly observable — showing an LLM fail to answer a company-specific question without help, then succeed once given the relevant retrieved information, using real API calls.

Example 1 — Simple

import anthropic

client = anthropic.Anthropic()

# WITHOUT any retrieved information -- the model has no way to know
# this specific company's actual policy.
response = client.messages.create(
    model="claude-sonnet-4-6", max_tokens=150,
    messages=[{"role": "user", "content":
               "What is TechCorp's reimbursement policy for "
               "international travel hotels?"}]
)
print("Without retrieval:", response.content[0].text)

Expected Output:

Without retrieval: I don't have specific information about
TechCorp's reimbursement policy for international travel hotels, as
I don't have access to your company's internal policy documents. To
get accurate details, I'd recommend checking your employee handbook,
HR portal, or reaching out to your HR department directly.

What we conclude from this example: exactly as Section 2-3 predicted, the model correctly admits it doesn’t know this specific company’s policy — a really honest response, but not a useful answer for the employee who asked.

Example 2 — Intermediate

import anthropic

client = anthropic.Anthropic()

# The "retrieved" information -- in a real system, this comes from
# searching a knowledge base (Modules 7-14 build this search
# mechanism). For now, we simulate having ALREADY found the relevant
# text.
retrieved_context = (
    "TechCorp Travel Policy, Section 4.2: Employees are eligible for "
    "accommodation reimbursement up to $200 per night for international "
    "travel. A special exception applies to London: up to $250 per "
    "night due to higher local hotel costs. Reimbursement requires "
    "submitting an itemized hotel receipt within 30 days of travel."
)

def answer_with_retrieved_context(question: str, context: str) -> str:
    """This IS the core RAG pattern from Section 6 -- augmenting the
    model's input with retrieved information before it generates."""
    response = client.messages.create(
        model="claude-sonnet-4-6", max_tokens=150,
        messages=[{"role": "user", "content":
                   f"Using ONLY the following context, answer the question.\n\n"
                   f"Context: {context}\n\n"
                   f"Question: {question}"}]
    )
    return response.content[0].text

result = answer_with_retrieved_context(
    "What is TechCorp's reimbursement policy for international travel hotels?",
    retrieved_context,
)
print("With retrieval:", result)

Expected Output:

With retrieval: TechCorp's policy allows employees to be reimbursed
up to $200 per night for hotel accommodation during international
travel. There's a special exception for London, where the limit is
raised to $250 per night due to higher local hotel costs. To claim
reimbursement, you'll need to submit an itemized hotel receipt within
30 days of your travel.

What we conclude from this example: the SAME model, given the SAME question, now produces a really accurate, specific, useful answer — purely because it was handed the relevant information as context. Nothing about the model changed; only what it had access to changed. This is the entire RAG idea, made concrete.

Example 3 — Production Grade

import anthropic
from dataclasses import dataclass

client = anthropic.Anthropic()

@dataclass
class RAGResult:
    question: str
    context_provided: bool
    answer: str

def compare_with_and_without_retrieval(question: str, context: str | None) -> RAGResult:
    """A small, production-style function making the RETRIEVAL
    decision EXPLICIT -- context_provided is tracked so downstream
    code (logging, evaluation, Module 32) can distinguish grounded
    answers from ungrounded ones."""
    if context:
        prompt = (f"Using ONLY the following context, answer the "
                   f"question. If the context doesn't answer it, say so.\n\n"
                   f"Context: {context}\n\nQuestion: {question}")
    else:
        prompt = question

    response = client.messages.create(
        model="claude-sonnet-4-6", max_tokens=150,
        messages=[{"role": "user", "content": prompt}]
    )
    return RAGResult(question=question, context_provided=context is not None,
                      answer=response.content[0].text)

retrieved_context = (
    "TechCorp Travel Policy, Section 4.2: Employees are eligible for "
    "accommodation reimbursement up to $200 per night for international "
    "travel, with a $250/night exception for London."
)

no_rag = compare_with_and_without_retrieval(
    "What is TechCorp's hotel reimbursement limit for London?", None
)
with_rag = compare_with_and_without_retrieval(
    "What is TechCorp's hotel reimbursement limit for London?", retrieved_context
)

print(f"[context_provided={no_rag.context_provided}] {no_rag.answer}\n")
print(f"[context_provided={with_rag.context_provided}] {with_rag.answer}")

Expected Output:

[context_provided=False] I don't have access to TechCorp's specific
travel policy details, including hotel reimbursement limits for
London. Please check your employee handbook or contact HR for this
information.

[context_provided=True] TechCorp's hotel reimbursement limit for
London is $250 per night, which is a special exception to the
standard $200/night international travel limit due to higher local
hotel costs.

What we conclude from this example: explicitly tracking context_provided on every result is a useful diagnostic pattern. It shows whether retrieved evidence was included in the request, but it does not prove that the answer used that evidence correctly. Actual groundedness needs claim-to-source verification, citations, and evaluation, which Modules 23 and 32 cover; Module 33 adds production observability.

17. Interview Questions

Q: Why can’t a large, capable LLM simply answer questions about a specific company’s internal policies without any additional help?

Ans: A base model’s learned parameters do not automatically contain a company’s private policy or update themselves when that policy changes. The surrounding application may provide outside information through tools, web search, memory, or RAG, but without such a source the model cannot reliably answer from the private policy. Raw model capability does not create access to information the application never supplied.

Q: Why doesn’t simply pasting an entire knowledge base into every prompt solve this problem at real scale?

Ans: At real scale (tens of thousands of documents), the knowledge base far exceeds any model’s context window, so it literally can’t fit. Even in scenarios where it might technically fit, the token cost of resending the entire knowledge base on every single question becomes prohibitively expensive, and the model has to search through mostly irrelevant content to find the few relevant paragraphs, which really degrades answer quality and reliability.

Q: Explain the core RAG idea in your own words, without using the acronym.

Ans: Instead of expecting the model to already know everything, or forcing an entire knowledge base into every prompt, first search for just the pieces of information that are actually relevant to the specific question being asked. Then hand only those relevant pieces to the model as context, so it can generate an answer using really relevant, current, specific information rather than relying solely on its frozen training knowledge.

Q: Does RAG change the underlying language model in any way?

Ans: No — RAG changes what information the model has access to for a given request, not the model itself. The model’s weights, training, and general capabilities remain completely unchanged. What changes is the input: instead of just the user’s question, the model now also receives relevant, retrieved context alongside it.

18. What You Should Remember

  • A base model’s learned parameters do not update automatically when private policies or current events change. Applications can provide newer information through retrieval and other tools.
  • Pasting an entire knowledge base into every prompt doesn’t scale — RAG’s core idea is to retrieve only what’s relevant, then hand that to the model as context.
  • RAG doesn’t change the model — it changes what information the model has access to for a given question, verified directly by giving the same model the same question with and without retrieved context.

19. Quick Practice

Think of a question you might ask about a private, personal knowledge base (like your own notes or a specific book you’ve read). Explain, in your own words, why a general-purpose LLM couldn’t answer it directly, and what “relevant retrieved context” would need to look like for it to answer correctly.

20. Next Step

Next: Module 2 — The Fundamental RAG Idea — formalizing the “without RAG vs. with RAG” mental model introduced here, and establishing it as the foundation for the rest of this course.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed