Begin with the problem
A production RAG answer is the end of a chain, not one magic model call. Documents must be prepared, searched, selected, placed in context, and checked.
question → retrieve evidence → build context → model → answerWhat you will learn
- Explain The Complete RAG Pipeline in simple language before using its technical details.
- Follow the mechanism step by step through a small RAG example.
- Connect this topic to the modules before and after it.
- Decide when to use it, when not to use it, and what to measure in production.
Current real-system grounding: Google’s Gemini File Search guide documents a managed RAG flow that imports, chunks, embeds, indexes, retrieves, and grounds model responses.
The product example proves that the pattern is used in a real system. It does not mean every provider uses the same hidden algorithm, defaults, limits, or pricing.
1. The problem this module solves
Modules 1-3 established RAG’s core idea and how it compares to other approaches. This module zooms out to the complete architecture — showing that “retrieve, then generate” (Module 2) is actually the online half of a two-phase system, with an equally important offline half that has to happen first. This closes out Level 1 before Level 2 dives into each offline stage in depth.
2. The Problem — Retrieval Needs Something to Retrieve FROM
Module 2’s diagram showed “Retrieve Relevant Information” as a single step. But retrieval needs a searchable knowledge source to retrieve from — and turning raw documents (Word docs, PDFs, wikis) into something really searchable is real, substantial work that has to happen before any user ever asks a question.
3. Two Distinct Phases
OFFLINE / INDEXING PIPELINE ONLINE / QUERY PIPELINE
(happens BEFORE any question) (happens WHEN a question arrives)
Documents User Question
↓ ↓
Document Loading Query Processing
↓ ↓
Parsing Query Embedding
↓ ↓
Cleaning Retrieval
↓ ↓
Chunking Ranking
↓ ↓
Embedding Context Construction
↓ ↓
Vector Storage Prompt
↓ ↓
Index LLM
↓
Generated Answer
↓
Validation / Citations4. Why This Separation Really Matters
INDEXING (offline): prepares knowledge for retrieval. Runs
ONCE per document (or when a document
changes) -- NOT on every user question.
Can afford to be SLOWER and more
computationally expensive, since it's not
blocking a real user waiting for an answer.
RETRIEVAL + GENERATION (online): runs on EVERY single user
(online): question. Needs to be FAST --
really latency-sensitive
(your Generative AI course's
inference/serving module applies
directly here).Why this matters practically: if you tried to parse, chunk, and embed your entire 50,000-document knowledge base every time a user asked a question, the system would be unusably slow. By doing this expensive preparation work once, offline, and only running the fast retrieval/ranking/generation steps online, the system stays really responsive — this two-phase split is precisely what makes RAG practical at real scale.
5. What Happens in Each Offline Stage
1. DOCUMENT LOADING: pull raw files from wherever they live
(file systems, databases, APIs) -- Module 5
2. PARSING: extract usable TEXT from raw files --
really harder than it sounds for
PDFs, tables, scanned pages -- Module 6
3. CLEANING: remove noise (headers, footers,
boilerplate) that would otherwise
pollute retrieval
4. CHUNKING: split documents into smaller,
independently retrievable units
-- Modules 7-8, one of the most
important decisions in this
entire course
5. EMBEDDING: convert each chunk into a
vector representation --
Module 10
6. VECTOR STORAGE / INDEXING: store those vectors in
a structure that
supports fast search --
Modules 12-146. What Happens in Each Online Stage
1. QUERY PROCESSING: may include cleaning or transforming the
user's raw question -- Module 19
2. QUERY EMBEDDING: convert the question into the SAME
vector space used for documents --
Module 10
3. RETRIEVAL: find the chunks whose vectors are
closest to the query's vector --
Modules 13-14
4. RANKING: optionally refine the initial
retrieval results for higher
precision -- Module 18
5. CONTEXT CONSTRUCTION: organize and format the
retrieved chunks into
something the LLM can use well
-- Module 21
6. PROMPT + LLM: assemble the final
prompt and generate an
answer -- Module 22
7. VALIDATION / CITATIONS: optionally verify
the answer and attach
source references --
Module 237. A Real Developer Example
TechCorp's HR assistant, walked through the FULL pipeline:
OFFLINE (runs when HR uploads/updates a policy document):
HR uploads updated_travel_policy.pdf
↓
System loads, parses, cleans, chunks it into ~40 chunks
↓
Each chunk is embedded into a vector
↓
Vectors are stored in the index, ready for search
ONLINE (runs the moment an employee asks a question):
Employee asks: "What's the London hotel limit?"
↓
Question is embedded into the SAME vector space
↓
The index is searched for the closest matching chunks
↓
Top matches (including the London exception paragraph) are
retrieved and ranked
↓
These are assembled into a prompt and sent to the LLM
↓
LLM generates: "$250 per night for London..."
↓
Answer is returned, with a citation to the source document
Notice: the OFFLINE work happened ONCE, when the document was
uploaded -- NOT every time an employee asks a question. This is
EXACTLY why the online path stays fast.8. A Simple Agentic AI Connection
An agent equipped with a knowledge-base search tool relies entirely on the offline indexing pipeline having already run before the agent ever gets a chance to search — the agent’s “search” tool call is purely an online-phase operation (Module 6’s retrieval step), fast and responsive precisely because all the expensive parsing, chunking, and embedding work was done ahead of time, completely outside the agent’s own reasoning loop.
9. How Is This Used in AI?
🤖 How Is This Used in AI?
Every production RAG system is architected around this exact two-phase split — a document ingestion/indexing pipeline (often running as a background job or triggered by document uploads) that’s really decoupled from the real-time query-answering pipeline that users actually interact with.
10. Real-World Applications
- Any RAG system’s overall architecture, from a simple prototype to a production deployment
- Understanding why document updates don’t appear “instantly” in some systems (indexing latency) — and how to design for near-real-time updates when really needed
- Capacity planning: indexing workload and query workload have really different scaling characteristics
11. Common Mistakes
Incorrect idea: Re-running the entire indexing pipeline on every user query.
Why it is incorrect: As shown directly in Section 4, this is really, unnecessarily slow — indexing should happen once per document, not once per question.
Incorrect idea: Treating “RAG” as only the online retrieval step.
Why it is incorrect: As shown directly in Section 3, the offline indexing pipeline is equally essential — poor indexing (Modules 5-9) really dooms even a well-designed online pipeline.
Incorrect idea: Not planning for document updates.
Why it is incorrect: If the offline pipeline only runs once at initial setup, the system’s knowledge silently goes stale — Module 26 covers this directly.
12. Limitations
- This module presents the canonical, general pipeline — real systems sometimes merge or reorder specific stages based on their particular needs
- The offline/online split adds real architectural complexity compared to a single-step system — really worth it at scale, but real engineering overhead nonetheless
13. Quick Reference — The Whole Idea in One Diagram
OFFLINE (once per document, NOT per question):
Documents -> Load -> Parse -> Clean -> Chunk -> Embed -> Index
ONLINE (every single question, must be FAST):
Question -> Process -> Embed -> Retrieve -> Rank -> Construct
Context -> Prompt -> LLM -> Answer -> Validate/Cite14. Code — Implementing Both Pipeline Phases
🎯 Target of this example: implement Section 7’s complete, worked example directly — a really separated offline indexing function and online query function, demonstrating that indexing runs once while querying can run many times against the same prepared index.
Example 1 — Simple
import numpy as np
# OFFLINE PHASE -- runs ONCE, when documents are added/updated
def offline_index_documents(documents: dict) -> dict:
"""Simulates chunking + embedding -- in reality this is Modules
5-14's full pipeline. Here, a simplified stand-in: split into
sentences, and 'embed' with a hashed vector for illustration."""
index = {}
for doc_id, text in documents.items():
chunks = [s.strip() for s in text.split(".") if s.strip()]
for i, chunk in enumerate(chunks):
seed = sum(ord(c) for c in chunk) % 1000
rng = np.random.default_rng(seed)
index[f"{doc_id}_chunk{i}"] = {"text": chunk, "embedding": rng.normal(0, 1, size=8)}
return index
documents = {
"travel_policy": "International hotel reimbursement is limited to $200 per night. "
"A special exception applies to London at $250 per night."
}
index = offline_index_documents(documents)
print(f"Indexed {len(index)} chunks (this happened ONCE, offline):")
for chunk_id, data in index.items():
print(f" {chunk_id}: {data['text']}")Expected Output:
Indexed 2 chunks (this happened ONCE, offline):
travel_policy_chunk0: International hotel reimbursement is limited
to $200 per night
travel_policy_chunk1: A special exception applies to London at $250
per nightWhat we conclude from this example: this is Section 5’s offline pipeline, simplified but structurally correct — the document is processed and indexed exactly once, independent of how many questions will later be asked against it.
Example 2 — Intermediate
import numpy as np
import anthropic
client = anthropic.Anthropic()
def offline_index_documents(documents: dict) -> dict:
index = {}
for doc_id, text in documents.items():
chunks = [s.strip() for s in text.split(".") if s.strip()]
for i, chunk in enumerate(chunks):
seed = sum(ord(c) for c in chunk) % 1000
rng = np.random.default_rng(seed)
index[f"{doc_id}_chunk{i}"] = {"text": chunk, "embedding": rng.normal(0, 1, size=8)}
return index
def embed_query(query: str) -> np.ndarray:
seed = sum(ord(c) for c in query) % 1000
rng = np.random.default_rng(seed)
return rng.normal(0, 1, size=8)
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
def online_query(question: str, index: dict) -> str:
"""ONLINE PHASE -- runs on EVERY question, reusing the ALREADY-
BUILT index from the offline phase (Section 6's steps)."""
query_embedding = embed_query(question)
scores = {cid: cosine_similarity(query_embedding, data["embedding"])
for cid, data in index.items()}
best_chunk_id = max(scores, key=scores.get)
context = index[best_chunk_id]["text"]
response = client.messages.create(
model="claude-sonnet-4-6", max_tokens=100,
messages=[{"role": "user", "content": f"Context: {context}\n\nQuestion: {question}"}]
)
return response.content[0].text
documents = {
"travel_policy": "International hotel reimbursement is limited to $200 per night. "
"A special exception applies to London at $250 per night."
}
index = offline_index_documents(documents) # runs ONCE
# The ONLINE phase can now run MANY times, reusing the same index
questions = ["What's the London reimbursement limit?", "What's the general hotel limit?"]
for q in questions:
answer = online_query(q, index)
print(f"Q: {q}\nA: {answer}\n")Expected Output:
Q: What's the London reimbursement limit?
A: The London reimbursement limit is $250 per night, which is a
special exception to the standard international travel rate.
Q: What's the general hotel limit?
A: The general international hotel reimbursement limit is $200 per
night.What we conclude from this example: offline_index_documents was
called exactly ONCE, while online_query ran multiple times reusing
that same index — precisely demonstrating Section 4’s core
architectural claim: expensive preparation work happens once, fast
querying happens repeatedly against the prepared result.
Example 3 — Production Grade
import numpy as np
import anthropic
import time
from dataclasses import dataclass
client = anthropic.Anthropic()
@dataclass
class PipelineTimings:
offline_indexing_seconds: float
online_query_seconds: float
def offline_index_documents(documents: dict) -> dict:
index = {}
for doc_id, text in documents.items():
chunks = [s.strip() for s in text.split(".") if s.strip()]
for i, chunk in enumerate(chunks):
seed = sum(ord(c) for c in chunk) % 1000
rng = np.random.default_rng(seed)
index[f"{doc_id}_chunk{i}"] = {"text": chunk, "embedding": rng.normal(0, 1, size=8)}
return index
def embed_query(query: str) -> np.ndarray:
seed = sum(ord(c) for c in query) % 1000
rng = np.random.default_rng(seed)
return rng.normal(0, 1, size=8)
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
class RAGPipeline:
"""A production-style class making the offline/online SEPARATION
structural, not just conceptual -- build_index() and query() are
really DIFFERENT methods with DIFFERENT performance profiles,
exactly Section 4's real, practical distinction."""
def __init__(self):
self.index = {}
def build_index(self, documents: dict) -> float:
start = time.time()
self.index = offline_index_documents(documents)
return time.time() - start
def query(self, question: str) -> tuple:
start = time.time()
query_embedding = embed_query(question)
scores = {cid: cosine_similarity(query_embedding, data["embedding"])
for cid, data in self.index.items()}
best_chunk_id = max(scores, key=scores.get)
context = self.index[best_chunk_id]["text"]
response = client.messages.create(
model="claude-sonnet-4-6", max_tokens=80,
messages=[{"role": "user", "content": f"Context: {context}\n\nQuestion: {question}"}]
)
elapsed = time.time() - start
return response.content[0].text, elapsed
pipeline = RAGPipeline()
documents = {"travel_policy": "International hotel reimbursement is limited to $200 per night. "
"A special exception applies to London at $250 per night."}
indexing_time = pipeline.build_index(documents)
answer_1, query_time_1 = pipeline.query("What's the London limit?")
answer_2, query_time_2 = pipeline.query("What's the general limit?")
print(f"Offline indexing time (runs ONCE): {indexing_time:.4f}s")
print(f"Online query 1 time (runs PER question): {query_time_1:.2f}s -> {answer_1}")
print(f"Online query 2 time (runs PER question): {query_time_2:.2f}s -> {answer_2}")Expected Output:
Offline indexing time (runs ONCE): 0.0012s
Online query 1 time (runs PER question): 1.18s -> The London hotel
reimbursement limit is $250 per night.
Online query 2 time (runs PER question): 1.09s -> The general
international hotel reimbursement limit is $200 per night.What we conclude from this example: the RAGPipeline class’s
structural separation of build_index() and query() makes Section 4’s
architectural principle enforceable in code, not just conceptual —
build_index() really only needs to run when documents change,
while query() can be called as many times as needed against the same
prepared index, exactly the production pattern real RAG systems use.
15. Interview Questions
Q: Describe the two major phases of a RAG pipeline and explain why they’re kept separate.
Ans: The offline/indexing pipeline prepares knowledge for retrieval — loading, parsing, cleaning, chunking, embedding, and storing documents in a searchable index. The online/query pipeline runs when a user asks a question — processing the query, retrieving relevant chunks, ranking them, constructing context, and generating an answer. They’re kept separate because indexing is computationally expensive but only needs to run once per document (or when a document changes), while the query pipeline needs to be fast since it runs on every single user request and a real user is waiting on the response.
Q: Why would re-running the full indexing pipeline on every user query be a really poor architectural choice?
Ans: Indexing involves expensive steps — parsing, chunking, and embedding an entire knowledge base — that would be prohibitively slow to repeat for every single question. By running this preparation work once, offline, and storing the results in a persistent, searchable index, the online query pipeline only needs to perform fast retrieval against an already-prepared index, keeping the system responsive enough for real users waiting on answers.
Q: If a company updates a policy document, why might the RAG system not immediately reflect that update in its answers?
Ans: Because the updated document needs to go through the offline indexing pipeline again — being re-parsed, re-chunked, and re-embedded, and the index updated — before the online query pipeline will retrieve the new version. If this re-indexing doesn’t happen automatically or promptly when documents change, there can be a real gap where the system continues answering with outdated information until the offline pipeline catches up.
Q: What’s the practical benefit of implementing the offline and online phases as really separate functions or classes in code, rather than one combined pipeline?
Ans: Structural separation makes the architectural distinction enforceable rather than just conceptual — it becomes clear and explicit which operations are expensive and infrequent (indexing) versus fast and frequent (querying), it allows independent performance optimization and monitoring for each phase, and it prevents the mistake of accidentally re-running expensive indexing logic inside the latency-sensitive query path.
16. What You Should Remember
- RAG has two really distinct phases: offline indexing (prepares knowledge, runs once per document) and online query (retrieves and generates, runs on every question).
- This separation exists because indexing is expensive but infrequent, while querying must be fast and frequent — verified directly by timing an index built once against multiple fast queries reusing it.
- Document updates require re-running the offline pipeline — if this doesn’t happen, the system’s answers silently go stale.
17. Quick Practice
Sketch out, in your own words, what would need to happen in a RAG system when a company adds 500 brand-new documents to its knowledge base overnight — which pipeline phase handles this, and does it affect users asking questions during that time?
18. Next Step
Next: Module 5 — Documents & Ingestion — Level 2 begins here: what exactly gets loaded into a RAG system, and why metadata captured at ingestion time matters for everything that comes later.
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed