Begin with the problem
When RAG fails, changing the model is often the wrong first move. The fault may be ingestion, parsing, chunking, retrieval, ranking, context construction, or generation.
observe failure → locate pipeline stage → change one component → evaluateWhat you will learn
- Explain RAG Failure Modes in simple language before using its technical details.
- Follow the mechanism step by step through a small RAG example.
- Connect this topic to the modules before and after it.
- Decide when to use it, when not to use it, and what to measure in production.
Current real-system grounding: OpenAI’s evaluation guidance supports testing changes against datasets rather than trusting a few demos. RAG needs separate retrieval and answer checks.
The product example proves that the pattern is used in a real system. It does not mean every provider uses the same hidden algorithm, defaults, limits, or pricing.
1. The problem this module solves
Every module so far has covered how to do one pipeline stage well. This module deliberately steps back and asks the opposite question: where can this ENTIRE pipeline actually break? Understanding the complete failure landscape is really essential for diagnosing real problems, not just building things that work when everything goes right.
2. The Complete Failure Landscape
INGESTION FAILURE (Module 5): metadata not captured, wrong
source information, permissions
missing
PARSING FAILURE (Module 6): tables flattened incorrectly,
columns interleaved, scanned
documents with no OCR
CHUNKING FAILURE (Modules 7-9): important context split
across chunk boundaries,
headings separated from
content
EMBEDDING FAILURE (Module 10): poor semantic
representation, query and
document embedded with
MISMATCHED models
RETRIEVAL FAILURE (Modules 11-18): relevant chunk
really never
retrieved at all --
search missed it
entirely
RANKING FAILURE (Module 18): relevant chunk
retrieved, but
ranked too LOW to
make the final
cut
CONTEXT FAILURE (Module 21): too much
irrelevant
content, poor
ordering,
real
duplication
GENERATION FAILURE (Module 22-23): model
produces
content
UNSUPPORTED
by the
provided
context
CITATION FAILURE (Module 23): answer
cites the
WRONG or
a
FABRICATED
source3. The Single Most Important Principle in This Module
A GOOD LLM CANNOT COMPENSATE FOR BAD RETRIEVAL.If the really correct information never reaches the model’s context window, the model cannot reliably answer correctly — no matter how capable that model is. This directly connects back to Module 2’s retrieval/generation split: debug retrieval FIRST, before assuming the model itself is the problem.
This single principle should guide how you approach ANY RAG failure — it’s really worth internalizing as the default starting assumption when something goes wrong.
4. Diagnosing a Failure — Working Backward Through the Pipeline
Bad answer observed
↓
Was the RIGHT chunk in the retrieved candidate set AT ALL? (check
BEFORE ranking, Module 18)
NO -> RETRIEVAL failure (Modules 11-18) -- check embedding
quality, search parameters, chunking (did the RIGHT
information even get chunked properly in the first place?)
YES -> continue
Was the RIGHT chunk ranked HIGH enough to survive top-k (Module 15)
and reranking (Module 18)?
NO -> RANKING failure -- check reranking logic, k value
YES -> continue
Was the RIGHT chunk actually INCLUDED in the final constructed
context (Module 21)?
NO -> CONTEXT CONSTRUCTION failure -- check filtering thresholds
YES -> continue
Did the MODEL actually use the provided context correctly (Module
23's groundedness check)?
NO -> GENERATION failure -- check prompt construction (Module
22), grounding instructionsThis is really the practical, step-by-step diagnostic process a real RAG engineer follows.
5. A Real Developer Example — A Complete Diagnostic Walkthrough
TechCorp's HR assistant gives a WRONG answer about the London hotel
limit.
Step 1: check RETRIEVAL -- was the London-exception chunk in the
raw candidate set? YES, it was retrieved.
Step 2: check RANKING -- was it ranked in the FINAL top-k? NO -- it
was ranked #7, but top-k was set to k=5 (Module 15).
DIAGNOSIS: this is a RANKING failure, not a generation failure. The
fix is NOT to blame the LLM or rewrite the prompt -- it's
to either INCREASE k, improve the reranking model (Module
18), or improve the underlying embedding/hybrid search
quality (Modules 10-17) so this chunk ranks higher in the
FIRST place.
This is EXACTLY why Section 3's principle matters practically -- a
team that assumed "the LLM answered wrong" and focused ENTIRELY on
prompt engineering would have COMPLETELY missed the real, actual
problem.6. A Simple Agentic AI Connection
An agent’s multi-step task can fail at ANY of this module’s stages during ANY individual tool call within its loop — a really thorough diagnostic approach for a misbehaving agent should apply this same backward-working process to EACH search or retrieval step the agent performed, not just examine the agent’s final, overall output in isolation.
7. How Is This Used in AI?
🤖 How Is This Used in AI?
This complete failure taxonomy directly shapes how real RAG engineering teams build observability (Module 33) and debugging workflows — logging enough information at EACH pipeline stage to really distinguish a retrieval failure from a ranking failure from a generation failure, rather than only being able to observe the final, end-to-end output.
8. Real-World Applications
- Systematic debugging of underperforming production RAG systems
- Designing observability and logging strategies that capture per-stage information
- Prioritizing engineering effort toward the pipeline stage that’s really causing the most failures
9. Common Mistakes
Incorrect idea: Assuming every bad answer is a “prompt problem” or “model problem.”
Why it is incorrect: As shown directly in Section 5, the real cause is very often earlier in the pipeline — retrieval or ranking.
Incorrect idea: Not logging enough information to distinguish failure stages.
Why it is incorrect: Without visibility into what was actually retrieved, ranked, and included in context, diagnosing a failure becomes really guesswork rather than systematic.
Incorrect idea: Fixing the wrong stage based on a guess rather than real diagnosis.
Why it is incorrect: As shown directly in Section 4-5, working backward through the pipeline systematically avoids wasted effort on the wrong fix.
10. Limitations
- Even with excellent observability, some failures are really ambiguous or span multiple stages simultaneously, requiring real judgment to diagnose precisely
- This taxonomy covers the pipeline stages this course has built — real production systems may have additional custom stages with their own real failure modes
11. Quick Reference — The Whole Idea in One Diagram
Ingestion -> Parsing -> Chunking -> Embedding -> Retrieval -> Ranking
-> Context Construction -> Generation -> Citation
EVERY arrow is a place something can really go wrong.
CORE PRINCIPLE: a good LLM cannot compensate for bad retrieval
-- always check EARLIER pipeline stages before
assuming the LATER (generation) stage is at fault12. Code — Implementing a Diagnostic Pipeline
🎯 Target of this example: implement Section 4-5’s diagnostic process directly — a function that examines each pipeline stage in order and correctly identifies WHICH stage caused a given failure, exactly Section 5’s real developer example, made into reusable diagnostic code.
Example 1 — Simple
def diagnose_rag_failure(correct_chunk_id: str, retrieved_candidates: list,
final_top_k: list, final_context_chunk_ids: list) -> str:
"""Directly implements Section 4's backward-working diagnostic
process -- checking each pipeline stage in order to identify
EXACTLY where a failure occurred."""
if correct_chunk_id not in retrieved_candidates:
return "RETRIEVAL FAILURE -- the correct chunk was never found by search at all."
if correct_chunk_id not in final_top_k:
return "RANKING FAILURE -- the correct chunk was retrieved, but ranked too low for top-k."
if correct_chunk_id not in final_context_chunk_ids:
return "CONTEXT CONSTRUCTION FAILURE -- the correct chunk was in top-k, but filtered out during construction."
return "No failure detected before generation -- check GENERATION/GROUNDING (Module 23) next."
# Section 5's exact scenario: chunk was retrieved, but ranked #7,
# missing a top-5 cutoff.
retrieved_candidates = ["chunk_1", "chunk_2", "chunk_london", "chunk_4", "chunk_5", "chunk_6", "chunk_7"]
final_top_k = ["chunk_1", "chunk_2", "chunk_4", "chunk_5", "chunk_6"] # chunk_london ranked #3, dropped after re-ranking to outside top-5
final_context_chunk_ids = final_top_k
diagnosis = diagnose_rag_failure("chunk_london", retrieved_candidates, final_top_k, final_context_chunk_ids)
print(diagnosis)Expected Output:
RANKING FAILURE -- the correct chunk was retrieved, but ranked too
low for top-k.What we conclude from this example: this function correctly identifies Section 5’s exact scenario as a RANKING failure, not a retrieval or generation failure — exactly the systematic diagnosis that prevents wasted effort fixing the wrong pipeline stage.
Example 2 — Intermediate
def diagnose_rag_failure(correct_chunk_id: str, retrieved_candidates: list,
final_top_k: list, final_context_chunk_ids: list) -> str:
if correct_chunk_id not in retrieved_candidates:
return "RETRIEVAL FAILURE -- the correct chunk was never found by search at all."
if correct_chunk_id not in final_top_k:
return "RANKING FAILURE -- the correct chunk was retrieved, but ranked too low for top-k."
if correct_chunk_id not in final_context_chunk_ids:
return "CONTEXT CONSTRUCTION FAILURE -- the correct chunk was in top-k, but filtered out during construction."
return "No failure detected before generation -- check GENERATION/GROUNDING (Module 23) next."
def run_diagnostic_suite(test_cases: list) -> None:
"""Runs the diagnostic function across MULTIPLE distinct failure
scenarios, directly demonstrating that different real root
causes produce really different, correctly-identified
diagnoses."""
for case in test_cases:
diagnosis = diagnose_rag_failure(
case["correct_chunk_id"], case["retrieved_candidates"],
case["final_top_k"], case["final_context_chunk_ids"],
)
print(f"[{case['scenario_name']}] -> {diagnosis}")
test_cases = [
{"scenario_name": "Retrieval never found it",
"correct_chunk_id": "chunk_london", "retrieved_candidates": ["chunk_1", "chunk_2"],
"final_top_k": ["chunk_1", "chunk_2"], "final_context_chunk_ids": ["chunk_1", "chunk_2"]},
{"scenario_name": "Ranked too low",
"correct_chunk_id": "chunk_london", "retrieved_candidates": ["chunk_london", "chunk_1", "chunk_2", "chunk_3", "chunk_4", "chunk_5"],
"final_top_k": ["chunk_1", "chunk_2", "chunk_3", "chunk_4", "chunk_5"],
"final_context_chunk_ids": ["chunk_1", "chunk_2", "chunk_3", "chunk_4", "chunk_5"]},
{"scenario_name": "Filtered during context construction",
"correct_chunk_id": "chunk_london", "retrieved_candidates": ["chunk_london", "chunk_1"],
"final_top_k": ["chunk_london", "chunk_1"], "final_context_chunk_ids": ["chunk_1"]},
{"scenario_name": "Everything correct up to generation",
"correct_chunk_id": "chunk_london", "retrieved_candidates": ["chunk_london", "chunk_1"],
"final_top_k": ["chunk_london", "chunk_1"], "final_context_chunk_ids": ["chunk_london", "chunk_1"]},
]
run_diagnostic_suite(test_cases)Expected Output:
[Retrieval never found it] -> RETRIEVAL FAILURE -- the correct chunk
was never found by search at all.
[Ranked too low] -> RANKING FAILURE -- the correct chunk was
retrieved, but ranked too low for top-k.
[Filtered during context construction] -> CONTEXT CONSTRUCTION
FAILURE -- the correct chunk was in top-k, but filtered out during
construction.
[Everything correct up to generation] -> No failure detected before
generation -- check GENERATION/GROUNDING (Module 23) next.What we conclude from this example: all four really distinct failure scenarios are correctly diagnosed at exactly the right pipeline stage — this systematic function, run against real production logging data, would let a team immediately know which stage to investigate, rather than guessing based on the final answer alone.
Example 3 — Production Grade
from dataclasses import dataclass
from enum import Enum
class FailureStage(Enum):
RETRIEVAL = "retrieval"
RANKING = "ranking"
CONTEXT_CONSTRUCTION = "context_construction"
GENERATION_OR_NONE = "generation_or_none_detected"
@dataclass
class DiagnosticReport:
query_id: str
failure_stage: FailureStage
recommended_action: str
class RAGFailureDiagnostics:
"""A production-style diagnostics engine mapping EACH failure
stage (Section 2's taxonomy) to a SPECIFIC recommended action --
turning diagnosis into an actionable next step, not just a label."""
RECOMMENDED_ACTIONS = {
FailureStage.RETRIEVAL: "Check embedding quality (Module 10), search parameters (Modules 11-17), and whether the source chunk was properly created (Modules 7-9).",
FailureStage.RANKING: "Increase top-k (Module 15), improve reranking (Module 18), or improve underlying retrieval quality (Modules 10-17).",
FailureStage.CONTEXT_CONSTRUCTION: "Review filtering thresholds and deduplication logic (Module 21).",
FailureStage.GENERATION_OR_NONE: "Check prompt construction (Module 22) and run a groundedness check (Module 23) -- the correct information WAS available to the model.",
}
def diagnose(self, query_id: str, correct_chunk_id: str, retrieved_candidates: list,
final_top_k: list, final_context_chunk_ids: list) -> DiagnosticReport:
if correct_chunk_id not in retrieved_candidates:
stage = FailureStage.RETRIEVAL
elif correct_chunk_id not in final_top_k:
stage = FailureStage.RANKING
elif correct_chunk_id not in final_context_chunk_ids:
stage = FailureStage.CONTEXT_CONSTRUCTION
else:
stage = FailureStage.GENERATION_OR_NONE
return DiagnosticReport(
query_id=query_id, failure_stage=stage,
recommended_action=self.RECOMMENDED_ACTIONS[stage],
)
diagnostics = RAGFailureDiagnostics()
report = diagnostics.diagnose(
query_id="query_4471",
correct_chunk_id="chunk_london",
retrieved_candidates=["chunk_london", "chunk_1", "chunk_2", "chunk_3", "chunk_4", "chunk_5"],
final_top_k=["chunk_1", "chunk_2", "chunk_3", "chunk_4", "chunk_5"],
final_context_chunk_ids=["chunk_1", "chunk_2", "chunk_3", "chunk_4", "chunk_5"],
)
print(f"Query: {report.query_id}")
print(f"Failure stage: {report.failure_stage.value}")
print(f"Recommended action: {report.recommended_action}")Expected Output:
Query: query_4471
Failure stage: ranking
Recommended action: Increase top-k (Module 15), improve reranking
(Module 18), or improve underlying retrieval quality (Modules
10-17).What we conclude from this example: attaching a specific
recommended_action to each FailureStage turns this diagnostic
engine from a passive label generator into a really actionable tool
— a real engineer reviewing this report immediately knows exactly which
part of the pipeline to investigate and what kind of fix to consider,
directly operationalizing this entire module’s diagnostic principle.
13. Interview Questions
Q: State and explain the single most important principle for diagnosing RAG failures.
Ans: A good LLM cannot compensate for bad retrieval — if the really correct information never reaches the model’s context window, the model cannot reliably answer correctly, regardless of how capable it is. This means when diagnosing a bad RAG answer, you should always check the earlier pipeline stages (retrieval, ranking, context construction) before assuming the problem is with the model’s generation itself, since the real root cause is very often earlier in the pipeline.
Q: Walk through the backward diagnostic process for a RAG system producing a wrong answer.
Ans: First, check whether the really correct chunk was even present in the raw retrieved candidate set — if not, it’s a retrieval failure. If it was retrieved but didn’t survive into the final top-k results, it’s a ranking failure. If it survived ranking but was filtered out during context construction, that’s a context construction failure. Only if the correct chunk made it all the way into the final context given to the model should you investigate generation itself — checking prompt construction and running a groundedness check.
Q: Why is it important to log information at each individual pipeline stage, rather than only observing the final generated answer?
Ans: Without visibility into what was retrieved, how it was ranked, and what was ultimately included in the constructed context, diagnosing a bad answer becomes real guesswork rather than a systematic process. Logging at each stage allows a team to directly determine which specific stage failed, following the backward diagnostic process, rather than only seeing the final output and having to speculate about where in the pipeline the actual problem occurred.
Q: Describe a real scenario where assuming “the LLM answered wrong” would lead a team to fix the wrong problem, and explain what the real root cause might actually be.
Ans: If a RAG system gives a wrong answer because the really correct chunk was retrieved but ranked too low to survive a top-k cutoff, a team that assumes the LLM is at fault might spend significant effort rewriting prompts or trying different models — none of which would fix the actual problem, since the model never even saw the correct information. The real fix would be increasing the top-k value, improving the reranking stage, or improving the underlying retrieval quality so the correct chunk ranks higher in the first place — a completely different area of the system than prompt engineering.
14. What You Should Remember
- The complete RAG pipeline has failure points at every single stage — ingestion through generation — not just at the final generation step.
- A good LLM cannot compensate for bad retrieval — this is the single most important diagnostic principle, always check earlier stages first.
- A systematic, backward diagnostic process — verified directly through a working diagnostic function correctly identifying four really distinct failure scenarios at exactly the right stage — prevents wasted effort fixing the wrong part of the pipeline.
15. Quick Practice
For a RAG system where a user reports “the answer mentioned a policy detail that doesn’t actually exist anywhere in our documents,” walk through this module’s diagnostic process to determine which specific failure stage this scenario most likely represents.
16. Next Step
Next: Module 25 — Hallucination in RAG & Conflicting Documents — going deeper into the generation-stage failure this module’s diagnostic process points toward, and the practical problem of multiple documents disagreeing with each other.
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed