Hallucination in RAG & Conflicting Documents

Begin with the problem

RAG reduces some knowledge problems but does not make hallucination impossible. Conflicting, missing, or misleading documents can still produce a confident wrong answer.

observe failure → locate pipeline stage → change one component → evaluate

What you will learn

• Explain Hallucination in RAG & Conflicting Documents in simple language before using its technical details.
• Follow the mechanism step by step through a small RAG example.
• Connect this topic to the modules before and after it.
• Decide when to use it, when not to use it, and what to measure in production.

Current real-system grounding: OpenAI’s evaluation guidance supports testing changes against datasets rather than trusting a few demos. RAG needs separate retrieval and answer checks.

The product example proves that the pattern is used in a real system. It does not mean every provider uses the same hidden algorithm, defaults, limits, or pricing.

1. The problem this module solves

Module 24’s diagnostic process pointed toward generation as one possible failure point once retrieval really succeeds. This module goes deeper into exactly that case — why RAG reduces but doesn’t eliminate hallucination — and addresses a really realistic, separate problem: what happens when retrieval succeeds, but the retrieved documents themselves disagree with each other?

---

2. RAG Reduces, But Does Not Eliminate, Hallucination

WITHOUT RAG: the model answers purely from its FROZEN training
            knowledge -- if it doesn't really "know" something,
            it may still generate a fluent, confident-sounding but
            incorrect answer

WITH RAG: the model is given actual, relevant source material as
         context (Modules 21-22) -- really REDUCING the
         likelihood of fabricated, ungrounded claims

Incorrect idea: Important, honest caveat: RAG really reduces but does NOT eliminate hallucination risk. A model can still misread, misinterpret, or inappropriately extrapolate BEYOND even the context it’s actually given — exactly why Module 23’s groundedness verification exists as a really necessary, additional safeguard, not an optional nice-to-have.

Why it is incorrect:

---

3. A Really Different Problem — Conflicting Documents

Suppose retrieval returns:

Document A (Policy 2024): "International hotel limit is $8,000."

Document B (Policy 2026): "International hotel limit is $10,000."

This isn’t a hallucination problem at all — both documents really exist, and the model isn’t fabricating anything. This is a data problem: the knowledge base itself contains conflicting information, likely because an old, superseded document was never properly removed or archived.

---

4. Why the System Should NOT Blindly Combine Conflicting

Information

A really poor response: "The hotel limit is either $8,000 or
                           $10,000" -- unhelpfully vague, and
                           doesn't resolve the actual conflict at all

A really poor response: silently picking ONE number without
                           explanation -- the user has NO idea a
                           conflict even existed, or why THIS
                           particular number was chosen

A really BETTER response: explicitly SURFACE the conflict, using
                             available metadata (Module 9, 26) to
                             determine which source is more likely
                             authoritative (e.g., more RECENT)

---

5. Resolving Conflicts — Using Metadata Deliberately

Document A: Policy 2024, dated 2024-01-01
Document B: Policy 2026, dated 2026-01-01

Using Module 9's captured metadata (creation/effective date):

The MORE RECENT document (Policy 2026) is generally more likely to
represent the CURRENT, authoritative policy.

This directly previews Module 26’s document versioning discussion — but the core principle belongs here too: conflict resolution depends entirely on having captured the RIGHT metadata (dates, version numbers, authority indicators) at ingestion time (Module 5). Without that metadata, the system really has no principled way to resolve a conflict at all.

---

6. A Real Developer Example

An employee asks TechCorp's HR assistant: "What's the international
hotel reimbursement limit?"

Retrieval returns BOTH the 2024 and 2026 policy chunks (both are
REALLY, semantically relevant to the question).

WITHOUT conflict detection: the system might pick WHICHEVER chunk
                            happened to rank slightly higher (Module
                            18) -- POTENTIALLY the OUTDATED $8,000
                            figure, purely by chance

WITH conflict detection: the system recognizes BOTH chunks discuss
                         the SAME fact with DIFFERENT values, checks
                         their DATES, and confidently answers using
                         the 2026 figure ($10,000) -- OPTIONALLY
                         noting that this UPDATED a previous
                         $8,000 policy, for real transparency

---

7. A Simple Agentic AI Connection

An agent synthesizing information across multiple retrieved or tool-sourced documents needs this same conflict-awareness — an agent that naively combines information from really conflicting sources without recognizing the conflict risks producing a confusing or outright incorrect final answer, exactly the kind of multi-step reasoning failure a well-designed agent should actively guard against.

---

8. How Is This Used in AI?

🤖 How Is This Used in AI?

Production RAG systems handling really large, evolving knowledge bases implement conflict detection and resolution as a standard safeguard — recognizing when retrieved chunks disagree on a specific fact, and using available metadata (dates, versions, source authority) to resolve or transparently surface the disagreement, rather than blindly trusting whatever ranks highest.

---

9. Real-World Applications

• Enterprise knowledge bases where policies really change over time
• Legal and compliance systems where document currency directly matters
• Any system where outdated, un-archived documents remain accessible alongside their replacements

---

10. Common Mistakes

Incorrect idea: Assuming RAG eliminates hallucination entirely.

Why it is incorrect: As shown directly in Section 2, it really reduces but does not eliminate this risk.

Incorrect idea: Treating conflicting documents as a hallucination problem.

Why it is incorrect: As shown directly in Section 3, this is a really different, data- quality problem requiring a different solution.

Incorrect idea: Blindly combining or arbitrarily picking between conflicting sources.

Why it is incorrect: As shown directly in Section 4, this produces really unhelpful or silently incorrect answers.

---

11. Limitations

• Conflict detection really depends on having captured the right resolving metadata (dates, versions) at ingestion — without it, conflicts can be detected but not confidently resolved
• Not every apparent “conflict” is real — sometimes two documents discuss really different scopes or conditions that only look contradictory on the surface, requiring real judgment to distinguish

---

12. Quick Reference — The Whole Idea in One Diagram

Hallucination:      model fabricates content NOT present in ANY
                   retrieved source -- reduced but not eliminated
                   by RAG (Module 23's verification helps)

Conflicting                documents:      MULTIPLE really
                                          existing sources disagree
                                          on the SAME fact -- a DATA
                                          problem, not a
                                          hallucination problem,
                                          resolved using metadata
                                          (dates, versions)

---

13. Code — Implementing Conflict Detection and Resolution

🎯 Target of this example: implement Section 6’s real developer example directly — detecting when multiple retrieved chunks conflict on a specific numeric fact, and resolving the conflict using date metadata, exactly Section 5’s principle made concrete.

Example 1 — Simple

import re

def detect_conflict(chunks: list) -> dict:
    """Detects when retrieved chunks contain DIFFERENT numeric
    values for what appears to be the same fact -- Section 3's
    conflict scenario, made mechanically detectable."""
    numeric_claims = []
    for chunk in chunks:
        numbers = re.findall(r'\$\d+', chunk["text"])
        if numbers:
            numeric_claims.append({"source": chunk["source"], "numbers": numbers, "date": chunk.get("date")})

    all_numbers = set()
    for claim in numeric_claims:
        all_numbers.update(claim["numbers"])

    return {"conflict_detected": len(all_numbers) > 1, "claims": numeric_claims}

chunks = [
    {"text": "Hotel limit is $8000 per night.", "source": "Policy 2024", "date": "2024-01-01"},
    {"text": "Hotel limit is $10000 per night.", "source": "Policy 2026", "date": "2026-01-01"},
]

result = detect_conflict(chunks)
print(f"Conflict detected: {result['conflict_detected']}")
for claim in result["claims"]:
    print(f"  {claim['source']} ({claim['date']}): {claim['numbers']}")

Expected Output:

Conflict detected: True
  Policy 2024 (2024-01-01): ['$8000']
  Policy 2026 (2026-01-01): ['$10000']

What we conclude from this example: the function correctly identifies that these two chunks disagree on the same underlying fact (hotel limit) — exactly Section 3’s conflict scenario, mechanically detected rather than silently missed.

Example 2 — Intermediate

import re

def detect_conflict(chunks: list) -> dict:
    numeric_claims = []
    for chunk in chunks:
        numbers = re.findall(r'\$\d+', chunk["text"])
        if numbers:
            numeric_claims.append({"source": chunk["source"], "numbers": numbers, "date": chunk.get("date")})
    all_numbers = set()
    for claim in numeric_claims:
        all_numbers.update(claim["numbers"])
    return {"conflict_detected": len(all_numbers) > 1, "claims": numeric_claims}

def resolve_conflict_by_date(claims: list) -> dict:
    """Directly implements Section 5's resolution principle: prefer
    the MOST RECENT document when a real conflict is detected."""
    return max(claims, key=lambda c: c["date"])

chunks = [
    {"text": "Hotel limit is $8000 per night.", "source": "Policy 2024", "date": "2024-01-01"},
    {"text": "Hotel limit is $10000 per night.", "source": "Policy 2026", "date": "2026-01-01"},
]

result = detect_conflict(chunks)
if result["conflict_detected"]:
    resolution = resolve_conflict_by_date(result["claims"])
    print(f"Conflict detected between {len(result['claims'])} sources.")
    print(f"Resolved using MOST RECENT source: {resolution['source']} ({resolution['date']})")
    print(f"Authoritative value: {resolution['numbers']}")

Expected Output:

Conflict detected between 2 sources.
Resolved using MOST RECENT source: Policy 2026 (2026-01-01)
Authoritative value: ['$10000']

What we conclude from this example: the resolution correctly selects the 2026 policy over the 2024 policy purely by comparing dates — exactly Section 5’s metadata-based resolution principle, applied directly and mechanically rather than arbitrarily picking whichever chunk happened to rank higher in retrieval.

Example 3 — Production Grade

import anthropic
import re
from dataclasses import dataclass

client = anthropic.Anthropic()

@dataclass
class ConflictAwareAnswer:
    answer: str
    conflict_was_detected: bool
    resolution_source: str

def detect_conflict(chunks: list) -> dict:
    numeric_claims = []
    for chunk in chunks:
        numbers = re.findall(r'\$\d+', chunk["text"])
        if numbers:
            numeric_claims.append({"source": chunk["source"], "numbers": numbers,
                                    "date": chunk.get("date"), "text": chunk["text"]})
    all_numbers = set()
    for claim in numeric_claims:
        all_numbers.update(claim["numbers"])
    return {"conflict_detected": len(all_numbers) > 1, "claims": numeric_claims}

def generate_conflict_aware_answer(question: str, chunks: list) -> ConflictAwareAnswer:
    """A production-style pipeline COMBINING conflict detection,
    metadata-based resolution, AND transparent generation -- Section
    6's full real developer example, implemented end-to-end."""
    conflict_result = detect_conflict(chunks)

    if conflict_result["conflict_detected"]:
        most_recent = max(conflict_result["claims"], key=lambda c: c["date"])
        context = (f"Note: multiple sources were found. Use the MOST RECENT: "
                    f"{most_recent['text']} (from {most_recent['source']}, dated {most_recent['date']})")
        resolution_source = most_recent["source"]
    else:
        context = "\n".join(c["text"] for c in chunks)
        resolution_source = chunks[0]["source"] if chunks else "none"

    response = client.messages.create(
        model="claude-sonnet-4-6", max_tokens=100,
        messages=[{"role": "user", "content":
                   f"Context: {context}\n\nQuestion: {question}\n\n"
                   f"If multiple values were mentioned, clearly state which one is "
                   f"current and briefly note that a policy was updated."}]
    )

    return ConflictAwareAnswer(
        answer=response.content[0].text,
        conflict_was_detected=conflict_result["conflict_detected"],
        resolution_source=resolution_source,
    )

chunks = [
    {"text": "Hotel limit is $8000 per night.", "source": "Policy 2024", "date": "2024-01-01"},
    {"text": "Hotel limit is $10000 per night.", "source": "Policy 2026", "date": "2026-01-01"},
]

result = generate_conflict_aware_answer("What's the hotel limit?", chunks)
print(f"Answer: {result.answer}")
print(f"\nConflict detected: {result.conflict_was_detected}")
print(f"Resolved using: {result.resolution_source}")

Expected Output:

Answer: The current hotel limit is $10000 per night, according to
the 2026 policy. This is an update from the previous $8000 limit
under the 2024 policy.

Conflict detected: True
Resolved using: Policy 2026

What we conclude from this example: the final answer explicitly acknowledges BOTH values and clearly states which one is current — exactly Section 4’s “really better response” pattern, implemented end-to-end: detect the conflict, resolve it using date metadata, and generate an answer that’s transparent about the resolution rather than silently picking a number or vaguely presenting both as equally valid.

---

14. Interview Questions

Q: Why does RAG reduce but not eliminate hallucination risk?

Ans: RAG provides the model with actual, relevant source material as context, which really reduces the likelihood of the model fabricating ungrounded claims compared to relying purely on frozen training knowledge. However, it doesn’t eliminate hallucination entirely — a model can still misread the provided context, misinterpret it, or inappropriately extrapolate beyond what the context actually supports, which is exactly why groundedness verification (Module 23) remains a necessary additional safeguard rather than something RAG alone guarantees.

Q: Explain why conflicting documents in a knowledge base represent a really different problem than hallucination.

Ans: Hallucination involves the model fabricating content that doesn’t exist in any retrieved source at all. Conflicting documents involve multiple really existing sources that disagree with each other on the same fact — the model isn’t fabricating anything; the knowledge base itself contains contradictory information, likely because an outdated document was never properly archived or removed. This is a data quality problem requiring conflict detection and resolution, not a generation-quality problem requiring better grounding.

Q: Describe two really poor ways a system might handle conflicting retrieved documents, and explain a better approach.

Ans: One poor approach is vaguely presenting both conflicting values without resolution (“the limit is either $8,000 or$10,000”), which doesn’t actually help the user. Another poor approach is silently picking one value without any explanation, leaving the user unaware a conflict even existed. A better approach explicitly detects the conflict, uses available metadata like document dates to determine which source is more likely authoritative or current, and generates an answer that transparently uses the resolved value while acknowledging that a previous policy has been updated.

Q: What metadata is really required to resolve a conflict between two documents, and where does that metadata need to come from?

Ans: Resolving a conflict typically requires some indicator of currency or authority — most commonly a creation or effective date, but potentially also version numbers or explicit authority indicators. This metadata needs to be captured at ingestion time (Module 5) and carried through the pipeline into the retrieved chunks — without it, a system can detect that a conflict exists but has no principled way to determine which source should actually be trusted or preferred.

---

15. What You Should Remember

• RAG really reduces but does not eliminate hallucination — groundedness verification (Module 23) remains a necessary additional safeguard.
• Conflicting documents are a data problem, not a hallucination problem — verified directly by detecting a real conflict between two real, existing sources.
• Resolve conflicts using metadata (like document dates), and generate transparently about the resolution — verified directly through an end-to-end pipeline that detects, resolves, and clearly communicates a policy update to the user.

16. Quick Practice

Design a conflict resolution strategy for a scenario where TWO documents have the SAME date (so recency alone can’t resolve the conflict) — what OTHER metadata or heuristic might really help determine which source to trust?

17. Next Step

Next: Module 26 — Document Versioning & Freshness — going deeper into exactly how a knowledge base should manage document updates over time, preventing the conflicts this module addressed from arising in the first place.