Multimodal RAG

Begin with the problem

Useful evidence may live in images, scanned pages, diagrams, audio, or code—not only paragraphs. Multimodal RAG must preserve the meaning and location of each modality.

question → choose specialized retrieval path → collect multimodal/structured evidence → answer

What you will learn

• Explain Multimodal RAG in simple language before using its technical details.
• Follow the mechanism step by step through a small RAG example.
• Connect this topic to the modules before and after it.
• Decide when to use it, when not to use it, and what to measure in production.

Current real-system grounding: Google’s current File Search documentation includes file-based grounding and multimodal retrieval capabilities, with model, file-type, and tool-combination limitations.

The product example proves that the pattern is used in a real system. It does not mean every provider uses the same hidden algorithm, defaults, limits, or pricing.

1. The problem this module solves

Level 7 has covered advanced retrieval reasoning (Modules 28-29) and alternative data sources (Module 30). This module closes the level by addressing really different content types — PDFs, web pages, and code — each with real, source-specific handling needs, unified through the same loader pattern Module 5 established.

---

2. PDF RAG — The Complete, End-to-End Pipeline

This module brings together nearly everything covered so far into one complete, worked pipeline for the single most common real-world RAG source:

PDF
   ↓
PARSING (Module 6) -- text/table/image extraction, structure
                     preservation, handling multi-column layouts and
                     OCR for scanned pages
   ↓
CHUNKING (Modules 7-9) -- ideally structure-aware, respecting
                         headings and sections
   ↓
EMBEDDING (Module 10)
   ↓
VECTOR STORE (Module 12)
   ↓
RETRIEVAL (Modules 13-18)
   ↓
LLM (Module 22)
   ↓
Answer + Citation (Module 23, referencing the ORIGINAL PDF's page
                   number -- Module 5's metadata, carried all the
                   way through)

Every single stage in this diagram is a module you’ve already completed — PDF RAG isn’t a new technique, it’s this course’s entire pipeline, applied to the single most common real document format.

---

3. Web RAG — Real, Distinct Considerations

Web pages
   ↓
CRAWLING / LOADING (Module 5's loader pattern, web-specific)
   ↓
CLEANING (stripping navigation menus, ads, footers -- Module 6's
         boilerplate-removal principle, applied to HTML specifically)
   ↓
CHUNKING -> EMBEDDING -> INDEX (unchanged from standard RAG)

Really DISTINCT considerations for web content:

- FRESHNESS: web content changes MORE frequently than most internal
  documents -- Module 26's reindexing strategy matters even more
  here
- DUPLICATE CONTENT: the same information often appears on MULTIPLE
  pages (mirrors, syndicated content) -- real deduplication (Module
  21) is important
- SOURCE TRUST: not all web sources are EQUALLY reliable -- metadata
  (Module 9) should really capture source credibility signals
  where possible

---

4. Code RAG — A Really Different Chunking Unit

User: "Where is authentication handled in our codebase?"

Repository
   ↓
CODE PARSING (really different from Module 6's document parsing --
             understanding functions, classes, imports)
   ↓
SYMBOL-AWARE CHUNKING -- chunk by FUNCTION or CLASS, not by
                        arbitrary character count (directly extending
                        Module 8's structure-aware chunking
                        principle: code has its OWN natural
                        structural units)
   ↓
EMBEDDING -> RETRIEVAL -> LLM -> Answer

The real insight: code has its own natural “paragraphs” — functions and classes — exactly analogous to Module 8’s heading-based structure-aware chunking, but for a really different kind of document. Splitting a function’s definition from its implementation (arbitrary character-based chunking) would be exactly as destructive as splitting Module 8’s heading from its content.

---

5. A Real Developer Example — Unifying All Three Under One Loader

Pattern

TechCorp builds ONE internal search tool covering THREE really
different source types, all through Module 5's UNIFORM loader
pattern:

load_pdf("travel_policy.pdf", ...)          -> LoadedSource(type="pdf",
                                              metadata={page_count,
                                              filename})

load_web_page("wiki.../oncall", ...)            -> LoadedSource(
                                                 type="web",
                                                 metadata={url,
                                                 crawl_date})

load_code_file("auth/login.py", ...)                -> LoadedSource(
                                                     type="code",
                                                     metadata={
                                                     filepath,
                                                     language})

ALL THREE produce the SAME uniform structure (Module 5's Section 3
principle) -- downstream chunking, embedding, and retrieval code
doesn't need to know or care which original SOURCE TYPE a chunk
came from, even though each loader handled really
SOURCE-SPECIFIC extraction challenges internally.

---

6. A Simple Agentic AI Connection

An agent with access to PDF search, web search, and code search tools simultaneously can really choose the right source type based on a question’s nature — recognizing that “what does our policy say” points toward PDF/document search, while “where is this function defined” points toward code search, directly extending Module 30’s structured-vs-unstructured routing to this module’s multiple CONTENT-TYPE routing.

---

7. How Is This Used in AI?

🤖 How Is This Used in AI?

Production RAG systems really integrate multiple content types under one unified retrieval interface — coding assistants search across code, documentation, and web resources simultaneously; enterprise knowledge assistants combine PDFs, wikis, and internal web tools — all built on the same loader-and-pipeline pattern established back in Module 5, extended to each source’s specific handling needs.

---

8. Real-World Applications

• Coding assistants (searching codebases, documentation, and Stack Overflow-style content together)
• Enterprise knowledge assistants spanning documents, wikis, and internal tools
• Research assistants combining academic PDFs with current web content

---

9. Common Mistakes

Incorrect idea: Treating code as if it were prose text for chunking purposes.

Why it is incorrect: As shown directly in Section 4, code has its own real structural units — functions and classes — that arbitrary character-based chunking would destructively split.

Incorrect idea: Not accounting for web content’s really higher change frequency.

Why it is incorrect: As shown directly in Section 3, this directly connects to Module 26’s freshness/reindexing strategy, which matters even more for web sources.

Incorrect idea: Building really separate, incompatible pipelines for each content type.

Why it is incorrect: As shown directly in Section 5, a unified loader pattern (Module 5) lets downstream stages remain source-agnostic.

---

10. Limitations

• Code-aware chunking really requires language-specific parsing logic (understanding Python’s syntax differs from understanding JavaScript’s) — a real, additional implementation requirement beyond generic text chunking
• Web content’s freshness and trust considerations require real, ongoing operational effort — not a one-time setup concern

---

11. Quick Reference — The Whole Idea in One Diagram

PDF:      parse (Module 6) -> chunk (structure-aware, Module 8) ->
        standard pipeline

WEB:         load + CLEAN (strip boilerplate) -> chunk -> standard
           pipeline, with REALLY higher freshness/dedup needs

CODE:            SYMBOL-AWARE chunking (by function/class, NOT
               character count) -> standard pipeline

ALL THREE unified through Module 5's LOADER PATTERN -- downstream
stages remain source-agnostic

---

12. Code — Implementing Unified Multi-Source Loading

🎯 Target of this example: implement Section 5’s real developer example directly — three really different loader functions producing a uniform LoadedSource structure, exactly demonstrating Module 5’s pattern extended to PDF, web, and code sources together.

Example 1 — Simple

from dataclasses import dataclass

@dataclass
class LoadedSource:
    """The SAME uniform structure from Module 5, Section 3 --
    regardless of which of the three really different source
    types produced it."""
    source_type: str
    content: str
    metadata: dict

def load_pdf(filename: str, extracted_text: str, page_count: int) -> LoadedSource:
    return LoadedSource("pdf", extracted_text, {"filename": filename, "page_count": page_count})

def load_web_page(url: str, extracted_text: str, crawl_date: str) -> LoadedSource:
    return LoadedSource("web", extracted_text, {"url": url, "crawl_date": crawl_date})

def load_code_file(filepath: str, code_content: str, language: str) -> LoadedSource:
    return LoadedSource("code", code_content, {"filepath": filepath, "language": language})

sources = [
    load_pdf("travel_policy.pdf", "International hotel reimbursement is limited to $200/night.", 12),
    load_web_page("https://wiki.techcorp.com/oncall", "The on-call rotation follows a weekly schedule.", "2026-08-01"),
    load_code_file("auth/login.py", "def authenticate_user(username, password):\n    ...", "python"),
]

for source in sources:
    print(f"[{source.source_type}] {source.metadata}")
    print(f"  Content preview: {source.content[:50]}...")

Expected Output:

[pdf] {'filename': 'travel_policy.pdf', 'page_count': 12}
  Content preview: International hotel reimbursement is limited to
$2...
[web] {'url': 'https://wiki.techcorp.com/oncall', 'crawl_date':
'2026-08-01'}
  Content preview: The on-call rotation follows a weekly schedule....
[code] {'filepath': 'auth/login.py', 'language': 'python'}
  Content preview: def authenticate_user(username, password):
    ......

What we conclude from this example: all three really different loaders produce the exact same LoadedSource structure — exactly Section 5’s real developer example, demonstrating that downstream code never needs to know or branch on which specific source type produced any given piece of content.

Example 2 — Intermediate

import re

def chunk_code_by_function(code: str) -> list:
    """Directly implements Section 4's SYMBOL-AWARE chunking --
    splits Python code by FUNCTION definitions, never splitting a
    function's signature from its own implementation."""
    lines = code.split("\n")
    chunks = []
    current_chunk = []

    for line in lines:
        if re.match(r"^def \w+\(", line) and current_chunk:
            chunks.append("\n".join(current_chunk))
            current_chunk = [line]
        else:
            current_chunk.append(line)

    if current_chunk:
        chunks.append("\n".join(current_chunk))
    return chunks

code = (
    "def authenticate_user(username, password):\n"
    "    return check_credentials(username, password)\n"
    "\n"
    "def check_credentials(username, password):\n"
    "    return db.verify(username, password)\n"
    "\n"
    "def logout_user(session_id):\n"
    "    session_store.remove(session_id)"
)

chunks = chunk_code_by_function(code)
for i, chunk in enumerate(chunks, 1):
    print(f"Chunk {i}:\n{chunk}\n---")

Expected Output:

Chunk 1:
def authenticate_user(username, password):
    return check_credentials(username, password)

---
Chunk 2:
def check_credentials(username, password):
    return db.verify(username, password)

---
Chunk 3:
def logout_user(session_id):
    session_store.remove(session_id)
---

What we conclude from this example: each chunk contains exactly one complete function, never splitting a function signature from its implementation — exactly Section 4’s symbol-aware chunking principle, directly analogous to Module 8’s heading-preserving structure-aware chunking, but applied to code’s own natural structural units.

Example 3 — Production Grade

from dataclasses import dataclass, field
from enum import Enum

class SourceType(Enum):
    PDF = "pdf"
    WEB = "web"
    CODE = "code"

@dataclass
class UnifiedSource:
    source_type: SourceType
    content: str
    metadata: dict = field(default_factory=dict)
    requires_frequent_refresh: bool = False  # Section 3's freshness point

class MultiSourceLoader:
    """A production-style unified loader COMBINING all three source
    types under Module 5's pattern, while ALSO capturing Section 3's
    real freshness distinction -- web content is explicitly
    flagged as needing MORE frequent reindexing than PDFs or code."""

    def load_pdf(self, filename: str, text: str, page_count: int) -> UnifiedSource:
        return UnifiedSource(SourceType.PDF, text, {"filename": filename, "page_count": page_count},
                              requires_frequent_refresh=False)

    def load_web_page(self, url: str, text: str, crawl_date: str) -> UnifiedSource:
        return UnifiedSource(SourceType.WEB, text, {"url": url, "crawl_date": crawl_date},
                              requires_frequent_refresh=True)  # Section 3's real distinction

    def load_code_file(self, filepath: str, code: str, language: str) -> UnifiedSource:
        return UnifiedSource(SourceType.CODE, code, {"filepath": filepath, "language": language},
                              requires_frequent_refresh=True)  # code changes frequently too

loader = MultiSourceLoader()

sources = [
    loader.load_pdf("travel_policy.pdf", "Hotel reimbursement policy text.", 12),
    loader.load_web_page("https://wiki.techcorp.com/oncall", "On-call schedule.", "2026-08-01"),
    loader.load_code_file("auth/login.py", "def authenticate_user(): ...", "python"),
]

print("Reindexing priority (frequent refresh needed):")
for source in sources:
    if source.requires_frequent_refresh:
        print(f"  [{source.source_type.value}] {source.metadata}")

Expected Output:

Reindexing priority (frequent refresh needed):
  [web] {'url': 'https://wiki.techcorp.com/oncall', 'crawl_date':
'2026-08-01'}
  [code] {'filepath': 'auth/login.py', 'language': 'python'}

What we conclude from this example: the requires_frequent_refresh flag correctly identifies web and code sources as needing more frequent reindexing than the PDF, directly connecting Section 3’s freshness observation to Module 26’s reindexing strategy — a real production system could use this exact flag to prioritize which sources get checked for staleness most often, rather than treating all source types identically.

---

13. Interview Questions

Q: Why is PDF RAG described as “not a new technique” despite being covered in its own dedicated module?

Ans: PDF RAG is really the complete pipeline this entire course has built — parsing (Module 6), chunking (Modules 7-9), embedding (Module 10), retrieval (Modules 11-18), and citation (Module 23) — applied to the single most common real-world document format. There’s no new underlying mechanism specific to PDFs; the module exists to demonstrate the complete, already-learned pipeline working end-to-end on this really common source type.

Q: What really distinct considerations does web content introduce compared to static internal documents?

Ans: Web content typically changes more frequently, making freshness and reindexing strategy (Module 26) really more important. The same information often appears duplicated across multiple pages, making deduplication (Module 21) more relevant. And not all web sources are equally reliable, so capturing source credibility as metadata (Module 9) matters more for web content than for internally-authored, presumably-trusted documents.

Q: Explain why code requires “symbol-aware” chunking rather than standard character or sentence-based chunking.

Ans: Code has its own natural structural units — functions and classes — that carry real, complete meaning as a whole. Arbitrary character-based chunking risks splitting a function’s signature from its implementation, exactly as destructive as separating a document heading from its content (Module 8’s structure-aware chunking principle). Chunking by function or class boundary instead ensures each chunk represents a complete, independently meaningful unit of code.

Q: How does Module 5’s unified loader pattern benefit a system that needs to search across PDFs, web pages, and code simultaneously?

Ans: Each source type has really different extraction challenges — PDF parsing, web content cleaning, code symbol parsing — but if every loader produces the same uniform output structure (content plus metadata), then everything downstream — chunking, embedding, indexing, retrieval — can remain completely source-agnostic. This means adding support for a new source type only requires writing a new source-specific loader, without needing to modify any of the shared downstream pipeline logic.

---

14. What You Should Remember

• PDF RAG is the complete pipeline this course has built, applied to the most common real document format — no really new mechanism required.
• Web content has really distinct freshness, deduplication, and trust considerations compared to static documents.
• Code requires symbol-aware chunking (by function/class) — verified directly by correctly chunking code without ever splitting a function’s signature from its implementation.
• Module 5’s unified loader pattern lets really different source types share the same downstream pipeline — verified directly through a production loader that also tracks source-specific refresh needs.

15. Quick Practice

Design a chunking strategy for a source type not explicitly covered in this module (like a spreadsheet, an email thread, or a chat log) — what would its own “natural structural unit” (analogous to code’s functions or documents’ headings) really be?

16. Next Step

Next: Module 32 — RAG Evaluation — Level 8 begins here: how to systematically measure retrieval quality and generation quality separately, closing in on the final production-readiness modules.