Document Loaders: Bringing Real Files Into LangChain

Every retrieval pipeline starts with real, raw content sitting somewhere — a folder of text files, a PDF, a web page, a spreadsheet. Document loaders are LangChain’s answer to “how do I get that content into a form the rest of the pipeline can actually use.”

The Document object, properly explained

Every loader, regardless of source, produces the same output shape:

from langchain_core.documents import Document

doc = Document(
    page_content="Our return policy allows returns within 30 days of purchase.",
    metadata={"source": "policies.txt", "category": "returns"},
)

print(doc.page_content)
print(doc.metadata)

page_content is the actual text. metadata is a dictionary of anything else worth knowing about where this content came from — its source file, a page number, an author, a date. This matters more than it might seem: metadata is what lets you later filter retrieval results (“only search documents from this category”), or show a user exactly which source an answer came from — a real, practical need in genuinely trustworthy RAG applications.

Example 1: loading a plain text file

from langchain_community.document_loaders import TextLoader

loader = TextLoader("policies.txt")
documents = loader.load()

print(len(documents))
print(documents[0].page_content[:100])
print(documents[0].metadata)

documents[0].metadata already contains {"source": "policies.txt"}, filled in automatically — the loader knows where the content came from without you specifying it yourself.

Example 2: loading a PDF

from langchain_community.document_loaders import PyPDFLoader

loader = PyPDFLoader("employee_handbook.pdf")
documents = loader.load()

print(f"Loaded {len(documents)} pages.")
print(documents[0].metadata)

Notice this typically produces one Document per page, not one giant document for the whole PDF — each page’s metadata includes its page number, genuinely useful later when you want to tell a user exactly which page an answer’s source came from.

Example 3: loading a web page

from langchain_community.document_loaders import WebBaseLoader

loader = WebBaseLoader("https://example.com/faq")
documents = loader.load()

print(documents[0].page_content[:200])
print(documents[0].metadata)

This fetches the live page and extracts its readable text content — genuinely useful for building a RAG system over your own website’s public documentation or FAQ.

Example 4: loading a CSV

from langchain_community.document_loaders import CSVLoader

loader = CSVLoader("products.csv")
documents = loader.load()

print(f"Loaded {len(documents)} rows.")
print(documents[0].page_content)

Each row of the CSV becomes its own Document — useful for making structured, tabular data, like a product catalog, individually searchable.

Example 5: loading an entire directory at once

from langchain_community.document_loaders import DirectoryLoader

loader = DirectoryLoader("./company_docs", glob="**/*.txt")
documents = loader.load()

print(f"Loaded {len(documents)} documents from the directory.")
for doc in documents:
    print("-", doc.metadata["source"])

DirectoryLoader walks an entire folder structure, applying a matching loader to every file that fits the glob pattern — genuinely practical for building a RAG system over an entire existing folder of company documents, rather than loading each file individually by hand.

Common mistakes worth avoiding

Assuming every loader produces exactly one Document per file. Recall Example 2 — PyPDFLoader genuinely produces one Document per page, not per file. Code that assumes len(documents) == 1 for a single PDF will break in a confusing way the first time it’s given a multi-page file.

Ignoring metadata because it isn’t immediately needed. It’s tempting to treat metadata as optional boilerplate, but recall Module 26’s real payoff — genuine source citations depend entirely on metadata being captured correctly here, at the very start of the pipeline. Skipping it now means retrofitting it later, across every document you’ve already loaded.

Loading an entire directory without checking what actually got matched. Recall Example 5’s glob="**/*.txt" — a mistyped pattern can silently load zero files, or far more files than intended, with no obvious error. Always check len(documents) after a DirectoryLoader call before assuming it worked as expected.

What you should take away from this module

• Every loader, regardless of source, produces the same shape: a list of Document objects, each with page_content and metadata.
• metadata is filled in automatically with real, useful context — source file, page number — and matters for filtering and source attribution later.
• LangChain has a real, dedicated loader for essentially every common content type: text, PDF, web pages, CSV, and entire directories at once.

Where this goes next

The next module covers Text Splitting — because a loaded document, especially a long PDF or web page, is almost never the right size to hand directly to an embedding model or a retriever. You’ll learn how, and how much, to break it apart.