Building RAG in LangChain, Version by Version

Every piece is now on the table: loaders, splitters, embeddings, vector stores, retrievers, structured output, chains, memory. This module’s entire job is assembly — building one real RAG system, growing it version by version, so you feel exactly why each addition earns its place.

Setup, shared across every version below

from langchain_core.documents import Document
from langchain_openai import OpenAIEmbeddings
from langchain_core.vectorstores import InMemoryVectorStore

docs = [
    Document(page_content="Our return policy allows returns within 30 days of purchase.", metadata={"source": "returns.txt"}),
    Document(page_content="Refunds are processed within 5-7 business days after we receive the item.", metadata={"source": "refunds.txt"}),
    Document(page_content="Shipping typically takes 3-5 business days within the country.", metadata={"source": "shipping.txt"}),
]

vector_store = InMemoryVectorStore(OpenAIEmbeddings(model="text-embedding-3-small"))
vector_store.add_documents(docs)

Version 1: manual retrieval

from langchain.chat_models import init_chat_model

model = init_chat_model("openai:gpt-4o-mini")

def ask(question: str) -> str:
    results = vector_store.similarity_search(question, k=2)
    context = "\n".join(d.page_content for d in results)
    prompt = f"Answer using only this context:\n{context}\n\nQuestion: {question}"
    return model.invoke(prompt).content

print(ask("How long until I get my refund?"))

Every step written out by hand — genuinely the starting point everyone should understand before reaching for an abstraction.

Version 2: the retriever abstraction

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

retriever = vector_store.as_retriever(search_kwargs={"k": 2})
parser = StrOutputParser()

prompt = ChatPromptTemplate.from_template(
    "Answer using only this context:\n{context}\n\nQuestion: {question}"
)

def format_docs(docs):
    return "\n".join(d.page_content for d in docs)

rag_chain = (
    RunnablePassthrough.assign(context=lambda x: format_docs(retriever.invoke(x["question"])))
    | prompt
    | model
    | parser
)

print(rag_chain.invoke({"question": "How long until I get my refund?"}))

Genuinely the same job as Version 1, but composed from real, reusable Runnables — the direct benefit of everything from Module 8 onward.

Version 3: structured responses

from pydantic import BaseModel
from langchain.chat_models import init_chat_model
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough

class RAGAnswer(BaseModel):
    answer: str
    confidence: str  # "high", "medium", "low"

structured_model = init_chat_model("openai:gpt-4o-mini").with_structured_output(RAGAnswer)

structured_prompt = ChatPromptTemplate.from_template(
    "Using only this context, answer the question and rate your confidence:\n{context}\n\nQuestion: {question}"
)

structured_rag_chain = (
    RunnablePassthrough.assign(context=lambda x: format_docs(retriever.invoke(x["question"])))
    | structured_prompt
    | structured_model
)

result = structured_rag_chain.invoke({"question": "How long until I get my refund?"})
print(result)

Recall Module 18 — the final answer is now a genuine, typed object, not just a string.

Version 4: real source citations

A trustworthy RAG system tells you where its answer came from, not just what the answer is.

def ask_with_citations(question: str) -> dict:
    results = retriever.invoke(question)
    context = format_docs(results)
    prompt_text = f"Answer using only this context:\n{context}\n\nQuestion: {question}"
    answer = model.invoke(prompt_text).content
    sources = list({d.metadata["source"] for d in results})
    return {"answer": answer, "sources": sources}

result = ask_with_citations("How long until I get my refund?")
print(result["answer"])
print("Sources:", result["sources"])

Notice d.metadata["source"] — this is Module 22’s document metadata finally paying off directly: a real, honest answer to “where did this come from,” rather than asking a user to simply trust an unverified claim.

Version 5: conversation-aware RAG

Real usage rarely stops at one question. Recall MessagesPlaceholder from Module 7.

from langchain.messages import HumanMessage, AIMessage
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

conversational_prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer using only the provided context. Context:\n{context}"),
    MessagesPlaceholder("history"),
    ("human", "{question}"),
])

conversation_history = []

def ask_conversational(question: str) -> str:
    results = retriever.invoke(question)
    context = format_docs(results)
    formatted = conversational_prompt.invoke({
        "context": context, "history": conversation_history, "question": question,
    })
    answer = model.invoke(formatted).content
    conversation_history.append(HumanMessage(content=question))
    conversation_history.append(AIMessage(content=answer))
    return answer

print(ask_conversational("How long until I get my refund?"))
print(ask_conversational("And what about shipping?"))

Each turn retrieves fresh, relevant context and remembers what was already discussed — the retrieval pipeline from this entire phase, combined directly with Module 19’s conversation-history concepts.

Version 6: retrieval exposed as an agent tool

Recall Module 25’s real architectural choice.

from langchain.tools import tool
from langchain.agents import create_agent

@tool
def search_policies(query: str) -> str:
    """Search company policy documents."""
    results = retriever.invoke(query)
    return format_docs(results) if results else "No relevant policy found."

agent = create_agent(model=init_chat_model("openai:gpt-4o-mini"), tools=[search_policies])
result = agent.invoke({"messages": [{"role": "user", "content": "What's your refund timeline?"}]})
print(result["messages"][-1].content)

The complete RAG pipeline, now retrieved only when the agent genuinely decides it’s needed — the full arc of this entire module, from a fully manual Version 1 to a fully agentic Version 6.

Common mistakes worth avoiding

Letting the model answer from its own memory instead of the retrieved context. Every version in this module’s prompt says “answer using only this context” for a genuine reason — without that explicit instruction, a model will often blend retrieved content with what it already knew from training, quietly defeating the entire point of grounding an answer in your actual, current documents. Recall the Hallucination concerns from your earlier RAG course — this single instruction is a real, practical defense against exactly that risk.

Citing sources without actually verifying the citation is accurate. Recall Version 4 — d.metadata["source"] reports which document was retrieved, not necessarily which document the model’s specific answer actually drew from, especially when several documents were retrieved at once. For genuinely high-stakes applications, more rigorous citation-verification techniques exist, covered in your dedicated RAG evaluation coursework — the version here is a real, useful default, not a guaranteed-precise citation system.

Choosing agentic retrieval (Version 6) by default, without considering the added cost and latency. Recall Module 25’s honest trade-off — agentic retrieval costs one extra model decision on every request, even the ones that clearly need retrieval. If your application’s questions genuinely, reliably always need retrieval, Version 2’s simpler, fixed pipeline is often the better real choice, not a lesser one.

What you should take away from this module

• Real RAG systems grow in genuine, deliberate stages — manual, then composed, then structured, then cited, then conversational, then agentic — each addressing a real limitation of the version before it.
• Source citations are a direct, practical payoff of the metadata discipline established back in Module 22.
• Conversation-aware RAG combines this entire phase’s retrieval work with Module 19’s memory concepts directly.
• The choice between fixed-pipeline and agentic retrieval, from Module 25, applies to a complete, real system exactly as it did to a single tool.

Where this goes next

The next module shifts from building features to building resilience: Error Handling, Retries, and Fallbacks — what happens when a model call fails, a retriever times out, or a provider goes down, and how to keep an application working anyway.