Testing Prompts, Chains, and Agents

Recall Module 29’s closing point — a trace shows you what already went wrong. Testing exists to catch problems before a real user ever triggers them. This module writes real, runnable tests for the components you’ve built throughout this entire course, using pytest, the same testing tool you already know from your Python foundations.

Example 1: testing prompt formatting

Recall Module 7 — a prompt template is deterministic and doesn’t require calling a real model to verify it’s correct.

from langchain_core.prompts import ChatPromptTemplate

def test_tutoring_prompt_fills_variables_correctly():
    prompt = ChatPromptTemplate.from_messages([
        ("system", "You are a patient tutor."),
        ("human", "Explain {topic} to a {level} learner."),
    ])
    formatted = prompt.invoke({"topic": "RAG", "level": "beginner"})
    human_message = formatted.messages[-1].content
    assert "RAG" in human_message
    assert "beginner" in human_message

This test costs nothing to run — no API call, no real model — and genuinely catches a real, common mistake: a typo in a variable name, or a template that silently doesn’t include a value you expected it to.

Example 2: testing a tool directly

Recall Module 12 — a tool is just a well-described Python function. Test it exactly like one.

from langchain.tools import tool

@tool
def check_order_status(order_id: str) -> str:
    """Look up the current status of a customer order."""
    orders = {"1001": "Shipped"}
    return orders.get(order_id, f"No order found with ID {order_id}.")

def test_check_order_status_found():
    result = check_order_status.invoke({"order_id": "1001"})
    assert result == "Order 1001 status: Shipped" or "Shipped" in result

def test_check_order_status_not_found():
    result = check_order_status.invoke({"order_id": "9999"})
    assert "No order found" in result

Recall Module 12’s own emphasis on handling failure gracefully — this second test specifically verifies that graceful handling actually works, rather than just assuming it does.

Example 3: mocking a model, to test logic without real API calls

Real model calls cost real money and introduce real, non-deterministic output — genuinely undesirable in a test suite that needs to run quickly and reliably.

from unittest.mock import MagicMock
from langchain.messages import AIMessage

def test_summarization_logic_with_mocked_model():
    mock_model = MagicMock()
    mock_model.invoke.return_value = AIMessage(content="A short summary.")

    def summarize(model, text):
        return model.invoke(f"Summarize: {text}").content

    result = summarize(mock_model, "A long article about LangChain.")
    assert result == "A short summary."
    mock_model.invoke.assert_called_once()

MagicMock stands in for a real model, returning a fixed, predictable AIMessage instead of making a genuine API call. This tests your own logic — did you call the model correctly, did you handle its response correctly — completely independent of whether a real model would produce a good summary, which is a genuinely different question, covered next.

Example 4: testing structured output validation

Recall Module 18’s honest point that structured output can genuinely fail validation. Test that your schema actually enforces what you think it does.

import pytest
from pydantic import BaseModel, Field, ValidationError

class Rating(BaseModel):
    stars: int = Field(ge=1, le=5)

def test_rating_rejects_out_of_range_values():
    with pytest.raises(ValidationError):
        Rating(stars=10)

def test_rating_accepts_valid_values():
    rating = Rating(stars=4)
    assert rating.stars == 4

This test doesn’t call a model at all — it verifies your schema itself behaves correctly, independent of whether a model reliably produces valid values, which is a separate, real concern worth testing with actual model calls occasionally, not on every single test run.

Example 5: an integration test for a real chain

Some tests genuinely need to call a real model — verifying your prompt actually produces the behavior you intend, not just that your code is wired together correctly.

import pytest
from langchain.chat_models import init_chat_model
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

@pytest.mark.integration
def test_classification_chain_identifies_positive_sentiment():
    prompt = ChatPromptTemplate.from_messages([
        ("system", "Classify sentiment as exactly one word: positive, negative, or neutral."),
        ("human", "{text}"),
    ])
    chain = prompt | init_chat_model("openai:gpt-4o-mini") | StrOutputParser()

    result = chain.invoke({"text": "This is the best purchase I've made all year!"})
    assert "positive" in result.lower()

Notice @pytest.mark.integration — a real, deliberate convention for separating fast, free, mocked tests (Examples 1-4) from slower, real tests that genuinely call a model and cost real money. Real projects typically run the mocked tests constantly, and the integration tests less frequently — before a release, for instance.

Common mistakes worth avoiding

Running expensive integration tests on every single code change. Recall @pytest.mark.integration from Example 5 — every real model call costs real money and real time, echoing the Cost per Token concerns from earlier in your broader curriculum. Running the full integration suite on every save, rather than reserving it for genuine pre-release checks, is a real, avoidable, ongoing expense.

Testing only the happy path. Recall Module 12’s own emphasis on tools handling failure gracefully — a test suite that only ever checks check_order_status.invoke({"order_id": "1001"}) and never checks the “not found” case, exactly like Example 2’s second test, gives you real, false confidence about behavior nobody actually verified.

Treating a passing mocked test as proof the real prompt actually works well. Recall Example 3 — MagicMock verifies your code calls the model correctly; it says nothing about whether your actual prompt produces a genuinely good response from a real model. Both kinds of tests are necessary, and neither substitutes for the other.

What you should take away from this module

• Prompt formatting, tool logic, and schema validation can all be tested completely without calling a real model — fast, free, and deterministic.
• MagicMock lets you test your own application logic in isolation from a model’s actual, non-deterministic output.
• Genuine integration tests, calling a real model, verify behavior mocked tests can’t — but cost real money and time, so they’re run more deliberately, not on every single change.
• A real, mature test suite deliberately separates these two categories, rather than treating every test the same way.

Where this goes next

The next module steps back from hands-on code entirely for a genuinely important comparison: LangChain vs. calling a provider’s SDK directly — a balanced, honest look at when the abstraction you’ve spent this entire course learning is actually the right choice, and when it genuinely isn’t.