When Things Fail: Retries and Fallbacks

Every example in this course so far has assumed the happy path — the API call succeeds, the model responds, the tool works. Real applications don’t get that guarantee. A provider’s servers have a brief outage. A rate limit gets hit during a traffic spike. This module builds genuine resilience against exactly these real, inevitable failures.

The real failures worth naming

• Timeouts — recall timeout from Module 4 — a request takes too long and gets abandoned.
• Rate limits — a provider temporarily refuses requests because you’ve sent too many, too fast.
• Invalid structured output — recall Module 18’s honest caveat that validation can genuinely fail.
• A tool exception — recall Module 17’s automatic catching, though as noted there, this is a safety net, not a cure.
• A provider outage — the rare, but real, case where an entire provider is briefly unavailable.

Example 1: automatic retries with .with_retry()

Recall max_retries from Module 4’s init_chat_model parameters — a blunt, model-wide setting. .with_retry() gives you the same idea, applied to any Runnable, with finer control.

from langchain.chat_models import init_chat_model

model = init_chat_model("openai:gpt-4o-mini")
resilient_model = model.with_retry(stop_after_attempt=3, wait_exponential_jitter=True)

response = resilient_model.invoke("What is the capital of Kenya?")
print(response.content)

stop_after_attempt=3 genuinely retries up to three times before finally giving up. wait_exponential_jitter=True waits progressively longer between each attempt, with a little randomness added — a real, deliberate technique preventing many failed requests from all retrying at the exact same moment and overwhelming an already-struggling server further.

Example 2: fallback to a backup model

Retries help with brief, transient failures. They don’t help if a provider is genuinely down for an extended period. For that, you need a real fallback.

from langchain.chat_models import init_chat_model

primary = init_chat_model("openai:gpt-4o-mini")
backup = init_chat_model("google_genai:gemini-2.0-flash")

resilient_model = primary.with_fallbacks([backup])

response = resilient_model.invoke("What is the capital of Kenya?")
print(response.content)

If primary fails, resilient_model automatically tries backup instead — genuinely the same real, practical value Module 5’s provider abstraction promised from the start: a working backup, ready the moment it’s actually needed, requiring no code change to activate.

Example 3: combining retries and fallbacks together

primary = init_chat_model("openai:gpt-4o-mini").with_retry(stop_after_attempt=2)
backup = init_chat_model("google_genai:gemini-2.0-flash").with_retry(stop_after_attempt=2)

resilient_model = primary.with_fallbacks([backup])

response = resilient_model.invoke("What is the capital of Kenya?")
print(response.content)

This is genuinely the realistic, layered approach: retry the primary a couple of times for brief hiccups, and only fall back to a genuinely different provider if the primary keeps failing even after retrying.

Example 4: applying resilience to a whole chain, not just a model

Because retries and fallbacks are Runnable methods, and chains are Runnables too, exactly per Module 8’s core insight, this works on entire pipelines, not only individual models.

from langchain.chat_models import init_chat_model
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_template("Summarize in one sentence: {text}")
model = init_chat_model("openai:gpt-4o-mini")
parser = StrOutputParser()

chain = (prompt | model | parser).with_retry(stop_after_attempt=3)

result = chain.invoke({"text": "LangChain provides reusable building blocks for LLM applications."})
print(result)

.with_retry() was applied to the entire prompt | model | parser chain at once — one line of genuine, real resilience covering the whole pipeline, rather than needing to wrap each individual step separately.

Common mistakes worth avoiding

Retrying indefinitely, or with no limit at all. A request that keeps failing for a genuine, non-transient reason — a permanently invalid API key, for instance — will simply keep failing no matter how many times you retry it. Always set a real, finite stop_after_attempt.

Using fallbacks as a substitute for actually fixing the root cause. A fallback masks a failure gracefully; it doesn’t investigate why the primary failed in the first place. Recall this course’s own Observability module, coming up soon — genuinely knowing why your primary keeps failing matters, not just quietly routing around it forever.

Forgetting that a fallback provider may behave subtly differently. Recall Module 5’s honest warning about capability differences across providers. A fallback that silently activates during a real outage, using a provider with slightly different structured-output behavior, could produce results a user or downstream system doesn’t expect — worth testing your fallback path deliberately, not just trusting it blindly.

What you should take away from this module

• .with_retry() handles brief, transient failures automatically, with a genuine, deliberate limit and increasing wait time between attempts.
• .with_fallbacks() provides a real, working backup when a primary genuinely, persistently fails — not just a brief hiccup.
• Both work on any Runnable — a single model, or an entire chain — because of the same shared interface established back in Module 8.
• Resilience is a real, deliberate layering: retry first for brief issues, fall back only when retrying itself keeps failing.

Where this goes next

The next module covers Guardrails — practical, deliberate protection against a different category of risk entirely: not technical failure, but a model or user doing something it genuinely shouldn’t, building directly on the middleware concepts from Module 20.