TechByteByByte

The Checkpoint and Resume Pattern

Why a genuinely durable agent needs more than saving state — a precise four-guarantee framework distinguishing real durable execution from mere checkpointing, and a real, current statistic showing why this infrastructure dominates production agent codebases.

#AI Agents#Agent Design Patterns#Checkpoint and Resume#Agentic AI

What You Will Learn

  • What must be saved to resume correctly.
  • How checkpoints differ from logs and memory.
  • How side-effect records prevent duplicates.

How to read the evidence

The over 75%, 8% to 100%, up to 87%, and within 1.9% figures come from CRAB’s shell-intensive and code-repair workloads. The 1.6% / 98.4% Claude Code split is an estimate from reverse-engineering an extracted codebase, not an official Anthropic engineering measurement.

Module 23 covered what happens when a single call fails. This module is about a genuinely different failure: the process itself dying partway through a long-running task, and what it actually takes to resume correctly rather than starting over.

The architecture

Task
 ↓
Step 1
 ↓
Checkpoint
 ↓
Step 2
 ↓
CRASH
 ↓
Restart
 ↓
Load Checkpoint
 ↓
Continue Step 2

Clarifying distinction

This is worth knowing precisely before anything else, because it’s easy to conflate three genuinely different mechanisms. “A retry just calls the same step again after a failure, which does nothing for the steps already completed before the crash. A checkpoint saves state you can reload later, but something still has to notice the process died and decide to reload it. Neither is sufficient on its own.” (Durable Execution: The Missing Runtime Primitive for Agents, Diagrid)

Genuine durable execution is the missing piece: an execution model where every step is persisted as it completes, so a workflow can be detected as failed, its state recovered, and the run restarted from where it stopped — all three problems solved together, not separately.

Striking statistic for why this infrastructure matters so much

It’s worth knowing exactly how much of a real, current production agent’s codebase this kind of infrastructure actually represents. A 2026 design-space analysis of Claude Code found that only 1.6% of its codebase is AI decision logic — the other 98.4% is operational infrastructure: context management, tool routing, and recovery. (Durable Execution in LangGraph, Vadim’s blog)

Read this precisely. In one of the most widely-used real coding agents, the part that actually “thinks” is a genuinely small fraction of the total system. The overwhelming majority of the engineering is exactly what this module covers — making sure the thinking part can survive real-world failure without losing its place.

The four guarantees durable execution requires

This is worth knowing precisely, because “checkpointing” alone doesn’t automatically deliver all of them. Persistence: the workflow’s state survives a process crash, a pod restart, a region failover. Exactly-once execution of side-effects: a tool call that already ran won’t run again on replay, even after being resumed many times. Suspend and resume across arbitrary delays: the workflow can wait minutes, hours, or days for a signal — a human approval, a webhook — without holding a running thread the whole time.

Deterministic replay: the workflow can be re-executed from its journal and reach the same state, which is what lets the system genuinely recover and what lets an engineer time-travel a production agent’s history to debug it. (Durable AI Agents in 2026, Reactify Solutions)

The same source’s own direct warning worth taking seriously: “Checkpointers are not durable execution.” A system that saves state periodically but doesn’t guarantee all four properties above is genuinely useful, but it isn’t the full pattern this module describes.

Concrete illustration of the cost of getting this wrong

It’s worth seeing this at the level of a genuinely relatable scenario. A 12-step agent workflow crashes at step 8. Steps 1 through 7 already consumed real model calls, wrote real database records, and called real external APIs. Without genuine durable execution, the only recovery option is to start over from step 1 — re-running every prior model call and re-executing every side effect, including the ones that already succeeded. (How Async AI Agent Workflows Survive Failures, Augment Code)

This is worth connecting directly to Module 21’s idempotency discipline — re-executing a side effect that already succeeded is precisely the duplicate-write problem that module already covered, now shown as a genuine consequence of poor recovery design, not just poor event handling.

Measured finding: naive checkpointing is mostly waste

This is worth knowing precisely, because it’s a genuine correction to the assumption that “checkpoint everything, as often as possible” is automatically the safe default. The 2026 Crab checkpoint/restore study found that over 75% of agent turns produce no recovery-relevant state at all — meaning blanket checkpointing after every single step is, for the large majority of steps, pure waste.

A semantics-aware approach, checkpointing selectively based on which steps genuinely matter for recovery, raised recovery correctness from 8% to 100%, while cutting the overhead blanket checkpointing would have cost. (Vadim’s blog)

This is worth reading as a genuinely important nuance: more checkpointing isn’t automatically better checkpointing. The real engineering question isn’t “should I checkpoint,” it’s “which specific steps, if lost, would actually prevent a correct resume” — and checkpointing precisely those, not everything indiscriminately.

Concrete trade-off worth knowing: sync versus async persistence

It’s worth knowing the actual, named choice a real, current framework offers. LangGraph’s persistence layer supports two modes: ‘async’ — persists changes while the next step executes concurrently, with “a small risk that LangGraph does not write checkpoints if the process crashes” — and ‘sync’ — writes every checkpoint before continuing, at “the cost of some performance overhead.” (Vadim’s blog)

The honest, real guidance worth remembering: for a genuinely long-running thread — a month-long agent session, say, where a missed checkpoint means a re-sent or skipped email — the stronger, synchronous durability is the correct trade, even at real performance cost.

Technical vocabulary worth knowing

It’s worth knowing the actual terms current runtime architecture uses to separate these concerns cleanly, drawn from Anthropic’s own real technical writing on scaling managed agents.

Five distinct runtime responsibilities: the harness drives the agent forward step by step; the session is an append-only log of everything that happened — model calls, tool calls, results, errors, approvals; the sandbox is where commands actually execute; the checkpoint is what the next worker reads on resume; the trace is what an engineer reads days later to understand what went wrong. (Long-Running AI Agent Runtime in 2026, Edge of Context)

A genuinely important distinction worth holding onto, extending Module 23’s own theme: “Traces provide the shape of an execution journal, but observability is not the same thing as recovery. A trace can explain what happened; a durable journal must decide what may be replayed, skipped, compensated, or resumed.” (Zylos Research, Durable Execution for AI Agent Runtimes)

This is worth taking as a real, recurring theme across this entire course’s closing modules: Module 23 distinguished genuine spend enforcement from mere spend monitoring; this module distinguishes genuine recovery from mere execution tracing. In both cases, the honest lesson is the same — having visibility into what happened is real and valuable, but it’s a genuinely different, weaker guarantee than having a system that actively does something correct in response, automatically, without a human needing to notice first.

What this looks like in code

Before reading the syntax, follow the execution flow: identify the incoming state, the component making the decision, the function doing the work, and the condition that returns a result or stops the loop. The code is a small teaching model of the pattern, not hidden framework magic.

import asyncio
from temporalio import workflow

@workflow.defn
class ResearchAgentWorkflow:
    @workflow.run
    async def run(self, query: str) -> str:
        sources = await workflow.execute_activity(search_sources, query)
        analysis = await workflow.execute_activity(analyze_sources, sources)
        return await workflow.execute_activity(synthesize_report, analysis)

This is genuinely the entire pattern in a real, named framework: each execute_activity call is a durable checkpoint boundary. “If the worker crashes during LLM call #2, it resumes from exactly that point once a new worker picks up the task.” (Building Durable AI Agents with Temporal, NiteAgent) Killing the worker process mid-execution and restarting it resumes from the last completed activity — no data loss, no duplicate LLM calls, no re-running of steps that already succeeded.

Applying this to a concrete scenario

It’s worth running this module’s real distinctions against your Multi-Agent Systems coursework’s recurring legal-contract pipeline, since it clarifies exactly where genuine durable execution earns its place there.

A single contract review completing in minutes genuinely doesn’t need the full four-guarantee treatment this module described — a crash mid-review is rare enough, and the cost of simply re-running the whole thing is low enough, that basic retry logic from Module 23 alone is a reasonable, proportionate choice. The real case for this module’s discipline shows up at the firm’s actual production scale: reviewing 500 contracts from a portfolio acquisition, a job that might genuinely run for hours, makes a mid-run crash a real, likely event rather than a rare edge case.

Losing all progress on contract 347 of 500 because contracts 1 through 346 have to be re-processed from scratch is exactly the 12-step-workflow-crashes-at-step-8 scenario this module described, scaled up. Applying the Crab study’s own finding directly: not every one of those 500 contract reviews needs a full checkpoint — the semantics-aware lesson suggests checkpointing specifically after each contract’s Critic-approved final result, the genuinely recovery-relevant state, rather than after every intermediate Executor step within each individual review.

Interview-relevant framing

Q: What’s the actual difference between a checkpoint and genuine durable execution?

Ans: A checkpoint alone just saves state you could reload later — something still has to notice the process died and actually decide to reload it. Genuine durable execution solves detection, recovery, and restart together as one guaranteed property, not three separate things you have to wire up yourself. A real production statistic makes the scale of this concrete: an analysis of Claude Code found only 1.6% of its codebase is actual AI decision logic — the other 98.4% is exactly this kind of operational infrastructure.

Q: Should you checkpoint after every single step in an agent workflow?

Ans: No, and real research found this is actually counterproductive — a 2026 study found over 75% of agent turns produce no recovery-relevant state at all, meaning blanket checkpointing is mostly wasted overhead. A semantics-aware approach, checkpointing specifically the steps that genuinely matter for a correct resume, raised recovery correctness from 8% to 100% in that same study. The real question isn’t how often to checkpoint — it’s which specific steps would actually prevent correct recovery if lost.

Q: Why isn’t tracing or observability the same thing as genuine recovery?

Ans: Because a trace only explains what already happened — it’s a record for a human to read afterward. A durable journal has to make an active decision about what happens next: what can be replayed safely, what needs to be skipped because it already succeeded, what needs compensating because it partially completed, and what should resume from exactly where it stopped. Observability answers ‘what went wrong.’ Durable execution has to answer ‘what do we do about it, automatically, right now.’

Common Misconception

Incorrect idea: Saving the latest conversation is enough to resume.

Why it is incorrect: Correct recovery may need plan position, tool results, approvals, versions, idempotency keys, and completed external effects.

Key takeaways

  • Retries, checkpoints, and genuine durable execution are three distinct things — a retry ignores prior progress, a checkpoint alone still needs something to notice the crash and reload it, and durable execution solves detection, recovery, and restart together as one guaranteed property.
  • A real, current statistic makes the stakes concrete: an analysis of Claude Code found only 1.6% of its codebase is actual AI decision logic, with the remaining 98.4% being operational infrastructure — much of it exactly this kind of recovery machinery.
  • Genuine durable execution requires four real guarantees together: persistence across crashes, exactly-once side-effect execution, suspend-and-resume across arbitrary delays, and deterministic replay — “checkpointers are not durable execution” on their own.
  • A real study found blanket, every-step checkpointing mostly wasteful — over 75% of agent turns produce no recovery-relevant state — while a semantics-aware, selective approach raised recovery correctness from 8% to 100%.
  • Real frameworks offer a genuine sync-versus-async persistence trade-off; the honest guidance is that long-running, high-stakes threads warrant the stronger, synchronous guarantee despite its real performance cost.
  • Real runtime architecture separates five distinct responsibilities — harness, session, sandbox, checkpoint, trace — a vocabulary worth using precisely rather than treating “the agent’s state” as one undifferentiated concept.
  • Observability and genuine recovery are different things: a trace explains what happened after the fact; a durable journal has to actively decide what gets replayed, skipped, compensated, or resumed, extending the same observability-versus-enforcement theme from Module 23.

Module 25 covers the discipline that keeps every pattern in this course from running forever when none of these recovery mechanisms resolve the underlying problem: The Bounded Agent Loop.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed