./maidul
Back to Blog
Agentic AIDurable ExecutionLangGraphTemporalDBOSRestateDaprReliabilityIdempotency

Durable Execution for Agentic Workflows: Why Your Agent Needs a Checkpoint, Not Just a Retry

Retries replay your agent from scratch and re-fire side effects. Checkpoints resume from the last completed step. Here is the real spectrum from retry to durable execution, and how to pick.

If you shipped an agent in 2024, your reliability story was probably a decorator. Wrap the tool call in @retry, set max_attempts=3, add exponential backoff, and move on. That works when a step is a stateless HTTP GET. It stops working the moment your agent has a fourteen-step plan, calls a payment API at step nine, and the pod gets rescheduled at step eleven.

A plain retry restarts the whole run. For an agent, “the whole run” means re-planning, re-calling the LLM, and re-firing every side effect that already succeeded. The customer gets charged twice, the ticket gets created twice, the email goes out twice. The retry did not make your agent reliable. It made it reliably wrong, faster.

This post is for engineers and architects putting agents into production paths where a silently failed or double-executed run costs money or corrupts data. I will walk the spectrum from retry to checkpoint to durable execution, show what each actually guarantees with real code, and then push back on the framing in my own title: a checkpoint is a real improvement over a retry, but a checkpoint you have to manage yourself is still not durable execution. Knowing that difference is the decision.

Table of contents

  • Why retries alone fail agents
  • Three points on the reliability spectrum
  • Checkpointing: what LangGraph actually gives you
  • Durable execution: what the runtime takes over
  • The catch nobody mentions: determinism and at-least-once
  • Comparison: where each tool sits
  • A decision framework
  • What to actually build

Why retries alone fail agents

A retry has one mental model: the unit of work is a pure function, calling it again is free, and the only thing that can go wrong is a transient error. Agent steps violate all three assumptions.

Consider the shape almost every agent has under the hood - a loop that plans, acts, and observes:

# The naive version. Every retry re-runs everything above the failure.
def run_agent(goal):
    plan = llm.plan(goal)                      # expensive, non-deterministic
    inventory = check_inventory(plan.item)     # external read
    charge = payments.charge(plan.card, plan.amount)  # side effect, NOT idempotent
    ticket = crm.create_ticket(plan.summary)   # side effect
    return llm.summarize([inventory, charge, ticket])

# Reliability strategy: hope.
result = retry(run_agent, goal, max_attempts=3)

If crm.create_ticket times out, the retry re-enters run_agent from the top. llm.plan runs again and may return a different plan, because the model is non-deterministic. payments.charge runs again against a card that was already charged. The retry has no memory of what already happened, so it cannot skip completed work, and it has no idempotency, so repeating that work is not safe.

The two failure modes are worth naming precisely, because the rest of the post is about closing them:

  1. Lost progress. Work that already succeeded is thrown away and redone. For a fourteen-step agent that spends real tokens per step, this is pure waste and latency.
  2. Unsafe repetition. Side effects that are not idempotent fire more than once. This is the one that pages you at 2am.

A retry addresses neither. It is a control-flow primitive pretending to be a reliability primitive.

Three points on the reliability spectrum

It helps to stop treating “reliability” as one thing. There are three distinct guarantees, and most teams conflate them.

Retry re-invokes a unit of work from the beginning. Guarantee: a transient failure gets another attempt. No memory, no resumption, no safety.

Checkpoint saves the state of the workflow after each step to a durable store, so a later run can resume from the last saved point instead of the start. Guarantee: completed steps are not redone - if something detects the failure and triggers the resume with the right identity.

Durable execution is checkpointing plus an execution runtime that owns the workflow lifecycle: it detects failure, reactivates the workflow, replays completed steps from a log, and drives the run to completion without you writing recovery code. Guarantee: the workflow runs to completion, exactly-progressed, across process and node crashes.

The jump from retry to checkpoint is the one my title is about, and it is real. The jump from checkpoint to durable execution is the one most “we added checkpointing” posts skip, and it is where production reliability actually lives.

Checkpointing: what LangGraph actually gives you

LangGraph is the cleanest mainstream example of checkpointing, so it is worth being precise about what it does. When you compile a graph with a checkpointer, the runtime saves a snapshot of graph state at every superstep, scoped to a thread_id. To recover, you re-invoke with the same thread_id and None as input, and the graph resumes from the node where it stopped (LangChain durable execution docs).

from langgraph.checkpoint.postgres import PostgresSaver

graph = builder.compile(checkpointer=PostgresSaver(conn))
config = {"configurable": {"thread_id": "run-123"}}

try:
    result = graph.invoke({"goal": "process order 55"}, config, durability="sync")
except Exception:
    # A later process resumes from the last checkpoint - same thread_id, None input.
    result = graph.invoke(None, config)

Two details matter more than the happy path.

First, durability is a tunable, not a default guarantee. LangGraph exposes three modes: "exit" persists only when the graph finishes, "async" persists in the background while the next step runs, and "sync" persists before the next step starts (docs). Only "sync" survives a mid-execution process crash without a gap, and it is the slowest. The default trades some crash-safety for speed, which is fine for a chat thread and not fine for an order pipeline. Pick deliberately.

Second, and this is the part that determines whether checkpointing is enough: on resume, LangGraph does not restart from the exact line where it stopped. It restarts from the beginning of the node and replays. Any side effect inside that node that already ran will run again unless you wrapped it in a task whose result was recorded, or made it idempotent (docs). The framework says this directly: design your workflow to be deterministic and idempotent. The checkpointer saves state; it does not make your side effects safe to repeat.

Here is the harder truth, and it is not a knock on LangGraph specifically. The open-source checkpointer gives you a save point, not a supervisor. Yaron Schneider, CTO of Diagrid, made this argument sharply in February 2026: LangGraph, CrewAI, and Google ADK all ship checkpointing or resumability, but none of them detect that a process crashed, none automatically re-invoke the failed run, and none prevent two workers from resuming the same thread_id at once and doing the work twice (Diagrid: “Checkpoints Are Not Durable Execution”). Schneider works for a vendor selling the alternative, so weigh the framing accordingly, but the technical claim checks out against LangGraph’s own docs: the open-source library runs in a single process, and failure detection and re-invocation are the caller’s job.

LangGraph has been adding runtime hooks that narrow the gap. Graceful shutdown (requires langgraph>=1.2, currently in alpha) lets a SIGTERM handler call request_drain() so an in-flight run stops at the next superstep boundary and saves a resumable checkpoint instead of dying mid-step (docs). That is genuinely useful for planned pod eviction. It still does not answer the question “who notices and resumes the run after an unplanned crash?” That question is what durable execution runtimes exist to answer.

Durable execution: what the runtime takes over

The defining move of durable execution is that the runtime, not your code, owns recovery. You write the workflow as straight-line business logic. The runtime records each completed step in a log and, on any crash, replays the workflow function from the start while returning the stored results of already-completed steps instead of re-running them. Execution converges on where it left off, with local variables restored, and then continues.

DBOS is the most compact way to see the model, because it is a library rather than a service. You add a Postgres connection string, decorate your workflow and its steps, and if the process dies mid-run it resumes from the last completed step on restart (DBOS Transact docs, dbos-transact-py).

from dbos import DBOS

@DBOS.step()
def charge(card, amount):
    return payments.charge(card, amount)   # runs once; result is checkpointed

@DBOS.step()
def create_ticket(summary):
    return crm.create_ticket(summary)

@DBOS.workflow()
def run_agent(goal):
    plan = plan_step(goal)                  # a step wrapping the LLM call
    receipt = charge(plan.card, plan.amount)
    ticket = create_ticket(plan.summary)
    return summarize_step([receipt, ticket])
# Crash after charge() but before create_ticket()? On restart, the workflow
# replays, charge() returns its recorded result, and only create_ticket() runs.

The behavior to internalize: a completed step is not re-executed on replay - its recorded output is returned. A step that failed part way restarts from its own beginning. So the granularity of your steps is the granularity of your recovery. Wrap the payment call as its own step and a crash after it will not double-charge. Bury the payment inside a larger step alongside three other calls and you are back to the retry problem within that step.

Temporal is the mature, service-based version of the same idea, aimed at scale rather than minimal footprint. It records an event history per workflow, and on worker restart it replays the workflow code from the beginning, skipping activities that already completed and feeding back their recorded results (Temporal Workflow docs). At its Replay 2026 conference, Temporal leaned directly into the agent use case, announcing Serverless Workers, Standalone Activities, Workflow Streams, and first-party integrations with the Google ADK and OpenAI Agents SDK (The New Stack). There is also a documented pattern for durable agents on top of the Vercel AI SDK (Temporal blog).

Restate takes a third shape: application code records completed operations in a journal, and on recovery it replays the journal and skips work already done, with a lighter operational profile than a full Temporal cluster. Dapr Workflows applies the durable-execution model inside the Dapr runtime and underpins Dapr Agents, an agent framework with durable execution built in that Diagrid co-maintains with NVIDIA (Diagrid). The category is crowded now - Temporal, DBOS, Restate, Dapr, Inngest, Hatchet - and the differences that matter are operational footprint and integration story, not whether the core replay mechanism works. It works in all of them.

The important part for agent builders: you do not have to abandon your agent framework to get this. DBOS integrates with Pydantic AI, where wrapping an agent in DBOSAgent turns Agent.run into a durable workflow and model requests and MCP calls into steps automatically (Pydantic AI DBOS docs). Temporal ships ADK and OpenAI Agents SDK integrations. The pattern is to keep the framework for authoring the agent and put a durable-execution runtime underneath it for the lifecycle.

The catch nobody mentions: determinism and at-least-once

Durable execution is not free, and the cost is not the infrastructure. It is a constraint on how you write code, and the LLM sits right in its blast radius.

Replay only works if the workflow function is deterministic. Given the same recorded history, it must issue the same steps in the same order, so the runtime can match recorded results to the calls that produced them. Anything non-deterministic in the workflow body - a random number, a system clock read, and yes, a direct LLM call - breaks replay, because the second time through it produces a different value than the log expects. Temporal is explicit that workflows must be deterministic (Temporal docs); LangGraph says the same for its replay model (LangChain docs). The fix is mechanical but non-negotiable: the non-deterministic work goes inside a step or activity, whose result is recorded, so replay reads the recorded output instead of re-rolling the dice. This is exactly why the LLM call belongs in a step, not in the workflow body.

Then there is the guarantee people misread. Durable execution runtimes give you at-least-once execution of a step, not exactly-once. If a worker crashes after a step’s side effect completed but before the runtime recorded that completion, the step runs again on recovery (Temporal community discussion on determinism and history). Determinism is not idempotency. The runtime guarantees your workflow reaches the end; it does not guarantee a given external side effect fired exactly once.

Which lands us back where the retry discussion started, one level up: your side-effecting steps still need to be idempotent. Pass an idempotency key to the payment API. Upsert the ticket on a natural key instead of blind-inserting. Check-then-act inside the step. Durable execution shrinks the window and removes the “lost progress” failure mode entirely, but the “unsafe repetition” failure mode is only fully closed when the step itself is safe to repeat. Any post that sells durable execution as “you never think about idempotency again” is selling you the demo, not the pager rotation.

Comparison: where each tool sits

PropertyPlain retryFramework checkpoint (LangGraph)Durable execution runtime (Temporal / DBOS / Restate / Dapr)
Skips completed work on recoveryNoYes, per node/stepYes, per step/activity
Detects a crash automaticallyNoNo - caller’s jobYes - runtime owns it
Re-invokes the failed run automaticallyNoNoYes
Prevents two workers doing the same runNoNo built-in coordinationYes
Runs across processes / nodesNoSingle process (open source)Yes, distributed
Requires deterministic workflow bodyNoYes, for replayYes
Side effects still need idempotencyYesYesYes (at-least-once)
Operational cost~ZeroLow (a checkpoint store)Library (DBOS) to full service (Temporal)

The checkpoint column is a real, large step up from retry: it kills lost progress within a single process and it is the right tool for human-in-the-loop pause and resume. The durable-execution column is what you need when a crash must not require a human to notice.

A decision framework

Match the guarantee to what a failed run actually costs.

  • If a failed run just means the user re-asks and nothing external changed - a research or Q&A agent with no writes - a plain retry plus a checkpointer for conversation state is enough. Do not add a workflow engine to a chatbot.
  • If your agent pauses for human approval and resumes later, possibly on another process, and the steps are mostly reads - framework checkpointing (LangGraph with durability="sync" and a Postgres checkpointer) is the right fit. You get pause/resume and progress-skipping without new infrastructure.
  • If a silently failed or double-executed run means lost money, corrupted state, or a broken SLA, and you cannot rely on a human to notice and re-run it - you need a durable execution runtime. Reach for DBOS when you want the minimum operational footprint and already run Postgres; reach for Temporal when you need proven distributed scale and first-party agent-SDK integrations; look at Restate or Dapr when their operational model fits your stack better.
  • Regardless of the tier: make every side-effecting step idempotent. This is the one control that pays off at every level of the spectrum, and the only one that closes the failure mode a retry actually caused.

What to actually build

The title frames the choice as retry versus checkpoint, and for getting from a demo to something that does not throw away work, that is the right first move. But the honest version of the argument goes one step further. A checkpoint is a save point you are still responsible for using. The question that separates a resilient agent from a fragile one is not “do I save state?” It is “when this process dies at 3am, who notices, who resumes the run, and who guarantees the payment step did not fire twice?” If the answer to any of those is “a human, eventually,” you have checkpointing, not durable execution.

So build in this order. Make your side effects idempotent first, because that is the fix for the bug a bare retry introduces and it is required at every tier above. Add a checkpointer next, so you stop discarding completed work. Then, only for the workflows where a missed failure has a dollar figure attached, put a durable execution runtime underneath your agent so the runtime, not your on-call engineer, owns getting the run to completion. Most agents need the first two. The ones in your critical path need all three.

Sources

Primary docs

Repositories

Vendor and conference posts

Independent analysis and discussion

Related reading on this blog

Discussion

Reader comments

Loading comments…

0 / 1000

No account needed. Your comment will appear immediately.

    More articles