Production LangGraph agent architecture

An agent that works in a notebook and an agent that works on production traffic are separated by roughly the same distance as a proof-of-concept query and a database. The notebook version needs the happy path to work once. The production version inherits partial failures, model timeouts, malformed tool arguments, users who phrase things nobody anticipated, and a finance team that will ask why last month cost $14,000.

None of that is a prompt engineering problem. It is control flow, state management and budget enforcement — ordinary distributed systems work that agent frameworks make easy to skip.

The demo-to-production gap is a control-flow problem

The typical prototype is a while loop. Call the model, look at the response, call a tool if it asked for one, feed the result back, repeat until it stops. That loop is why prototypes are quick to build and why they fail in ways that are hard to diagnose.

The loop has no explicit states, so there is nothing to inspect when it misbehaves — only a transcript. It has no boundaries, so a model that keeps requesting the same tool will keep being given it. It has no persistence, so an exception at step nine discards the work of steps one through eight. And it has no budget, so the cost of any individual request is unbounded and unknown until it has already been incurred.

Every fix below is about replacing an implicit loop with explicit structure.

Model the agent as a state machine, not a conversation

The single highest-leverage decision is to represent the agent as a graph of named nodes with typed state passed between them, which is what LangGraph exists to provide. The value is not the library — it is that the control flow becomes a data structure you can reason about instead of an emergent property of a prompt.

Define the state as a typed schema, not a free-form dictionary. When state is a dict, every node reads and writes keys by string, nothing validates that a node produced what the next one needs, and a typo becomes a silent None three nodes later. A typed state object turns that into an error at the boundary where it happened.

Keep nodes small and single-purpose: one node classifies intent, one retrieves, one calls a specific tool, one composes the answer. Small nodes are individually testable without invoking the whole graph, which is the only way to build a meaningful test suite for an agent. A three-node graph where one node “handles everything” has the same debuggability as the while loop it replaced.

Edges carry the routing logic, and conditional edges should branch on values in state rather than on the model re-deciding. Every routing decision delegated to the model is a decision that can go differently on identical input.

Checkpointing is the difference between a retry and a restart

An agent run is a multi-step transaction over unreliable dependencies. Model APIs time out, rate-limit and return 500s. Tools call systems that have their own outages. If state lives in memory, any of those failures discards the entire run, and the user’s recovery path is to start over and pay for the tokens twice.

A durable checkpointer persists graph state after each node commits. Recovery becomes resumption from the last committed step rather than a restart, which changes both the cost and the latency of a failure by an order of magnitude on a long run.

Persistence buys three more things that are difficult to retrofit. Human-in-the-loop review becomes possible, because a graph that can pause and resume can wait for a person without holding a process open. Multi-turn conversations get real memory, since the thread’s state is on disk rather than in a variable. And post-hoc debugging becomes possible: you can load the exact state at the step where the run went wrong instead of trying to reproduce it from logs.

Use Postgres for the checkpointer if the rest of your system is already on Postgres. The temptation to add Redis for “fast” state is usually a false economy — checkpoint writes are not the bottleneck in a workflow whose steps take hundreds of milliseconds each, and a second stateful datastore is a new failure mode and a new sync problem.

Cost guardrails belong in the graph, not the invoice

An unbounded agent is an unbounded bill. The failure mode is not a gradual overspend; it is one pathological run that loops forty times against a frontier model, or one user who discovers that a particular phrasing sends the agent into an expensive retrieval cycle.

Enforce budget inside the run, not in a monthly report. Carry accumulated token count and estimated cost in the graph state, increment it after every model call, and check it in the conditional edge that decides whether to continue. When the ceiling is hit, route to a terminal node that returns the best answer available so far along with an explicit signal that the run was truncated. A truncated answer with a known cost is a manageable product decision. A $40 answer nobody predicted is not.

Set the ceiling per run, not per user or per day. Daily caps fail open for exactly as long as it takes to notice, and the pathological single run is the case that actually hurts.

Where the token spend actually is

Teams reach for a cheaper model first. It is usually the wrong lever, because the spend is rarely concentrated in the generation — it is in what gets sent.

The conversation history is normally the largest line item and grows quadratically: every turn resends every prior turn. Summarising older turns into a compact running state and keeping only recent turns verbatim removes most of that growth with very little quality cost, because the model almost never needs turn three verbatim by turn twenty.

Retrieved context is the second. Passing twenty chunks when the answer reliably sits in the top three is a straight multiple on input tokens for every call that follows in the run. Tune retrieval breadth against an evaluation set rather than setting it high for safety.

Tool schemas are the quiet one. Every tool definition is resent on every model call in the loop. A dozen verbose tool definitions with long descriptions can dominate the input on a short run. Expose only the tools a given node can actually use — which a state machine lets you do and a monolithic loop does not.

After all three, model routing becomes worth doing: classification and extraction steps run on a small model, only the synthesis step needs the frontier one. Applied in that order, we have seen agent workloads drop well over half their token spend with no measurable change in output quality.

Deterministic fallbacks

The question that separates production systems from prototypes is: what does this return when the model is unavailable?

Every node that calls a model needs a defined answer. Sometimes that is a retry with backoff, sometimes a cheaper model, sometimes a templated response, sometimes an escalation to a human queue. What it must never be is an unhandled exception surfacing as a 500, and it must never be silent — a fallback that quietly degrades answer quality without recording that it fired is worse than a visible failure, because it corrupts your quality metrics as well as the answer.

Make the fallback path a real node in the graph. Fallbacks buried in try/except inside a node are invisible to tracing, untested, and the first thing to break when the node is refactored.

The same applies to malformed tool arguments, which are common enough to design for rather than treat as exceptional. Validate arguments against the tool’s schema before execution and route validation failures to a repair node that feeds the error back for one bounded correction attempt. One retry, then fall back — an unbounded repair loop is the most expensive bug in this class.

Bounding the loop

Every cycle in the graph needs a hard recursion limit, and hitting it should be a logged, alertable event rather than a swallowed exception. In practice, a run that hits the limit is almost never a run that needed more steps — it is a run that was stuck, most often alternating between two tools because neither produced what the model expected.

Track the tool-call sequence in state and break on repetition. If the agent has called the same tool with the same arguments twice, a third identical call will not produce a different result. Detecting that directly is cheaper and clearer than waiting for a recursion ceiling several thousand tokens later.

Latency is a p99 problem

Agent latency is additive across steps, and the tail dominates the experience. A five-step agent where each step has a 95th-percentile latency of two seconds does not feel like a ten-second system; the compounding tail means a meaningful share of users wait considerably longer.

The structural fix is parallelism. Independent nodes — retrieving from two sources, calling two unrelated tools — should run concurrently rather than sequentially, which a graph makes explicit and a loop makes nearly impossible. Beyond that: stream tokens so time-to-first-token is decoupled from total completion time, cache aggressively at the retrieval layer where the same questions recur, and set per-node timeouts so one slow dependency degrades to a fallback instead of holding the whole run open.

Measure p95 and p99 per node, not just for the run. An average that looks fine almost always conceals one node with a heavy tail.

What to build first

If an agent is already in production without this, the order that recovers the most ground per unit of work is: durable checkpointing first, because it makes every subsequent failure diagnosable; then per-run cost ceilings, because it caps the worst-case downside; then explicit fallback nodes; then history and context trimming; then model routing.

Prompt tuning is last. It is the most visible lever and the least durable one — every improvement it buys is re-litigated on the next model version, while a bounded, checkpointed, budgeted graph keeps paying out regardless of what runs inside it.

Whether you need multiple agents at all is a separate question, and the answer is usually no — we covered when the complexity is justified in multi-agent systems with LangGraph, and the framework choice itself in LangGraph vs LangChain.

Tools referenced

LangGraph
LangGraph persistence and checkpointers
Pydantic — typed state and tool-argument validation
OpenTelemetry — span-level tracing across agent steps

EpochC builds AI agents and multi-agent orchestration that run on production traffic. See the clinical multi-agent case study — four specialised agents behind one conversational API — or book a technical discovery call and bring your current agent’s trace log. We will walk it step by step and tell you which of these five failure modes it has, before you commit to anything.

Production LangGraph agents: checkpointing, cost ceilings and deterministic fallbacks