Agentic AI · Fault Tolerance

Without deliberate design,
multi-agent systems fail 41–87% of the time.

Agents rely on the SRE toolkit for reliability engineering, including retries, circuit breakers, fallback chains, error budgets, and chaos testing. However, each concept requires significant adaptation to be effective. When an agent loses its data source, it must improvise, presenting a unique challenge for design.

See the recovery ladder → Learn how circuit breakers adapt
closed SLIDE 01 : AGENT RELIABILITY ENGINEERING Cover slide: Agent Reliability Engineering
Quick answer

Agent reliability engineering utilizes the SRE toolkit for autonomous AI agents, tailored for handling failures. semantic, not just operationalIn a traditional service, responses are given or clean errors are thrown; a failing agent may instead improvise, hallucinate missing data, or use the wrong tool confidently. The fundamental discipline remains the same. recovery ladderWhen faced with failures, it is essential to categorize them, implement retry strategies with backoff and jitter, activate a circuit breaker to prevent cascading failures, gracefully switch to an alternative agent or model, save state to avoid duplicate side effects during retries, and escalate to human intervention as a last resort. Research on production multi-agent systems reveals failure rates ranging from 41 to 87 percent without these practices in place, highlighting the importance of reliability being a function of the surrounding infrastructure rather than the agent itself.

failure mode SLIDE 02 : WHY AGENT FAILURES BREAK THE OLD PLAYBOOK Why AI agent failures differ from traditional microservice failures
Semantic failure, not just operational failure

A retry doesn't produce the same output twice

If a microservice call fails, retrying with the same input consistently yields the same output. However, when an LLM call fails during reasoning, retrying can lead to a completely different chain of thought, causing the agent to follow a different plan with different tools in a different sequence.

When an agent loses access to its main data source, it doesn't simply halt and display an error like a regular service would. Instead, it adapts by imagining missing data, using the wrong tool, or providing an incomplete response without indicating any issue. Detecting a 500 error is straightforward, but spotting a tool that silently returns outdated data is much more challenging.

The full response, rung by rung

Seven steps between a failure and a clean escalation

Self-healing agents effectively manage failures by categorizing and directing each one to the appropriate recovery action, preventing crashes.

recovery loop SLIDE 03 : THE RECOVERY LADDER The seven-rung recovery ladder for agent failure handling
Every failure, wrapped and routed

Classify first : everything else follows from that

All actions are enclosed in a try-catch loop, with failures categorized as transient, permanent, or critical to dictate the appropriate handling. While a rate limit and an authentication failure may appear similar, they require distinct responses based on their classification.

1ClassifyTransient, permanent, or critical : the classification decides everything downstream.
2Retry + backoffLimited retries with increasing time intervals and randomness for truly temporary failures.
3Circuit breakerInstead of persisting failures, fail quickly when a dependency is clearly unavailable.
4FallbackDirect to a different agent, model, or stored response in a specified sequence.
5Checkpoint & resumeMaintain state to ensure a clean retry without duplicating side effects.
6EscalateDirect the query to a human with all necessary information once automated solutions have been exhausted.
7LearnIncorporate each failure into the classification rules and evaluation suite.
retry logic SLIDE 04 : RETRY STRATEGIES THAT DON'T MAKE THINGS WORSE Retry strategies for AI agents: backoff, jitter, and knowing what's actually retryable
Throttling isn't the same as failing

Know which errors deserve another attempt

An agent hitting rate limits occasionally is not considered a failure; it is simply being throttled. A well-designed retry policy with exponential backoff and jitter can manage this situation effectively. On the other hand, an agent experiencing authentication failures with every call is a genuine failure. Continuously retrying in such cases not only wastes resources but also delays the detection of the issue.

LLM API calls fail independently around one to five percent of the time during production, even in regular conditions. This necessitates a retry policy that can differentiate between the average failure rate and a prolonged outage to avoid either retrying excessively or abandoning too soon.

The three-state machine

Circuit breakers, adapted for what agents actually do wrong

In a conventional setup, a circuit breaker is in place to safeguard call volume. The true danger for an agent lies not in the number of calls, but in how they respond when a call is unsuccessful.

half-open SLIDE 05 : THE CIRCUIT BREAKER STATE MACHINE Circuit breaker three-state machine: closed, open, half-open, adapted for agent failures
Closed, open, half-open

A proactive cutoff neither retries nor fallbacks can provide

A circuit breaker encases a call path within a compact state machine. closed allow calls to flow smoothly; if failures exceed a certain limit, it will trigger an alarm. open and quickly gives up before attempting; then after a break, it progresses to half-open A single probe is used to test if the dependency has recovered. If the probe succeeds, it closes; if it fails, it re-opens and waits again.

The key adaptation that matters is agent-specific: the breaker must safeguard against the agent's response to a failed call, not just the volume of calls : otherwise, an agent could bypass the breaker and cause unforeseen damage.

fallback route SLIDE 06 : FALLBACK CHAINS & DEAD LETTER QUEUES Fallback chains and dead letter queues for AI agent reliability
Nothing gets silently dropped

An ordered chain, and a record when it runs out

If the primary agent's circuit opens, a designated fallback chain decides the next destination for traffic : whether it be an alternate agent, a scaled-down model, or a cached response. Every fallback must be verified for behavioral consistency: directing traffic to a modified output format in a smaller model can introduce a more nuanced bug compared to the initial failure.

When all options have been tried and failed - the main circuit is open, all backups have been exhausted, and the retry limit has been reached - the request is sent to a. dead letter queueThis is not just any bin for throwing away information; it is a resilient storage system that securely holds the initial request, complete failure record, and timestamp. This allows operators to pinpoint the cause of the failure and rerun it once the circumstances are better. The most detrimental failure scenario in a production agent system is when data is lost without anyone realizing, leading to repercussions for the business at a later time.

state safety SLIDE 07 : IDEMPOTENCY & STATE MANAGEMENT Idempotency keys and state management to prevent duplicate side effects
A retry that doesn't send the payment twice

"Idempotent retry" doesn't mean the same thing for agents

In a traditional setup, repeating a task with the same input will consistently yield the same result, ensuring a safe retry. However, for agents, this assumption no longer holds true. A failed LLM call retry may generate a completely different chain of reasoning, necessitating safety measures from an idempotency key linked to the specific side effect rather than the reasoning process.

Creating a checkpoint before a risky action allows for a clean resume of a retry, preventing the need to repeat the action. Without this precaution, a failed payment confirmation could result in double charging a customer, despite the reasoning behind it appearing reasonable each time.

Testing resilience on purpose

Many agents who appear tough on paper end up failing ungracefully in practice.

Intentionally introducing failures is the most effective method to verify that graceful degradation functions properly, rather than simply appearing to do so.

fault injection SLIDE 08 : CHAOS ENGINEERING FOR AGENTS Chaos engineering for AI agents: deliberate failure injection with abort guards
A hypothesis and an abort guard, every time

Not a demo : a measurable experiment

Numerous teams conduct chaos testing as a one-off showcase: introducing a solitary fault, monitoring a dashboard, and deeming it resilient. However, autonomous agent systems demand greater rigor as policy dependencies can deteriorate in a secure or insecure manner, and replay paths may either recover smoothly or replicate side effects, all contingent on nuances that a singular demo fails to reveal.

In a chaos experiment, a hypothesis is proposed: if a tool returns a 403 error, the agent should log the failure, attempt no more than two retries, inform the user of the unavailability of the source, and proceed with existing context; it must not loop endlessly or create imaginary content. A strict abort guard is in place to ensure reality does not deviate significantly from this expectation.

DisciplineWithout itWith it
Fault tolerance design41–87% failure rateClassified, routed, contained
Circuit breakersCascading dependency failureFails fast, isolates the blast radius
Idempotency keysDuplicate side effects on retrySafe, resumable retries
Chaos testingResilience assumed, untestedResilience measured against a hypothesis
reliability targets SLIDE 09 : SLOs FOR AGENT BEHAVIOR, NOT JUST UPTIME SLOs and error budgets that cover agent decision quality, not just uptime and latency
Reliability means more than "it responded"

Decision quality is a reliability metric too

In a traditional service, an SLO monitors availability and latency, while an error budget dictates the allowable threshold before intervention is necessary. However, for an agent, reliability must also encompass decision accuracy, task completion consistency, cost limitations, and the crucial skill of knowing when to seek human input rather than relying on uncertain guesses.

This does not serve as a replacement for a traditional observability stack; instead, it integrates agent governance seamlessly within it, utilizing familiar operational terms like burn rates, error budgets, and health checks that SRE teams are already familiar with, rather than creating a new system from scratch.

final state SLIDE 10 : RELIABILITY IS INFRASTRUCTURE, NOT A MODEL PROPERTY Summary: reliability is a property of the infrastructure around an agent, not the model itself
The mindset that survives production

Design to expect failure, not to be surprised by it

Operating agents in a production environment without circuit breakers, health checks, and an on-call runbook is like managing a fleet of microservices without proper infrastructure - it's not acceptable for regular systems and it shouldn't be acceptable for agent-based systems either. The reliability of an autonomous agent doesn't solely depend on its model, but on the governance and fault-tolerance infrastructure that surrounds it.

The underlying philosophy of each pattern is consistent: anticipate failure, minimize its impact, and maintain essential functionality in adverse conditions, instead of viewing failure as a fatal error like older agent systems.

Frequently asked

Agent reliability engineering FAQ

Retrying a failed LLM call may not result in the same output as retrying a microservice call, as it can lead to a completely different sequence of actions, with the agent utilizing different tools in a different sequence. Agents' retry logic must consider this unpredictability, not just the eventual success of the call.

Identify the type of failure (transient, permanent, or critical), implement retry with backoff and random intervals, activate a circuit breaker if the dependency appears to be offline, switch to an alternative agent or model, save state to avoid repeating actions, involve a human when automated recovery efforts fail, and analyze the failure to improve the system's detection and evaluation process.

A conventional circuit breaker primarily guards against high call volume to prevent service overload. However, for an agent, the greater concern lies in how they respond when a dependency fails. They may resort to improvisation, hallucinate missing data, or use the wrong tool instead of ceasing operations. An agent-aware circuit breaker must consider this behavioral risk, not just the number of failures.

This storage is designed to hold failed requests that have been retried and exhausted all fallback options, including the original request, detailed failure history, and timestamp. It allows operators to diagnose and potentially replay the request, preventing silent data loss in production agent systems.

This involves intentionally introducing failures - such as tool errors, increased latency, and simulated rate limits - based on a specific hypothesis of the agent's expected behavior. The experiment includes a strict abort mechanism if the results deviate significantly from the hypothesis. This method is essential for verifying the effectiveness of graceful degradation, as many supposedly resilient agents falter under real-world conditions.

Beyond just uptime and latency, reliability for an agent encompasses decision quality, task completion fidelity, cost-per-operation limits, and the ability to recognize when to escalate to a human instead of making a low-confidence decision. The concept of reliability for an agent goes beyond that of a typical microservice.

Get started

Ready to fault-tolerance-test your agent?

Construct a seven-step recovery ladder, incorporate idempotency keys into all state-changing actions, and conduct your initial chaos experiment based on a predetermined hypothesis before your upcoming release.

Revisit circuit breakers → Back to chaos testing