Agent operations include post-launch activities such as SLO monitoring, on-call rotations, incident response, runbooks, and cost dashboards to ensure agents run smoothly at all times, even in the early hours of the morning.
Agent operations (AgentOps) Applying SRE principles to maintaining production AI agents involves setting service-level objectives, implementing on-call rotations, responding to incidents, creating runbooks, and managing costs and capacity for systems that have different failure patterns than traditional software. Anomaly detection is crucial due to the variability in agent outputs. On-call procedures are evolving with a tiered autonomy model allowing agents to handle routine incidents independently, while complex judgment calls are escalated to humans. The goal is for agents to handle the majority of on-call responsibilities, freeing up humans to focus on the critical twenty percent that requires human judgment.
Deploying an agent version into production is just the beginning. Operations ensures its ongoing health through defined severities, on-call rotations, incident response with runbooks, postmortems for learning, and proactive cost and capacity planning.
AgentOps was created to help organizations make intentional trade-offs between reliability and speed when implementing agent changes, as without service-level objectives, each change poses a risk with unpredictable outcomes.
Conventional SRE thresholds are based on the assumption that a metric remains consistent under normal conditions. However, agent outputs are intentionally diverse, even when functioning properly, making a fixed threshold either too sensitive or not sensitive enough. An alternative approach is statistical anomaly detection, where the system learns the typical behavior of an agent and only alerts when it deviates beyond preset boundaries, rather than reacting to individual abnormal outputs.
Composite SLOs bring together various signals to establish a unified reliability goal, as a single metric cannot accurately gauge the overall health of an agent fleet. Reliability, quality, speed, and cost each require their own target, integrated rather than monitored separately on different dashboards.
One faulty prompt can spread through numerous runs before anyone notices - the loop is in place to prevent it from escalating.
Revert a faulty prompt, but save the faulty version for analysis. Secure a model, but document the failure on the reverted version for a safe retry. Deleting evidence eliminates the opportunity to learn from incidents.
Statistical signals flag drift before it's visible to a user.
Stop the bleeding : quarantine the failing component.
Remove the cause reversibly, preserving evidence for later.
Unpause in measured tranches, verifying between each one.
Discover the underlying reason, not just the apparent symptom.
Traditional web monitoring tools are not equipped to detect agent failures, leading to potential cost increases of up to fifty times in just one hour, even when traditional error rates remain unchanged. A more comprehensive detection system is needed to monitor agent decision deviations, escalate issues to human operators, assess tool selection consistency in workflows, and identify outliers at the tenant level.
The existence of these signals relies on trace coverage at tool-call granularity and cost attribution per workflow, with the detection of the instrumentation layer depending entirely on this.
One agent may function with varying levels of autonomy depending on the task at hand – cache invalidation, service restarts, and database changes all warrant different levels of control.
Being on-call involves more than following a runbook; it requires identifying gaps in the runbook and making quick decisions under stress. Agents are not equipped to handle this aspect effectively, so a practical autonomy approach limits independent actions to a specific, pre-approved list of common, low-risk scenarios.
The agent scans logs to identify incidents that match a runbook signature and recommends a remediation, which is then approved by a human operator.
Agent suggests a precise action - 'undo the recent deployment?' - and displays the exact modifications. It waits for a direct approval from a human before taking any action.
The agent operates independently, expanding its scope cautiously as it proves the safety of restarting a crash-looped stateless service with a small set of pre-approved, low-risk patterns.
A static runbook page may be sufficient for a person reading at their leisure, but incidents cannot wait. To begin the migration process, start small by analyzing the last few months of incidents, identifying recurring patterns, and transforming the top three to five most frequent, low-risk incidents into organized steps that an agent can practice before implementation.
Integrate those plays into the incident tool initially as a recommendation for human execution, gradually progressing to one-click or automatic execution for the safest actions once the data confirms they align with the team's intended approach.
Establish quotas and cost budgets for each agent and user, consistently track token and dollar costs per task, and connect the generated dashboards directly to SLOs instead of assessing them monthly. Immediate alerting on any cost surges prevents a minor issue from becoming a major problem that goes unnoticed for weeks.
Capacity planning is further complicated by the incorporation of agentic forecasting models that take into account operational telemetry such as incident frequency, vendor lead times, and request seasonality, which are often overlooked by traditional forecasting methods, leading to continuous improvement in quarterly predictions.
In an agent postmortem, the most frequent failure mode is attributing the root cause to 'the agent hallucinated' and then closing the ticket without identifying the true reason. This is typically due to a lack of guardrails, incomplete evaluation coverage, errors in context engineering, or discrepancies between a tool's schema and the agent's expectations.
A postmortem template with clear prompts at every step ensures a comprehensive analysis of the system, rather than relying on the writer to uncover hidden issues independently.
Teams that effectively implement incident-response augmentation can expect a significant decrease in mean-time-to-resolution within a short period of deployment, without any alterations to the severity matrix or escalation policy. The improvement is mainly attributed to eliminating slow manual tasks within the existing runbook, such as reading logs, locating the necessary information, and drafting updates, rather than implementing a completely new process.
Instead of reducing the number of people involved, new positions are created: an Incident Commander to oversee the interaction between humans and technology in real-time during emergencies, and a Capacity Analyst whose predictions significantly impact operational decisions. The objective is not to decrease human involvement in operations, but to ensure that resources are utilized effectively.
Deploying a new agent version safely into production involves staged rollout gates, while ensuring ongoing system health through various operational tasks such as SLOs, on-call support, incident response, runbooks, postmortems, and cost and capacity management. This is essential for the sustained success of the system beyond just the initial release.
As agent outputs are intentionally diverse even under normal operation, using a fixed threshold can be either too chaotic or too limited in detecting actual issues. Statistical anomaly detection, on the other hand, focuses on learning an agent's typical behavior and notifies of significant deviations from that baseline, rather than comparing each output to a single predetermined value.
Detection involves identifying statistical signals indicating drift, containment focuses on halting the spread of failure, eradication entails removing the reversible cause while preserving evidence, recovery entails resuming operations in verified tranches, and postmortem involves identifying the true root cause rather than just the visible symptoms.
Autonomy should be limited to a specific, low-risk set of frequent actions, with each action having its own scope rather than being applied universally across all agents. For example, an agent may be able to independently perform cache invalidation but would need human confirmation for service restarts and manual review for any database-related tasks.
The reason for a hallucination is often a specific and fixable issue, such as a missing guardrail or a gap in evaluation coverage, rather than just the symptom of the failure.
Implementing agentic triage, log synthesis, and root-cause hypothesis generation typically results in a 50% reduction in mean-time-to-resolution within a quarter for teams, without altering the severity matrix or escalation policy. This enhancement is achieved by streamlining the cognitive processes within the current runbook, rather than introducing a new procedure.
Develop composite SLOs prior to your next incident, transform your top three most frequently used runbooks into organized plays, and establish autonomy for each action from the start.