At 3:14 AM on a Saturday, an alert fires. A core service is experiencing elevated latency and a 4% error rate.
The traditional response: PagerDuty wakes up a human. They SSH in, bleary-eyed, query Datadog, read logs, check recent deployments, and try to piece together the puzzle while the SLA burns.
The new response: An SRE agent that was already watching the metrics sees the anomaly. Within seconds, it correlates the latency across three dependent services, identifies a runaway query introduced in a deployment two hours ago, and proposes a rollback. All before the human's phone rings.
This is Agentic SRE.
From AIOps to Agentic SRE
AIOps has been around for years, but it was fundamentally passive. It offered better alert routing, smarter log correlation, and noise reduction. It told the human where to look.
Agentic SRE is active. It performs autonomous investigation, diagnosis, and, crucially, remediation.
Passive AIOps
Active Agentic SRE
The Architecture of an SRE Agent
An SRE agent is a multi-modal system equipped with specific operational tools.
| Capability Tier | Description |
|---|---|
| Monitoring Integration | Reads metrics, logs, and distributed traces via APIs (Datadog, Prometheus). |
| Investigation Tools | Queries dashboards, reads internal runbooks, and checks the Git commit history. |
| Remediation Tools | Executes scripts to restart services, scale capacity, or rollback deployments. |
The Human-Agent Handoff
The most important design decision in Agentic SRE is the human-agent handoff. When does the agent act autonomously, and when does it escalate?
We implement this via privilege rings:
- Tier 1 (Autonomous): Low-risk actions. Restarting a stateless worker pod, scaling up read replicas. The agent executes and notifies.
- Tier 2 (Propose + Await Approval): Medium-risk actions. Rolling back a deployment, pushing a config change. The agent prepares the PR or command, pages the human, and waits for a "yes/no".
- Tier 3 (Escalate Immediately): High-risk incidents. Data loss risks, security breaches, database schema corruptions. The agent gathers context but takes zero modifying actions.
Safety Rails for Autonomous Remediation
An agent that has the access to fix your infrastructure also has the access to destroy it.
You must implement blast radius limits. An agent might be allowed to restart 10% of the fleet, but a circuit breaker trips if it attempts to restart 50%. You need remediation budgets — if an agent has executed three automated fixes in an hour and the alerts are still firing, it must stop and escalate.
The Evolving Role of the Human SRE
Agentic SRE doesn't eliminate the Site Reliability Engineer; it elevates them.
The human shifts from firefighter to architect. SREs spend their time designing the agent's runbooks, reviewing its past actions to improve its prompts, calibrating its confidence thresholds, and handling the complex, novel failures that the agent can't understand.
What Can Go Wrong
The failure modes are new and terrifying:
- Misdiagnosis: The agent confidently misinterprets a spike in legitimate traffic as a DDoS attack and aggressively rate-limits real customers.
- The Remediation Loop: The agent restarts a service, the service takes 2 minutes to boot, the agent assumes the restart failed and restarts it again, trapping the system in a permanent boot loop.
// key takeaway