What changes
Before and after
The classic way
- Alerts fire on static thresholds and most of them are noise.
- Runbooks go stale between incidents; the person on call searches a wiki at 3 a.m.
- The postmortem is written days later from memory.
The AI way
- Google SRE describes augmenting threshold alerts with anomaly detection on deviations from regular behavior, agents that group, pre-process and enrich alerts, and autonomous alert handlers that can address or mitigate issues, with a likely significant reduction in the alerts SREs review [3].
- Agents monitor and improve playbooks from how they are used during incidents and can generate new playbooks from incidents; in incident management they summarize data, create hand-off documents, draft postmortems and manage communications [3].
- The limits are stated as clearly as the gains: an agentic approach does not necessarily imply removing people, specifically for higher-risk services and features, but it does significantly reduce the time people need to spend [3].
Step by step
The three steps of Operate
- 1
Signals, not thresholds AI
Agents collect signals and feed a model that flags deviations from regular behavior, then group and enrich what fires, so the person on call sees one enriched incident, not forty alerts [3].
- 2
Investigate and propose AI
Using observability data, topology and dependencies, an agent forms hypotheses and proposes mitigation steps; for some low-risk cases it mitigates on its own [3].
- 3
A person holds the risky ones Gate
For higher-risk services a person decides, with the agent's reasoning in front of them: Google's principles require agents to explain why and how they acted and what they rejected, favoring transparency over black-box automation [3].
The gate
Agents with a strong identity, roles and permissions, reliability SLOs of their own and well-defined backup options, automated or manual; people in the loop for higher-risk services [3].
What the AI does here
Detects anomalies, enriches and groups alerts, investigates, proposes or performs low-risk mitigations, keeps playbooks current, drafts postmortems and communications.
Tools people use at this stage
- OpenTelemetry
- Prometheus and Grafana
- PagerDuty and incident.io
- Gemini and Claude for operations
- TimesFM
Named as examples, not endorsements; the center is not affiliated with any of them.
What this stage adds
The risks, and the practices that answer them
Each risk is a weakness in the Learning Center, with the incidents where it happened; each practice has its sources there.
Risks
- Agents with standing write access to productionThe on-call agent has standing write access to production.
- Cascading failures in agent workflowsOne agent's mistake cascades through the others.
- Missing audit trail of agent actionsNobody can replay what the agent did during the incident.
Read for this stage