RCA the SRE Way
What a root cause analysis is really for, why there's rarely one root cause, and what a good RCA produces.
Use This When
You're expected to lead, review, or sign off root cause analyses and you're not sure what "good" looks like. This page gives you the mental model. Running an RCA gives you the steps.
What an RCA Is Actually For
An RCA (root cause analysis, also called a postmortem or post-incident review) has one job: make the same kind of incident less likely, and less damaging if it happens anyway. The document is just the record. What it should produce:
- A shared, accurate story of what happened, backed by evidence rather than memory.
- A few strong actions that improve your defenses, each with an owner, a date, and what "done" means.
- Learning that spreads beyond the people who were on the bridge: other pods, the NOC, leadership.
An RCA that produces a nice document and no completed actions has failed, however well written it is.
Why There's Rarely One Root Cause
In IT Ops problem management you may have been trained to find the root cause. In modern systems, incidents almost always come from a trigger meeting a set of conditions that let it become an outage. SRE calls those conditions contributing factors.
| Term | Plain meaning | Example (AI assistant) |
|---|---|---|
| Trigger | What changed or happened right before | A longer prompt template was deployed at 13:52. |
| Contributing factors | Conditions that turned the trigger into an outage | No canary for prompt changes. No alert on the model provider's rate limits. Retries with no backoff. |
| Impact | What users experienced, how many, for how long | 22% of requests failed or took over 30 seconds, for 83 minutes. |
| Detection | How and when we found out | An alert fired after 7 minutes. Escalation took another 19. |
| Mitigation | What stopped the bleeding | Rolling back the prompt template. |
Fixing only the trigger ("don't ship long prompts") leaves every condition in place for the next trigger. Strong RCAs fix conditions.
The Four Defenses
Think of reliability as layers. An incident means several layers failed at once. For every RCA, ask which layers failed, and make sure there's at least one action for each.
| Defense | The question | Example action |
|---|---|---|
| Prevent | Why could this happen at all? | Prompt changes go through the same canary as code. |
| Detect | Why didn't we know sooner? | Alert on model-provider throttling, routed to PagerDuty. |
| Mitigate | Why did it hurt this much, or for this long? | Retries with backoff, so a provider hiccup doesn't snowball. |
| Respond | Why did the human response take this long? | The NOC runbook says "page SRE after 5 minutes," not "monitor for 15." |
Most weak RCAs only have Prevent actions. Detection and response actions are often cheaper and pay off on every future incident, not just this one.
Blameless Doesn't Mean Accountability-Free
- Blameless means people aren't punished for honest mistakes made in a system that allowed them. If a different person on the same shift could have made the same mistake, it's a system problem.
- Accountable means every action has an owner who delivers it, and leaders follow through. You can, and should, be strict about follow-through.
- Blameless is also practical. If people get punished for being honest in RCAs, the next RCA will be missing the facts you most need.
Say This When Someone Names a Person
"Let's assume they did what made sense with what they knew at the time. What made that the reasonable choice? What would have stopped anyone from making it?"
What Good Looks Like
- Drafted within 3 working days and reviewed within 5, while memories are fresh.
- Every timeline entry is backed by evidence from Grafana, Application Insights, the Azure Activity Log, PagerDuty, or the incident channel.
- User impact is quantified: who, how many, how long, what they saw.
- Both the trigger and the contributing factors are named. "Human error" is never the end of the analysis.
- All four defenses are examined.
- 3–5 ranked actions, each with a named owner, a date, and a checkable definition of done. At least one improves detection.
- Reviewed in a meeting and shared beyond the team.
- Actions are tracked to closure in your monthly reliability review, and the top action is verified to actually reduce the risk.
From IT Ops Problem Management to SRE RCAs
| IT Ops habit | SRE version |
|---|---|
| Find the root cause | Find the trigger and the contributing factors |
| Problem ticket stays open until the fix ships | The RCA closes with a few owned actions, which are tracked separately until verified |
| Known error database | A searchable RCA library, plus repeat-incident tracking |
| Change caused it, so tighten the CAB | Make changes smaller, canaried, and easy to roll back |
Your problem management background is an advantage: you already know how to drive problems to closure. The SRE shift is in what you look for.
Example in Practice
An RCA lands on your desk: "Root cause: engineer deployed a bad config. Action: engineers to double-check configs."
You ask: "Could anyone on the team have made the same change? What would have caught it before users did?" The answers: there's no automated config validation, the change went to 100% of traffic at once, and the error-rate alert was set too loose to fire.
The rewritten actions: config validation in the pipeline (prevent), a canary for config changes (mitigate), and tightening the error-rate alert (detect). Each has an owner and a date. The engineer who deployed it ran the review, and became the person most committed to the fixes.
What Actually Happens
- RCAs get written for the record and nobody reads them.
- Counter: hold a short review meeting for every major incident. Send a 5-line summary to leadership and the NOC.
- The same "root cause" shows up again three months later.
- Counter: track RCA actions like commitments, and review open ones monthly. A repeat incident with an open action is the strongest argument you'll ever have for prioritization.
- Pods push back on actions that land in their code.
- Counter: the RCA names the action and the owning team. Agreeing the date is a manager-to-manager conversation, and you own having it.
- People get defensive, so the facts get vague.
- Counter: open every review with the blameless ground rule, out loud, every time.