Running an RCA
A step-by-step method you can lead without being the most technical person in the room, plus a 45-minute facilitation script.
Your Role
You usually shouldn't write the RCA. The incident lead or an engineer from the owning pod should. Your job is to make sure it happens on time, that it follows these steps, and that the right questions get asked. You can run every step below without reading code.
Step 1: Decide Whether It Needs an RCA
Write the trigger criteria down once, so it's never a debate. A sensible starting set:
- Any Sev 1 or Sev 2 incident.
- Customer-visible impact longer than 30 minutes.
- Any repeat of an incident from the last 90 days.
- A near miss that could have been major.
- Any data loss or security exposure.
Name the author and the review date before the incident channel goes quiet: a draft within 3 working days, a review within 5.
Step 2: Build the Timeline From Evidence
The timeline is the foundation. Build it from systems, not memory, because people are reliably wrong about times.
| Source | What it tells you |
|---|---|
| Grafana dashboards and annotations | When impact started and ended, and deploy or change markers |
| Application Insights / Log Analytics | Failed requests, slow requests, and calls to dependencies such as the model provider, by minute |
| Azure Activity Log | Infrastructure and configuration changes in Azure, and who or what made them |
| Deploy pipeline history | What code, config, or prompt changes shipped, and when |
| PagerDuty incident log | When alerts fired, who was paged, acknowledgement times, and escalations |
| Incident channel and bridge notes | Decisions, who joined when, and what was tried |
Mark these six timestamps, because they make the gaps visible:
- Impact started (from the graph, not from when someone noticed)
- Detected (first alert, or first human report)
- Acknowledged (someone owned it in PagerDuty)
- Escalated (the right team engaged)
- Mitigated (user impact stopped)
- Resolved (cause fixed or safely contained)
Then calculate time to detect (1→2) and time to mitigate (2→5). The bigger gap tells you where to invest.
Step 3: State the Impact in User Terms
One or two sentences a support agent would understand: who was affected, how many, for how long, and what they saw. For example: "From 14:05 to 15:28, about 22% of assistant requests failed or took over 30 seconds. Users saw 'Something went wrong.'" Get the numbers from Application Insights. Rough beats missing.
Step 4: Separate the Trigger From the Conditions
Ask two different questions and write both answers down:
- "Why did this happen now?" gives you the trigger: the change, the traffic spike, the provider event.
- "Why were we vulnerable to it?" gives you the contributing factors: the missing canary, alert, limit, or runbook step.
If the analysis lists only a trigger, it isn't finished.
Step 5: Walk the Four Defenses
For each layer, ask what failed and what held. (See The Four Defenses.)
- Prevent: what would have stopped this change or condition from reaching production?
- Detect: what signal would have told us sooner? Did the alert go to the right place in PagerDuty?
- Mitigate: what would have limited how many users were hurt, or for how long? Rollback, feature flags, failover, rate limits.
- Respond: where did humans lose time? Unclear ownership, runbook gaps, waiting on the wrong team, NOC escalation rules.
Step 6: Use "5 Whys" Without Falling Into Its Traps
Asking "why" repeatedly is useful, as long as you avoid the traps below. Done well, it looks like this:
| Why? | Because… |
|---|---|
| Why did requests time out? | Calls to the model provider were throttled. |
| Why were they throttled? | We exceeded our token-per-minute quota. |
| Why did we exceed it? | The new prompt template sends about 40% more tokens per request, at peak traffic. |
| Why did that reach all users at once? | Prompt changes are treated as config and skip the canary. (Contributing factor → prevent action) |
| Why didn't we see the quota approaching? | There's no alert on token usage or throttling. (A second branch → detect action) |
The Traps
- Stopping at a person ("because Sam deployed it").
- Ask instead: "What allowed that deploy to reach every user?"
- Following a single chain.
- Real incidents branch. Follow at least the prevent branch and the detect branch.
- Going too deep ("because of budget").
- Stop at the level where your team or a pod can own a concrete action.
- Asking "who" instead of "what."
- "Who" gets defensiveness. "What made this possible" gets facts.
Step 7: Write Fewer, Better Actions
Three to five actions, ranked, each labeled with the defense it strengthens. Every one needs a named person, a date, and what done means. Include at least one Detect action.
| Weak action | Strong action |
|---|---|
| Investigate root cause further. | Prompt changes use the canary pipeline. Owner: Priya. Due in 3 weeks. Done means the next prompt change rolls to 5% first and rolls back automatically on errors. |
| Improve monitoring. | Alert when provider throttling exceeds 1% for 5 minutes, routed to the SRE PagerDuty service. Owner: Sam. Due in 1 week. Done means it's tested by simulating throttling. |
| Engineers to be more careful. | Remove this as an action. Replace it with the system change that makes care unnecessary. |
| NOC to escalate faster. | NOC runbook: "if the slowest 5% of requests are over 8 seconds for 5 minutes, page SRE." Owner: NOC lead. Due in 1 week. Done means it's verified by teach-back with every shift. |
Step 8: Run the Review (45 Minutes)
| Time | Block | Ask |
|---|---|---|
| 00–05 | Ground rules & impactRead the blameless rule aloud, then the impact statement. | "Does everyone agree with the impact numbers?" |
| 05–20 | Timeline walkThe author walks the evidence-backed timeline. Mark the gaps. | "Where did we lose the most time?" |
| 20–30 | Trigger, factors & defensesConfirm the trigger, the contributing factors, and the four defenses. | "Which defenses failed?" |
| 30–40 | ActionsRank, then assign an owner, a date, and what done means. | "If we only did one, which?" |
| 40–45 | CloseAgree who it gets shared with, and when actions are reviewed. | "Who outside this room needs to read this?" |
We're here to understand how our system allowed this to happen, not to find someone to blame.
Assume everyone did what made sense with what they knew at the time.
We'll leave with a few actions, each with an owner, a date, and what "done" means.After the review: put the actions into your tracking system, review open ones in the monthly reliability review, and verify the top action actually reduces the risk. That could be a drill, a test, or watching the metric that failed.
Example in Practice
See the full worked example. It takes one fictional incident through all eight steps.
What Actually Happens
- The review turns into re-debugging the incident.
- Counter: "Good question for after. Let's park it and finish the timeline."
- The most senior engineer dominates, and the NOC analyst never speaks.
- Counter: ask the person closest to the first alert to walk the first 15 minutes. You'll learn the most from them.
- Actions get assigned to "the team."
- Counter: no name, no action. If the right person isn't in the room, the action becomes "[pod manager] names an owner by Friday," and it's yours to chase.
- The review slips a week, then two.
- Counter: book it while the incident channel is still open.