Judging an RCA
Someone hands you an RCA. Here's how to tell in five minutes whether it will actually prevent a repeat, and what to send back.
Use This When
An RCA lands in your inbox, or you're asked to approve one. You don't need to understand every technical detail to tell whether it will prevent a repeat. You need to check its structure.
The 5-Minute Check
- Impact is stated in user terms, with numbers: who, how many, how long, what they saw.
- Timeline entries link to evidence: Grafana, Application Insights, the Activity Log, PagerDuty.
- Time to detect and time to mitigate are both stated.
- Trigger and contributing factors are both named, and "human error" isn't the conclusion.
- All four defenses (prevent, detect, mitigate, respond) were examined.
- Previous incidents of the same kind are linked, along with what happened to their actions.
- 3–5 ranked actions, each with a person, a date, and a checkable definition of done. At least one improves detection.
- The summary makes sense to someone outside the team.
If it fails three or more of these, send it back before the review meeting. That's not bureaucracy. A weak RCA wastes the review.
Red Flags and What to Send Back
| Red flag | Why it matters | Send back |
|---|---|---|
| "Root cause: human error" | It prevents nothing. The next person will make the same mistake. | "What made that error possible, and what would have caught it?" |
| "Action: be more careful" or "add more review" | It relies on people never having a bad day. | "What system change would make care unnecessary?" |
| 15 actions, unranked | None of them will get done. | "Rank them. Keep the top 3–5 and move the rest to the backlog." |
| No detection action | The next incident will also be found late. | "What would have told us 10 minutes sooner?" |
| Owners are teams ("Assistant pod") | Nobody owns it. | "Which person on the pod?" |
| "Investigate further" as an action | There's no end state. | "What will be different in production when this is done?" |
| Timeline with no sources | It's memory, which is unreliable and invites argument. | "Link the graph, query, or PagerDuty entry for each time." |
| Treated as a first occurrence when it isn't | Hides a follow-through problem. | "Link the previous incident. Why didn't its actions prevent this?" |
Weak vs Strong, Side by Side
| Section | Weak | Strong |
|---|---|---|
| Summary | "The assistant had issues due to a config change. Fixed by rollback." | "From 14:05 to 15:28, about 22% of assistant requests failed or took over 30 seconds. A longer prompt template pushed us past the model provider's rate limit at peak. We had no canary for prompt changes and no alert on throttling. Rolled back at 15:20." |
| Root cause | "Engineer deployed a prompt change without testing." | "Trigger: a prompt change. Contributing factors: prompt changes skip the canary, no throttling alert, retries without backoff, and a NOC runbook that said wait 15 minutes." |
| Actions | "1. Test changes better. 2. Improve monitoring. 3. Investigate retries." | "1. Prevent: canary for prompt changes (Priya, 3 weeks). 2. Detect: throttling alert to PagerDuty (Sam, 1 week). 3. Mitigate: backoff on retries (Priya, 2 weeks). 4. Respond: new NOC escalation rule (NOC lead, 1 week)." |
A Reply You Can Send
Feedback on an RCA draft
Thanks for pulling this together. Before the review, a few things would make it stronger:
1. Impact: can we add numbers (% of requests affected, duration, what users saw)?
2. Timeline: please link the Grafana panel / App Insights query / PagerDuty entry for each timestamp.
3. Causes: "human error" isn't where we stop. What made the error possible, and what would have caught it?
4. Defenses: is there at least one action that helps us DETECT this sooner next time?
5. Actions: please rank them and keep the top 3-5, each with a person, a date, and what "done" means.
Happy to talk any of this through.Questions to Ask
Use these in the review meeting, or when you're replying to a draft.
What Actually Happens
- The author is senior and pushes back on your feedback.
- Counter: ask questions instead of judging. "What would have caught this sooner?" is hard to argue with, and it doesn't need technical depth.
- You approve weak RCAs because everyone is exhausted.
- Counter: approve the timeline and impact now, and give the actions 48 more hours. Never approve vague actions.
- The RCA is strong, but nothing happens afterwards.
- Counter: the RCA isn't done when it's approved. It's done when the top action is verified.