Worked Example: Assistant Timeouts at Peak

A fictional incident taken from raw evidence to a finished RCA, step by step, with the questions you'd ask at each stage.

Step 1: Does It Need an RCA?

Yes. It was a Sev 2 with 83 minutes of customer-visible impact, and it was the second throttling-related incident in 60 days. The incident lead (Sam, SRE) writes the draft by Monday, and the review is booked for Wednesday.

You: "Sam, can you own the draft? Priya, can you join to cover the pod side? I've booked Wednesday at 10."

Step 2: Timeline From Evidence

Time to detect: 7 minutes (good). Time from detection to mitigation: 68 minutes (the problem). Most of it was lost in NOC escalation (19 minutes) and in finding an owner (16 minutes).

// Application Insights / Log Analytics (KQL): throttled calls to the model provider, per minute
      dependencies
      | where timestamp between (datetime(2026-10-08 13:30) .. datetime(2026-10-08 15:45))
      | where target has "model-provider"
      | summarize throttled = countif(resultCode == "429"), total = count() by bin(timestamp, 1m)
      | render timechart

You: "Where did 14:05 come from, the graph or someone's memory?" "Can we link the PagerDuty entry for the acknowledgement time?"

Step 3: Impact in User Terms

"From 14:05 to 15:28 (83 minutes), about 22% of assistant requests (about 180,000) failed or took over 30 seconds. Users saw 'Something went wrong. Try again.' All regions and device types were affected."

You: "Would support recognize this description from the tickets they got?"

Step 4: Trigger vs Conditions

• Trigger (why now): the new prompt template sends about 40% more tokens per request, and it shipped just before the afternoon peak.
• Contributing factors (why we were vulnerable):
  • Prompt templates are treated as config and skip the canary.
  • There's no alert on throttling or on token usage approaching the quota.
  • Clients retry 3 times immediately, without backoff, which makes throttling worse.
  • The NOC runbook says "monitor for 15 minutes."
  • Nobody had written down who owns the provider quota.

You: "If someone else had shipped a different long prompt, would we have had the same outage?" (Yes. So the problem is the conditions, not the person.)

Step 5: The Four Defenses

You: "Every defense except detection failed somewhere. Do we have an action for each one?"

Step 6: 5 Whys, With Two Branches

1. Why did requests fail? Provider calls were throttled.
2. Why? We exceeded the token-per-minute quota at peak.
3. Why? The new prompt uses about 40% more tokens, and it went to 100% of traffic at once.
4. Why at once? Prompt changes skip the canary. → Prevent branch
5. Why didn't we see the quota coming? We don't alert on token usage or throttling. → Detect branch

A third branch, "Why did it take 68 minutes to mitigate?", leads to the NOC runbook rule and the unclear ownership. That's the respond branch.

Step 7: Actions (Ranked)

You: "If we only did one, which? Is #2 really a week? What does Priya need to hit three weeks?"

Step 8: Review and Follow-Through

• Review: 45 minutes. The NOC analyst who took the first alert walked through the first 15 minutes, and they were thanked for following the runbook exactly as written.
• Shared: a 5-line summary went to leadership, support, and the NOC.
• Tracked: actions are on the monthly reliability review. Two weeks later, #2 and #3 are done. #2 was verified with a simulated throttle in staging.
• Verified: the next prompt change went through the canary, hit throttling at 5% of traffic, and rolled back automatically. Users never noticed. Log that as a win.

Next: Use the template for your next RCA