Alerting & Incidents
How problems reach a person, how that person responds, and how the organization learns: alerting, on-call, incident management, postmortems, and the MTTx metrics.
Alerting (Symptom-Based and Burn-Rate)
Alerts should wake a person only when users are being hurt, or soon will be, and the person can do something about it.
In IT Ops Terms
The opposite of "alert on every threshold." Classic NOC alerting pages on causes like CPU at 90%. SRE alerting pages on symptoms (users failing) and sends cause-level signals to dashboards.
Why It Matters at Your Level
Noisy alerts burn out your SREs and train the offshore NOC to ignore pages. Fewer, better alerts are the quickest reliability and morale win you can sponsor.
How to Implement It
- SRE: list every alert that paged in the last 30 days and how often. For each, ask "what did someone do when it fired?" Delete or demote anything where the answer is "nothing."
- SRE: add burn-rate alerts on each SLO: page when the error budget is being used up fast (for example, 2% of the monthly budget in an hour), and open a ticket when it's being used slowly.
- SRE + NOC lead: every alert that reaches a person links to a runbook and has a clear escalation rule.
- You: track pages per on-call shift as a team health number.
Where it lives in Azure + Grafana: Alert rules in Grafana Alerting (on SLO and budget queries) or Azure Monitor alerts. Both are routed to PagerDuty services, with escalation policies per team. Low-urgency alerts go to tickets or chat, not pages.
What Done Looks Like
- Every paging alert is about user impact and links to a runbook.
- Burn-rate alerts exist for each SLO.
- Pages per shift are tracked and trending down.
- No alert has paged more than 5 times in a month without a fix or deletion decision.
Ask this: “When this alert fired last, what did someone do?” · “Which alert would you delete forever?” · Triage & Escalation questions
Go deeper: Workbook, Ch. 5: Alerting on SLOs (opens in a new tab) · SRE Book, Ch. 10: Practical Alerting (opens in a new tab) · Grafana docs: Grafana Alerting (opens in a new tab)
On-Call
A rotation of people who respond when production breaks, with clear hours, escalation paths, and limits on load.
In IT Ops Terms
Your 24x7 shift model, extended to engineers. In SRE, on-call load is treated as a reliability risk in its own right: tired people make slow, risky decisions.
Why It Matters at Your Level
On-call health predicts retention and incident quality. As a central SRE and NOC leader, you own the whole chain from the NOC's first look to the pod's engineer.
How to Implement It
- You: map the chain for the assistant: NOC (L1) → SRE on-call → owning pod on-call, with timeouts at each step.
- SRE: build it as PagerDuty escalation policies so the chain runs itself. If SRE doesn't acknowledge in 10 minutes, it goes to the secondary, then to you.
- You: set load limits, such as a target of at most 2 incidents per shift, and give recovery time after rough nights.
- You: review on-call load monthly using PagerDuty's reports.
Where it lives in Azure + Grafana: PagerDuty: schedules, escalation policies, and services per team. The NOC's alerts and escalations route into the same services, so there's one record.
What Done Looks Like
- The NOC → SRE → pod chain exists as PagerDuty escalation policies.
- Every critical service has an owning pod with its own on-call.
- On-call load is reviewed monthly and acted on.
Ask this: “What's the most annoying thing about on-call right now?” · “If SRE doesn't acknowledge, what happens next, automatically?” · 1:1 With an Engineer questions
Go deeper: SRE Book, Ch. 11: Being On-Call (opens in a new tab) · Workbook, Ch. 8: On-Call (opens in a new tab) · PagerDuty: Escalation policies (opens in a new tab)
Incident Management
A standard way to run an incident: declare it, assign roles, communicate on a rhythm, stop the bleeding, then learn from it.
In IT Ops Terms
ITIL incident management, with three SRE emphases: declare early, mitigate before diagnosing, and separate roles (one person leads, others fix, one communicates).
Why It Matters at Your Level
Most long incidents aren't technically hard. They're disorganized. Structure is where you, a non-coding leader, add the most value.
How to Implement It
- You: write severity criteria based on user impact (who, how many, how long), so the NOC can declare against criteria, not judgment.
- You: define three roles: incident lead, communications, and subject experts. Name who can play each.
- SRE: set up PagerDuty so declaring an incident pages the right services and creates the channel or bridge.
- You: set the update rhythm: every 30 minutes for Sev 1, every 60 for Sev 2, using a fixed template.
- You: run a short drill once a quarter, including the NOC.
Where it lives in Azure + Grafana: PagerDuty incidents (severity, responders, timeline), one chat channel per incident, and Grafana links pinned in the channel.
What Done Looks Like
- Written severity criteria that the NOC uses.
- Named roles in every Sev 1 and Sev 2.
- Updates go out on schedule.
- A drill was run this quarter.
Ask this: “Who's leading this incident right now?” · “What's the fastest way to stop the bleeding?” · First 15 Minutes questions
Go deeper: SRE Book, Ch. 14: Managing Incidents (opens in a new tab) · Workbook, Ch. 9: Incident Response (opens in a new tab) · PagerDuty: Incident Response guide (opens in a new tab)
Postmortems (RCAs)
A written, blameless review after an incident that explains what happened and produces a few owned actions that make a repeat less likely.
In IT Ops Terms
Problem management. The SRE differences are looking for contributing factors rather than one root cause, checking all four defenses, and tracking actions to verified closure.
Why It Matters at Your Level
This is how firefighting turns into prevention. Repeat incidents usually trace back to an RCA whose actions were never finished.
How to Implement It
- You: set the criteria for when an RCA is required, and the timing: draft in 3 working days, review in 5.
- Owning engineer: writes it using the RCA template, with an evidence-based timeline.
- You: check it with the 5-minute check and run the 45-minute review.
- You: track open actions monthly and verify the top one.
Where it lives in Azure + Grafana: Evidence from Grafana, Application Insights, the Activity Log, and the PagerDuty incident log. Actions tracked in your team's work tracker.
What Done Looks Like
- Every qualifying incident has an RCA within 5 working days.
- Actions are ranked, owned, dated, and checkable.
- Open actions are reviewed monthly.
- Repeat incidents trend down.
Ask this: “Has this happened before, and what did we say we'd do last time?” · “Which defenses failed?” · Judging an RCA questions
Go deeper: SRE Book, Ch. 15: Postmortem Culture (opens in a new tab) · Workbook, Ch. 10: Postmortem Culture (opens in a new tab) · PagerDuty: Postmortem guide (opens in a new tab)
MTTD, MTTA, MTTR (and Their Limits)
Average times to detect, acknowledge, and resolve incidents. They're useful for spotting where time goes, and misleading as targets.
In IT Ops Terms
The same metrics you've reported for years. The SRE caution is that averages across very different incidents hide what matters, and turning them into targets encourages closing incidents early.
Why It Matters at Your Level
Leadership will ask for MTTR. Give them context, not just one number, or it will get used as a stick against your team.
How to Implement It
- SRE: pull time to detect, acknowledge, and mitigate for each Sev 1 and Sev 2 from PagerDuty, and show them per incident, not just as averages.
- You: report them next to SLO attainment and repeat-incident counts, which say more about reliability.
- You: use the biggest gap (often acknowledge to escalate) to choose your next process fix.
Where it lives in Azure + Grafana: PagerDuty analytics for acknowledge and resolve times. Impact start times come from Grafana in the RCA timeline (detection is usually later than impact).
What Done Looks Like
- Times are reported per incident, alongside SLOs and repeats.
- The biggest gap has a named improvement.
- Nobody is penalized for an honest long resolution.
Ask this: “Is MTTR being used to judge people, or to find process gaps?” · “Which gap was bigger: detect, or mitigate?” · Judging an RCA questions
Go deeper: PagerDuty: What is MTTR? (opens in a new tab) · SRE Book, Ch. 16: Tracking Outages (opens in a new tab)