SRE Concepts: Overview & Roadmap

Every core SRE concept in one sentence, and the order to roll them out. Each links to a practical page: what it means, how to implement it in Azure and Grafana, and what done looks like.

How to Use This Section

Each concept is written for a senior manager: what it means in one sentence, the IT Ops equivalent, why it matters at your level, how to implement it in an Azure + Azure Managed Grafana + PagerDuty environment, what done looks like, what to ask, and where to read more in Google's free SRE books.

A Rollout Order That Works

If you're starting from firefighting, roll these out in order. Each one makes the next one possible.

  1. SLIs for your 2–3 most important user journeys.
  2. SLOs agreed with pods and product, shown on a Grafana dashboard.
  3. Alerting on SLOs: burn-rate pages in PagerDuty, and delete the noise.
  4. An error budget policy signed by leadership.
  5. Postmortems with tracked, verified actions (see Root Cause Analysis).
  6. Toil measured and cut, with the time reinvested in prevention.

Reliability Targets

ConceptIn one sentence
SLI (Service Level Indicator)A number that measures how well the service is working from the user's point of view, usually "the percentage of requests that were good."
SLO (Service Level Objective)The target for an SLI over a time window, for example "99.5% of assistant requests are good, measured over 28 days."
SLA (Service Level Agreement)A promise to customers, usually in a contract, with consequences such as credits if it's broken.
Error BudgetThe amount of unreliability your SLO allows. A 99.5% SLO means 0.5% of requests can fail before you've missed it.
Error Budget PolicyA written agreement about what happens when the error budget runs low or is spent, for example "pause risky launches and put engineers on reliability fixes."

Observability & Telemetry

ConceptIn one sentence
Monitoring vs ObservabilityMonitoring tells you that something is wrong. Observability lets you work out why, including for problems nobody predicted.
TelemetryThe data your systems emit about themselves: metrics, logs, and traces. Observability is built from it.
Metrics, Logs & TracesThe three kinds of telemetry. Metrics are numbers over time. Logs are detailed records of events. Traces show one request's path through every service it touched.
The Four Golden SignalsThe four things to watch for any user-facing service: latency (how slow), traffic (how much), errors (how many fail), and saturation (how full).
DashboardsPages of graphs that help someone answer a specific question quickly. A good dashboard has a purpose and an audience. A bad one is a wall of every metric.
AI Quality SignalsMeasures of whether the assistant's answers are good, not just fast and error-free. An AI product can be up and still wrong.

Alerting & Incidents

ConceptIn one sentence
Alerting (Symptom-Based and Burn-Rate)Alerts should wake a person only when users are being hurt, or soon will be, and the person can do something about it.
On-CallA rotation of people who respond when production breaks, with clear hours, escalation paths, and limits on load.
Incident ManagementA standard way to run an incident: declare it, assign roles, communicate on a rhythm, stop the bleeding, then learn from it.
Postmortems (RCAs)A written, blameless review after an incident that explains what happened and produces a few owned actions that make a repeat less likely.
MTTD, MTTA, MTTR (and Their Limits)Average times to detect, acknowledge, and resolve incidents. They're useful for spotting where time goes, and misleading as targets.

Change & Toil

ConceptIn one sentence
Change Management (the SRE Way)Making change safe by making it small, tested, gradual, and easy to undo, rather than by adding approval meetings.
Canary, Feature Flags & RollbackCanary: release to a small slice of users first. Feature flag: turn a feature on or off without a deploy. Rollback: return to the last good version quickly.
ToilManual, repetitive operational work that scales with traffic and leaves nothing lasting behind, like restarting the same service every week.
AutomationReplacing a repeated manual task with something that runs itself, done when the time saved is clearly worth the effort and the risk.
RunbooksStep-by-step instructions for handling a specific alert or task, written so the person on shift can follow them under pressure.

Capacity & Readiness

ConceptIn one sentence
Capacity Planning & Load TestingKnowing how much traffic the system can handle, what breaks first, and making sure there's room for the load that's coming.
Production Readiness Review (PRR)A checklist review before a new service or major feature goes live, to confirm it can be run safely: monitored, alerted, documented, owned, and reversible.
Dependency ManagementKnowing what your service relies on but doesn't control (the model provider, identity, third-party APIs), and what happens when each one is slow or down.
Cost per RequestWhat it costs to serve one user request, or 1,000 of them. For AI products this is driven heavily by model and token usage.