Reliability Targets

SLIs, SLOs, SLAs, and error budgets: how to define reliability in numbers, agree on it, and use it to make decisions.

SLI (Service Level Indicator)

A number that measures how well the service is working from the user's point of view, usually "the percentage of requests that were good."

In IT Ops Terms

Like the availability or response-time stats in a service report, but measured for each user request, not by pinging a server. A green server ping doesn't count. What counts is whether real users got good answers.

Why It Matters at Your Level

Every reliability conversation, from targets to alerts to headcount, starts with an agreed SLI. Without one, "is it reliable?" is a matter of opinion, and the loudest person wins.

How to Implement It

  1. You + product: pick the 2–3 user journeys that matter most. For example: "ask the assistant a question and get an answer," "sign in," "sync across devices."
  2. SRE + pod: for each journey, define "good": for example, the request succeeded and answered in under N seconds. Confirm N with product.
  3. SRE: measure it from real traffic in the backend's request data. In other words, count good requests and divide by total requests.
  4. SRE: show it on one Grafana panel per journey, as a percentage over 28 days and over the last hour.

Where it lives in Azure + Grafana: Request and dependency data from Application Insights, queried with KQL in Log Analytics, shown as a panel in Azure Managed Grafana using the Azure Monitor data source.

What Done Looks Like

  • 2–3 SLIs written down, each in one sentence a non-engineer understands.
  • Each is measured from real user traffic, not synthetic pings alone.
  • Each has a Grafana panel that the SRE Daily looks at.

Ask this: “Is this measuring what the user experiences, or what our servers experience?” · “If the assistant gave wrong answers fast, would this SLI notice?” · Dashboard Review questions

Go deeper: SRE Book, Ch. 4: Service Level Objectives · Workbook, Ch. 2: Implementing SLOs · Microsoft Learn: Application Insights overview

SLO (Service Level Objective)

The target for an SLI over a time window, for example "99.5% of assistant requests are good, measured over 28 days."

In IT Ops Terms

Like an internal OLA (operational level agreement) target, but deliberately below 100%, and used to make decisions, not just to report.

Why It Matters at Your Level

An SLO turns "be more reliable" into a number you can manage to, report up, and trade against feature work. It's the clearest answer to "how reliable are we?"

How to Implement It

  1. SRE: look at the last 4–8 weeks of the SLI to see where you actually are.
  2. You + product + pod: set a target a little better than today but achievable, such as 99.5%, not 99.99%. Too high and it's ignored. Too low and users suffer.
  3. SRE: build an SLO dashboard: the SLI line, the target line, and error budget remaining (see below).
  4. You: write it down in one page (an SLO document): journey, SLI, target, window, owner, and what happens when it's missed.
  5. You: review it weekly in the SRE Daily and monthly with leadership. Adjust the target after one quarter if needed.

Where it lives in Azure + Grafana: Azure Managed Grafana has no built-in SLO feature, so build an SLO dashboard: a KQL query for the SLI against Application Insights or Log Analytics, a constant target line, and an "error budget remaining" stat panel.

What Done Looks Like

  • One SLO per critical journey, written in a one-page SLO document.
  • Agreed by the owning pod and product, not just SRE.
  • Visible on a Grafana dashboard and reviewed weekly.
  • Reported monthly to leadership as "met / at risk / missed."

Ask this: “What would make you confident reliability is improving: what number?” · “Who agreed to this target besides SRE?” · Owning the Reliability Story

Go deeper: SRE Book, Ch. 4: Service Level Objectives · Workbook, Appendix A: Example SLO Document · Microsoft Learn: Azure Managed Grafana overview

SLA (Service Level Agreement)

A promise to customers, usually in a contract, with consequences such as credits if it's broken.

In IT Ops Terms

The same SLA you know from IT service management. The SRE twist is that your internal SLO should be stricter than the SLA, so you get warned before you break the promise.

Why It Matters at Your Level

SLAs are business commitments made by sales or legal, sometimes without SRE input. If the SLA is tighter than what you can actually deliver, that's a risk you need to raise.

How to Implement It

  1. You: find out whether any customer or partner SLAs exist for the assistant, and what they promise.
  2. You + SRE: make sure each SLO is stricter than the matching SLA. For example, SLO 99.5% and SLA 99.0%.
  3. You: agree with sales and legal that new SLAs get an SRE review before they're signed.

Where it lives in Azure + Grafana: SLAs live in contracts, not tools. Track your SLO against the SLA line on the same Grafana dashboard, so the gap is visible.

What Done Looks Like

  • You know every SLA that exists and its number.
  • Each has a stricter internal SLO.
  • There's an agreed rule that SRE reviews new SLAs before signature.

Ask this: “Has anyone promised a customer a number we haven't measured?” · “Is our SLO stricter than the SLA?” · Boss & Execs questions

Go deeper: SRE Book, Ch. 4: Service Level Objectives

Error Budget

The amount of unreliability your SLO allows. A 99.5% SLO means 0.5% of requests can fail before you've missed it.

In IT Ops Terms

Like a change-risk allowance: the room you have to take risks before you have to slow down. A 99.9% monthly target allows about 43 minutes of full outage a month.

Why It Matters at Your Level

The error budget turns arguments about reliability versus features into a rule. If there's budget left, ship faster. If it's spent, focus on reliability. It's the most powerful tool a central SRE leader has.

How to Implement It

  1. SRE: calculate the budget from each SLO (1 − target), and how much is left this window.
  2. SRE: add "budget remaining" and "burn rate" (how fast it's being used) to the SLO dashboard.
  3. You: report budget remaining weekly. That one number tells leadership whether to worry.

Where it lives in Azure + Grafana: A Grafana stat panel showing 1 − (bad requests ÷ allowed bad requests) over the window, calculated from the same Application Insights query as the SLI.

What Done Looks Like

  • Budget remaining is visible for each SLO.
  • Burn rate is visible over the last hour and day.
  • Budget is mentioned in at least one real decision, such as delaying a launch or approving one.

Ask this: “How much error budget is left this month, and what used most of it?” · “At this burn rate, when do we run out?” · Roadmap & Priorities questions

Go deeper: SRE Book, Ch. 3: Embracing Risk · SRE Book, Appendix A: Availability Table

Error Budget Policy

A written agreement about what happens when the error budget runs low or is spent, for example "pause risky launches and put engineers on reliability fixes."

In IT Ops Terms

Like a change freeze triggered by stability problems, but agreed in advance and applied automatically, not decided in a heated meeting.

Why It Matters at Your Level

Without a policy, the error budget is just a graph. With one, you have pre-agreed leverage with pods and leadership, so you're not arguing from scratch during a bad month.

How to Implement It

  1. You: draft a one-page policy. At 50% budget used, review the top offenders. At 100%, non-critical launches pause until the budget recovers or leadership grants an exception.
  2. You + pod managers + your boss: agree it and sign it. Without leadership sign-off it won't hold.
  3. You: apply it the first time it triggers, and say so publicly. That first use is what makes it real.

Where it lives in Azure + Grafana: The policy is a document. The trigger is a Grafana alert on budget remaining, routed to you and the SRE leads through PagerDuty at low urgency, not as a page.

What Done Looks Like

  • A signed one-page policy.
  • An alert at 50% and 100% budget used.
  • The policy has been applied at least once, and leadership backed it.

Ask this: “When the budget is gone, what actually changes?” · “Who can grant an exception, and is it written down?” · Boss & Execs questions

Go deeper: Workbook, Appendix B: Example Error Budget Policy · Workbook, Ch. 2: Implementing SLOs