Change & Toil

Making change safe, and getting the team's time back: change management, safe deploys, toil, automation, and runbooks.

Change Management (the SRE Way)

Making change safe by making it small, tested, gradual, and easy to undo, rather than by adding approval meetings.

In IT Ops Terms

The CAB's goal (safe change) with a different method. Approvals catch little. Canaries, automated checks, and fast rollback catch most problems.

Why It Matters at Your Level

Most incidents start with a change. For an AI product, that includes prompt and model changes. How changes ship is the biggest lever on your incident count.

How to Implement It

  1. You + pods: list every type of production change: code, config, infrastructure, prompts, model versions, and feature flags. Ask how each one ships today.
  2. SRE + pods: make every change type visible as a Grafana annotation, so it shows up during incidents.
  3. Pods: move the riskiest change types onto canary and rollback (see next).
  4. You: agree change freeze rules for peak events, tied to the error budget policy.

Where it lives in Azure + Grafana: Deploy pipelines (confirm which tool your pods use), the Azure Activity Log for infrastructure changes, and Grafana annotations for every change type.

What Done Looks Like

  • Every change type is listed, with how it ships.
  • All change types appear as annotations.
  • Prompt and model changes go through the same safety steps as code.

Ask this: “What changed in the last 24 hours?” · “How do we roll this back, and how long does it take?” · Design Review questions

Go deeper: SRE Book, Ch. 8: Release Engineering · Microsoft Learn: Activity Log

Canary, Feature Flags & Rollback

Canary: release to a small slice of users first. Feature flag: turn a feature on or off without a deploy. Rollback: return to the last good version quickly.

In IT Ops Terms

Like a pilot group for a patch, a kill switch, and a tested backout plan, all automated.

Why It Matters at Your Level

These are the fastest ways to stop the bleeding in an incident, and the best way to keep a bad change from ever reaching all users.

How to Implement It

  1. Pods: release to 5% first, watch error rate and latency for an agreed time, then expand. Roll back automatically if the SLI drops.
  2. Pods: put risky features, including new prompts and models, behind flags that SRE can turn off during an incident.
  3. SRE: test rollback regularly and time it.

Where it lives in Azure + Grafana: Feature flags via Azure App Configuration feature management or your pods' flag tool (confirm with your team). Canary health is checked against Application Insights SLIs. Rollback lives in the deploy pipeline.

What Done Looks Like

  • Critical services canary every release.
  • SRE can turn risky features off without a deploy.
  • Rollback is tested and takes minutes, not hours.

Ask this: “What are our options to reduce impact right now?” · “Has anyone actually tried the rollback?” · Driving to Mitigation questions

Go deeper: Workbook, Ch. 16: Canarying Releases · Microsoft Learn: Feature management

Toil

Manual, repetitive operational work that scales with traffic and leaves nothing lasting behind, like restarting the same service every week.

In IT Ops Terms

The "keep the lights on" work you know well. SRE puts a cap on it: if toil takes most of the team's time, nothing ever improves.

Why It Matters at Your Level

Toil is why central SRE teams stay stuck firefighting. Measuring it gives you the data to win headcount or push work back to the pods that cause it.

How to Implement It

  1. SRE: for two weeks, tag time roughly as project, toil, or interrupts. Rough is fine.
  2. You: set a target, such as toil below 50% of team time, and report it monthly.
  3. SRE: rank the top toil items by hours. Automate them, fix them, or hand them back to the owning pod.

Where it lives in Azure + Grafana: Time tracking in your work tracker. Many toil items show up as repeat PagerDuty incidents, which gives you hard counts.

What Done Looks Like

  • Toil percentage is measured and reported monthly.
  • The top 3 toil items each have an owner and a plan.
  • The percentage is trending down.

Ask this: “What percentage of the team's time goes to unplanned work?” · “What do you do every week that a script could do?” · Roadmap & Priorities questions

Go deeper: SRE Book, Ch. 5: Eliminating Toil · Workbook, Ch. 6: Eliminating Toil

Automation

Replacing a repeated manual task with something that runs itself, done when the time saved is clearly worth the effort and the risk.

In IT Ops Terms

Like scripting your runbooks, but done deliberately: automate the frequent, well-understood steps first, and keep a human in the loop for risky ones.

Why It Matters at Your Level

Automation is how SRE scales without adding headcount, and how you lift work off a struggling L1 NOC. It needs prioritizing, or it never happens.

How to Implement It

  1. SRE: take the top toil list and estimate hours saved per month against the effort to build.
  2. You: protect time for the top item. Fund it from the toil budget, not engineers' evenings.
  3. SRE + NOC lead: automate NOC runbook steps that are pure lookups or restarts, so L1 does less copying and more escalating.

Where it lives in Azure + Grafana: Azure-native options such as Azure Monitor alert actions (Action Groups can trigger automation), or scripts in your pipelines (confirm with your team).

What Done Looks Like

  • One automation is shipped per quarter from the top-toil list.
  • Hours saved are measured afterwards.
  • At least one NOC runbook step is automated.

Ask this: “What's a piece of toil you'd automate first, and what would it save?” · “What should we decide not to automate?” · Hiring Loop questions

Go deeper: SRE Book, Ch. 7: The Evolution of Automation at Google · Microsoft Learn: Action groups

Runbooks

Step-by-step instructions for handling a specific alert or task, written so the person on shift can follow them under pressure.

In IT Ops Terms

Your SOPs. The bar is higher for an L1 team that struggles with instructions: one action per step, exact if/then rules, and a clear point where you stop and escalate.

Why It Matters at Your Level

Unclear runbooks are the most common reason an offshore NOC "doesn't follow instructions." Fixing them improves NOC performance faster than any training.

How to Implement It

  1. SRE + NOC lead: every paging alert gets a runbook link in the alert itself (PagerDuty or Grafana).
  2. SRE: write steps as action, expected result, then if not, do this, with a STOP: page SRE point and a time limit.
  3. NOC lead: test each runbook with the least experienced analyst using teach-back, and fix any step they stumble on.
  4. You: after each incident, ask "did the runbook help?" and update it within a week.

Where it lives in Azure + Grafana: A shared wiki or doc space, linked from PagerDuty incidents and Grafana alert notifications. Screenshots come from the actual Grafana and Azure screens.

What Done Looks Like

  • Every paging alert links to a runbook.
  • Each runbook has been tested by an L1 analyst with teach-back.
  • Each has a clear escalation point with a time limit.
  • Runbooks are updated after incidents.

Ask this: “Could a brand-new analyst follow this at 3 a.m. without asking anyone?” · “At what step do you stop and escalate?” · Instructions That Stick questions

Go deeper: SRE Book, Ch. 11: Being On-Call