Capacity & Readiness
Being ready for the load and the launches that are coming: capacity and load testing, production readiness reviews, dependencies, and cost per request.
Capacity Planning & Load Testing
Knowing how much traffic the system can handle, what breaks first, and making sure there's room for the load that's coming.
In IT Ops Terms
Classic capacity planning, with two cloud twists. Limits are often quotas (like model-provider tokens per minute) rather than hardware. And testing beats estimating.
Why It Matters at Your Level
Launches, campaigns, and new device rollouts arrive on business timelines. Without tested limits, you find out at peak, in front of customers.
How to Implement It
- SRE + pods: list the limits that matter: compute scaling limits, database limits, and especially model provider quotas.
- SRE: load test critical journeys at 2–3× today's peak before major launches, and record the first thing that breaks.
- You: get the launch calendar from product monthly, so load tests happen before launches, not after.
- SRE: alert when usage passes 80% of any hard quota.
Where it lives in Azure + Grafana: Azure Load Testing for load tests, Azure Monitor metrics for saturation, and provider quotas (for Azure OpenAI, tokens and requests per minute) tracked on a Grafana panel.
What Done Looks Like
- Known limits are listed with current headroom.
- A load test at 2× or more of peak ran before the last major launch.
- Quota alerts fire at 80%.
Ask this: “When did we last load test this, and at what multiple of peak?” · “What's the first thing to break at 10× traffic?” · Performance Engineering questions
Go deeper: SRE Book, Ch. 21: Handling Overload (opens in a new tab) · Workbook, Ch. 11: Managing Load (opens in a new tab) · Microsoft Learn: Azure Load Testing (opens in a new tab) · Microsoft Learn: Azure OpenAI quotas and limits (opens in a new tab)
Production Readiness Review (PRR)
A checklist review before a new service or major feature goes live, to confirm it can be run safely: monitored, alerted, documented, owned, and reversible.
In IT Ops Terms
Like an operational acceptance or service transition checklist, focused on what on-call will need at 3 a.m.
Why It Matters at Your Level
As central SRE, you inherit whatever gets launched. A PRR is your one chance to say "not yet" before the pager lands on your team and the NOC.
How to Implement It
- You: agree a short PRR checklist: SLIs and SLO defined, golden-signal dashboard, paging alerts with runbooks, a PagerDuty service and owning on-call, a tested rollback, a load test, and known dependencies.
- You + pod managers: agree that launches of new critical services need a PRR sign-off, and that exceptions need your boss's approval.
- SRE: run the review with the pod 2 weeks before launch. Track gaps as launch blockers or dated follow-ups.
Where it lives in Azure + Grafana: A checklist document. Evidence links point to Grafana dashboards, PagerDuty services, runbooks, and Azure Load Testing results.
What Done Looks Like
- A written PRR checklist, agreed with pod leadership.
- Every new critical launch had a PRR.
- Gaps were either fixed or accepted in writing by someone accountable.
Ask this: “Who gets paged for this, and can they actually do anything about it?” · “How will we know it's working in production?” · Design Review questions
Go deeper: SRE Book, Appendix E: Launch Coordination Checklist (opens in a new tab) · SRE Book, Ch. 32: The Evolving SRE Engagement Model (opens in a new tab) · Workbook, Ch. 18: SRE Engagement Model (opens in a new tab)
Dependency Management
Knowing what your service relies on but doesn't control (the model provider, identity, third-party APIs), and what happens when each one is slow or down.
In IT Ops Terms
Vendor and supplier management, which is your home turf, applied to runtime dependencies: contracts, status, contacts, and a fallback for each.
Why It Matters at Your Level
For an AI assistant, the model provider is often the biggest single point of failure. "It's a dependency issue" should never be a surprise.
How to Implement It
- SRE + pods: list the critical dependencies for each journey, with an owner, a support contact, and the status page for each.
- SRE: track success rate and latency for calls to each dependency.
- Pods: design a fallback for the top dependency, such as a backup model or a degraded but helpful response. Use timeouts and retries with backoff.
- You: review the dependency list quarterly, including contract limits and quotas.
Where it lives in Azure + Grafana: Application Insights dependency tracking (success rate and duration per target), shown in Grafana. Azure service health appears in Azure Monitor.
What Done Looks Like
- A dependency list with owners and contacts.
- A panel for the success rate and latency of each critical dependency.
- A tested fallback for the model provider.
Ask this: “What does this depend on that we don't control?” · “Is the model provider healthy, and how would we know?” · Driving to Mitigation questions
Go deeper: SRE Book, Ch. 22: Addressing Cascading Failures (opens in a new tab) · Microsoft Learn: Reliability in the Well-Architected Framework (opens in a new tab)
Cost per Request
What it costs to serve one user request, or 1,000 of them. For AI products this is driven heavily by model and token usage.
In IT Ops Terms
Unit cost, like cost per ticket or per seat, applied to cloud and AI spend.
Why It Matters at Your Level
Speed, reliability, and cost trade against each other. Showing cost per request alongside the SLOs lets you argue for (or against) changes with the number finance cares about.
How to Implement It
- You + finance: get the monthly cost for the assistant's backend and model usage from Azure Cost Management.
- SRE: divide by monthly request volume from Application Insights. Rough is fine to start.
- You: track it monthly on the Performance Engineering scorecard, and flag changes that move it, such as longer prompts or a new model.
Where it lives in Azure + Grafana: Azure Cost Management for spend, Application Insights for volume, and a monthly number on the Grafana or scorecard view.
What Done Looks Like
- A monthly cost-per-request number exists, with a trend.
- It's on the Performance Engineering scorecard.
- Major changes are checked for their effect on cost.
Ask this: “What does each request cost us, and is it going up?” · “How does pricing grow as our data grows?” · Performance Engineering questions
Go deeper: Microsoft Learn: Cost Management overview (opens in a new tab)