Observability & Telemetry
How you know what your system is doing: telemetry, metrics, logs, traces, the golden signals, dashboards, and quality signals for an AI product.
Monitoring vs Observability
Monitoring tells you that something is wrong. Observability lets you work out why, including for problems nobody predicted.
In IT Ops Terms
Monitoring is your classic NOC: thresholds on known things like CPU or disk. Observability is being able to ask new questions of the data during an incident without shipping new code.
Why It Matters at Your Level
Unreliable products fail in new ways. If your team can only see what was pre-configured, every new failure means guessing. That's what makes incidents long.
How to Implement It
- You: after each major incident, ask "what question couldn't we answer during it?" That's your observability gap list.
- SRE + pod: make sure requests carry an ID that follows them across services, so one slow request can be traced end to end (see Traces).
- SRE: make logs searchable by that request ID and by user-impacting fields such as region and app version.
Where it lives in Azure + Grafana: Monitoring: Azure Monitor metrics and alerts. Observability: Application Insights (requests, dependencies, exceptions, distributed traces) plus Log Analytics KQL for asking new questions.
What Done Looks Like
- During an incident, the team can split errors by region, app version, and dependency within minutes.
- One slow request can be followed end to end.
- The "questions we couldn't answer" list shrinks quarter over quarter.
Ask this: “What question couldn't we answer during the last incident?” · “Can we split the error rate by region and app version right now?” · Driving to Mitigation questions
Go deeper: SRE Book, Ch. 6: Monitoring Distributed Systems (opens in a new tab) · Microsoft Learn: Azure Monitor overview (opens in a new tab)
Telemetry
The data your systems emit about themselves: metrics, logs, and traces. Observability is built from it.
In IT Ops Terms
Like the event logs and performance counters you collected from servers, but emitted by the application itself, about every request.
Why It Matters at Your Level
Telemetry costs money (ingestion and storage) and engineering time. Too little and you're blind. Too much and the bill spikes. Deciding the balance is a leadership call.
How to Implement It
- SRE: agree a standard: every service emits request metrics, structured logs, and traces using one approach. OpenTelemetry is the open standard Application Insights supports.
- You: ask for the monthly telemetry bill from Azure Cost Management, and who is generating the most data.
- SRE + pods: set retention and sampling rules. Keep everything for errors; sample routine successful requests.
Where it lives in Azure + Grafana: Instrumentation via the Azure Monitor OpenTelemetry distro into Application Insights. Data lands in a Log Analytics workspace. Cost appears in Azure Cost Management.
What Done Looks Like
- Every production service emits metrics, logs, and traces in the agreed format.
- Telemetry cost is known monthly, with an owner.
- Sampling and retention rules are written down.
Ask this: “How does our telemetry bill grow as traffic grows?” · “Which service generates the most log data, and do we use it?” · Vendor & Tool Pitch questions
Go deeper: Microsoft Learn: OpenTelemetry in Azure Monitor (opens in a new tab) · Workbook, Ch. 4: Monitoring (opens in a new tab)
Metrics, Logs & Traces
The three kinds of telemetry. Metrics are numbers over time. Logs are detailed records of events. Traces show one request's path through every service it touched.
In IT Ops Terms
Metrics are your performance graphs. Logs are your event logs. Traces are new: like following one ticket through every queue it touched, with a timestamp at each hop.
Why It Matters at Your Level
Each answers a different question in an incident. Metrics answer "is it bad and since when?" Logs answer "what exactly failed?" Traces answer "where in the chain did the time go?" Knowing which to ask for makes you useful on the bridge.
How to Implement It
- You: in the next incident, ask "what do the metrics show, what do the logs show, and do we have a trace?" Note which of the three were missing.
- SRE: make sure traces cover the call to the model provider, since that's where AI requests often spend most of their time.
- SRE: link from Grafana panels to the matching Log Analytics query, so moving from graph to detail takes one click.
Where it lives in Azure + Grafana: Metrics: Azure Monitor metrics and Application Insights, graphed in Grafana. Logs: Log Analytics (KQL). Traces: Application Insights distributed tracing (the end-to-end transaction view).
What Done Looks Like
- All three exist for every critical journey.
- A trace shows time spent in the model provider call.
- Grafana panels link straight to the relevant logs.
Ask this: “Where does the time go in one slow request?” · “Do we have a trace for the request that failed?” · When It's Stuck questions
Go deeper: Microsoft Learn: Distributed tracing in Application Insights (opens in a new tab) · Microsoft Learn: Log Analytics overview (opens in a new tab)
The Four Golden Signals
The four things to watch for any user-facing service: latency (how slow), traffic (how much), errors (how many fail), and saturation (how full).
In IT Ops Terms
A short, standard version of your service health checks, focused on what users feel rather than on every component.
Why It Matters at Your Level
If every critical service shows these four on the top row of its dashboard, anyone, including you and the NOC, can tell healthy from unhealthy in seconds.
How to Implement It
- SRE: put the four golden signals on the top row of each critical service's Grafana dashboard. Show latency as the slowest 5% of requests, not the average.
- SRE + NOC lead: mark the normal range on each panel, so the NOC can see when a signal leaves it.
- You: use the top row for your green, yellow, or red call in the SRE Daily.
Where it lives in Azure + Grafana: Grafana dashboards using the Azure Monitor data source. Latency, traffic, and errors come from Application Insights requests. Saturation comes from Azure Monitor metrics: CPU, memory, queue depth, and provider quota usage.
What Done Looks Like
- Every critical service has the four signals on its top row.
- Latency shows the slowest 5% or 1%, not just the average.
- Normal ranges are marked, so the NOC can read them.
Ask this: “What does a normal day look like on this graph?” · “Which three graphs would you look at first at 3 a.m.?” · Dashboard Review questions
Go deeper: SRE Book, Ch. 6: Monitoring Distributed Systems (opens in a new tab)
Dashboards
Pages of graphs that help someone answer a specific question quickly. A good dashboard has a purpose and an audience. A bad one is a wall of every metric.
In IT Ops Terms
Like NOC wallboards, with the same lesson: the board that shows everything ends up helping nobody.
Why It Matters at Your Level
Your team, the NOC, and leadership need different views. One cluttered dashboard slows every incident and makes you look less in control in reviews.
How to Implement It
- SRE: create three levels: an executive view (SLOs and budget), a service view (golden signals per service), and debug views (deep detail, linked from the service view).
- SRE: give every dashboard an owner and a one-line purpose at the top. Delete any that haven't been viewed in 90 days.
- SRE: add deploy and change annotations, so changes show as markers on every graph.
Where it lives in Azure + Grafana: Azure Managed Grafana: folders per level, an owner in each dashboard description, and annotations from the deploy pipeline or Activity Log queries.
What Done Looks Like
- Exec, service, and debug levels exist and link to each other.
- Every dashboard has an owner and a purpose line.
- Deploys and changes show as annotations.
Ask this: “Who owns this dashboard, and when was it last changed?” · “Is there an annotation for the change before this spike?” · Dashboard Review questions
Go deeper: Grafana docs: Dashboard best practices (opens in a new tab) · Grafana docs: Annotations (opens in a new tab)
AI Quality Signals
Measures of whether the assistant's answers are good, not just fast and error-free. An AI product can be up and still wrong.
In IT Ops Terms
There's no direct IT Ops equivalent. The closest is a service desk satisfaction score, but measured continuously and per answer.
Why It Matters at Your Level
Quality problems from model or prompt changes often don't trip any error alert. If you're not measuring quality, customers find these incidents before you do.
How to Implement It
- Pod + product: pick one or two quality signals. Examples: thumbs-down rate, "I couldn't help with that" rate, fallback-answer rate, or scores from an automated evaluation of sampled answers.
- SRE: chart them next to the golden signals, with deploy and prompt-change annotations.
- You: agree a threshold that counts as an incident, such as "thumbs-down rate doubles for an hour," so quality problems get declared, not just discussed.
Where it lives in Azure + Grafana: Custom events and metrics sent to Application Insights by the pod's code (confirm with your team what they already log), charted in Grafana next to latency and errors.
What Done Looks Like
- At least one quality signal on the service dashboard.
- A written threshold that makes a quality drop an incident.
- Prompt and model changes appear as annotations.
Ask this: “If the assistant gave wrong answers fast, would anything turn red?” · “What changed in the prompts or model this week?” · Dashboard Review questions
Go deeper: Microsoft Learn: Application Insights overview (opens in a new tab)