Opscraft
Observability: metrics, logs, traces · Lesson 1 of 7 · 3 min read

Observability foundations

"How do you monitor it?" follows every design question in a Senior DevOps interview. You need to explain metrics, logs and traces and how they connect; run Prometheus and Grafana on Kubernetes (kube-prometheus-stack, PromQL, recording and alerting rules, Alertmanager routing, scaling with Thanos/Mimir/AMP); ship logs with Fluent Bit to Loki/OpenSearch/CloudWatch; instrument with OpenTelemetry; use CloudWatch well on AWS; and design alerts that page for real user impact instead of noise.

Monitoring vs observability

Monitoring answers known questions ("is CPU above 80%?"). Observability is the ability to answer new questions about system behaviour from its outputs — essential for distributed systems where failures are novel. In practice you need both: dashboards and alerts for known failure modes, and rich, correlated telemetry to debug unknown ones.

The three signals (plus one)

Signal What Strength Cost profile
Metrics Numeric time series with labels Cheap aggregation, alerting, trends Cost grows with cardinality (unique label combinations)
Logs Timestamped event records Detail, context, audit Cost grows with volume
Traces A request's path across services as spans Latency breakdown, dependencies Cost grows with request volume — sample
Profiles CPU/memory profiles over time (continuous profiling: Pyroscope, Parca) Why code is slow Moderate
Enlarge diagramMetricslatency p99 alertfiresTracesexemplar → theslow spanLogssame trace_id:the errorProfileswhich code isslow
Figure 1. The signals and how they connect: an alert on a metric leads to a trace of a slow request, which leads to the logs for that exact request.
Swipe sideways to see all of it, or tap to open it full screen.

Correlation is the goal: a latency alert (metric) → an exemplar trace ID on the histogram → the slow span → logs with the same trace_id.

Methods: what to measure

  • Four golden signals (Google SRE): latency, traffic, errors, saturation.
  • RED for request-driven services: Rate, Errors, Duration.
  • USE for resources: Utilisation, Saturation, Errors (CPU, memory, disk, network, connection pools).
Enlarge diagramRED: SERVICESUSE: RESOURCESRate: requests per secondErrors: failed requests (5xx)Duration: latency percentilesUtilisation: how busySaturation: how much is queuedErrors: error events
Figure 2. What to measure: RED for every service that handles requests, USE for every resource underneath.
Swipe sideways to see all of it, or tap to open it full screen.

Measure latency with percentiles (p50/p95/p99) from histograms — averages hide tail latency. And measure error rates distinguishing client errors (4xx, often not your fault) from server errors (5xx).

Enlarge diagramaverageslowestEach bar is one request's latency. Alert on p95 / p99, not the mean.
Figure 3. Why averages lie: most requests are fast, a few are very slow. The average looks fine while p99 users wait seconds.
Swipe sideways to see all of it, or tap to open it full screen.