Observability & reliability
Observability: metrics, logs, traces
"How do you monitor it?" follows every design question in a Senior DevOps interview. You need to explain metrics, logs and traces and how they connect; run Prometheus and Grafana on Kubernetes (kube-prometheus-stack, PromQL, recording and alerting rules, Alertmanager routing, scaling with Thanos/Mimir/AMP); ship logs with Fluent Bit to Loki/OpenSearch/CloudWatch; instrument with OpenTelemetry; use CloudWatch well on AWS; and design alerts that page for real user impact instead of noise.
After this topic you can
- Instrument services with metrics, logs and traces
- Build useful dashboards and alerts with Prometheus and Grafana
- Find the cause of a slowdown across services
Lessons
-
1Observability foundations
- Monitoring vs observability
- The three signals (plus one)
- Methods: what to measure
-
2Prometheus in depth
- Architecture
- Metric types
- Labels and cardinality
- PromQL essentials
- Kubernetes setup: kube-prometheus-stack
- Alertmanager
- Scaling and long-term storage
-
3Logging in depth
- Principles
- Kubernetes log pipeline
- Backends compared
-
4Distributed tracing and OpenTelemetry
- Concepts
- OpenTelemetry (OTel)
- 5AWS-native observability
-
6Alerting and dashboards that work
- Alerting principles
- Dashboards
- Cost control
-
7Interview questions and model answers
- "How do you monitor a microservice on Kubernetes?"
- "Counter vs gauge vs histogram?"
- "What is high cardinality and why does it matter?"
- "Metrics aren't showing up for a new service. Debug."
- "How do you reduce alert fatigue?"
- "How would you scale Prometheus for 20 clusters with one year of retention?"
- "Explain head-based vs tail-based sampling."
- "Loki vs OpenSearch for logs?"