Opscraft
Observability: metrics, logs, traces · Lesson 6 of 7 · 3 min read

Alerting and dashboards that work

Alerting principles

  • Page on symptoms, not causes: users seeing errors/latency (SLO burn), not "CPU 85%". Cause-based alerts become tickets or dashboard signals.
  • Every page must be actionable, urgent and have a runbook link and dashboard; otherwise it's a ticket or nothing.
  • Use for: durations and SLO burn-rate alerts (see the SRE topic) to avoid flapping.
  • Severity routing: page (wake someone), ticket (business hours), info (dashboard only).
  • Track alert quality: pages per week per on-call, % actionable, time to acknowledge; delete or fix noisy alerts in a regular review.
  • Alert on absence: absent(up{job="orders"}), dead-man's-switch (a "Watchdog" alert that always fires to an external service — silence means the alerting pipeline is broken).
Enlarge diagramPage someoneTicket or dashboardError budget burning fastCheckout latency p99 > SLOHas a runbook and a dashboardCPU 85% on one nodeA pod restarted onceDisk 70% full
Figure 1. Page on what users feel; everything else is a ticket or a dashboard.
Swipe sideways to see all of it, or tap to open it full screen.

Dashboards

  • Top-down hierarchy: service overview (RED + SLO) → service detail → dependencies → infrastructure (USE).
  • Grafana practices: variables (cluster, namespace, service), consistent units, annotations for deploys, dashboards as code (Grafonnet/JSON in Git, provisioned via sidecar ConfigMaps or Terraform), links between metrics → logs (Loki) → traces (Tempo).
Enlarge diagramService overviewRED + SLO status for every serviceService detailper route, per pod, deploy annotationsDependenciesdatabases, queues, external APIsInfrastructurenodes, USE metrics
Figure 2. A dashboard hierarchy that answers "is it broken?" first and "why?" second.
Swipe sideways to see all of it, or tap to open it full screen.

Cost control

Observability bills can rival compute. Controls: drop unused metrics at scrape (metric_relabel_configs with action: drop), reduce label cardinality, recording rules + shorter raw retention, log sampling and level hygiene, exclude health-check logs, retention tiers, trace sampling, per-team usage visibility.