Postmortems and learning
Blameless postmortems
Assume people acted reasonably with the information they had; focus on systems, processes and conditions that allowed the failure. Blame leads to hidden mistakes and less learning.
Template:
- Summary — what happened, impact (users, duration, revenue, SLO budget consumed), severity.
- Timeline — detection, key decisions, mitigation, resolution (UTC/IST consistently).
- Root cause and contributing factors — use "5 whys" or causal analysis; usually several factors (a bad config + missing validation + alert gap + slow rollback).
- What went well / what went poorly / where we got lucky.
- Action items — specific, owned, prioritised, tracked to completion (prevent, detect faster, mitigate faster). E.g. "Add schema validation for feature flag config in CI (owner, due date)".
- Lessons shared broadly — postmortem reviews, a searchable library.
Avoid action items like "be more careful" — they're not actions. Prefer mechanisms: automation, guardrails, tests, alerts.
Example root-cause chain (5 whys)
Checkout failed for 25 minutes → why? pods couldn't connect to the database → why? connection pool exhausted → why? a new release opened connections per request without pooling → why? the change wasn't load-tested and the canary had too little traffic to show it → why? canary analysis only checked error rate, not DB connection saturation. Actions: pool configuration review, connection-saturation metrics in canary analysis, load test stage for DB-touching changes, RDS Proxy.