SRE & incident management · Lesson 5 of 7 · 3 min read
On-call, toil and operational excellence
Sustainable on-call
- Rotation of at least 6–8 engineers per primary rotation; primary + secondary; follow-the-sun where teams are distributed.
- Clear escalation policies (PagerDuty/Opsgenie/incident.io): ack within 5–15 minutes or escalate.
- Handoffs with notes on open issues; on-call reviews weekly: pages, noise, toil.
- Target: few pages per shift (Google suggests at most ~2 incidents per 12-hour shift); every page actionable with a runbook.
- Compensation/time off after heavy nights; protect focus time — on-call engineers handle interrupts, not roadmap work.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Runbooks
Short, actionable, linked from the alert: what this alert means, impact, dashboards, first diagnostic commands, mitigation steps (with exact commands), escalation contacts. Keep them in Git/wiki; automate steps that are always the same (runbook automation, SSM Automation documents, ChatOps).
Toil
Toil = manual, repetitive, automatable, tactical work that scales linearly with service size and has no enduring value (manual certificate renewals, hand-run deploys, resizing disks by ticket). SRE guidance: keep toil under ~50% of time; measure it, and invest in automation that removes it. Good interview story material: "I automated X, which removed N hours/week of toil."
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.