Opscraft
SRE & incident management · Lesson 5 of 7 · 3 min read

On-call, toil and operational excellence

Sustainable on-call

  • Rotation of at least 6–8 engineers per primary rotation; primary + secondary; follow-the-sun where teams are distributed.
  • Clear escalation policies (PagerDuty/Opsgenie/incident.io): ack within 5–15 minutes or escalate.
  • Handoffs with notes on open issues; on-call reviews weekly: pages, noise, toil.
  • Target: few pages per shift (Google suggests at most ~2 incidents per 12-hour shift); every page actionable with a runbook.
  • Compensation/time off after heavy nights; protect focus time — on-call engineers handle interrupts, not roadmap work.
Enlarge diagram6–8 peopleper primary rotationPrimary + secondaryescalate if no ack≤ 2 incidents / shiftevery page actionableRunbook per alertlinked from the pageWeekly reviewnoise and toilRest after bad nightsprotect the people
Figure 1. What sustainable on-call looks like.
Swipe sideways to see all of it, or tap to open it full screen.

Runbooks

Short, actionable, linked from the alert: what this alert means, impact, dashboards, first diagnostic commands, mitigation steps (with exact commands), escalation contacts. Keep them in Git/wiki; automate steps that are always the same (runbook automation, SSM Automation documents, ChatOps).

Toil

Toil = manual, repetitive, automatable, tactical work that scales linearly with service size and has no enduring value (manual certificate renewals, hand-run deploys, resizing disks by ticket). SRE guidance: keep toil under ~50% of time; measure it, and invest in automation that removes it. Good interview story material: "I automated X, which removed N hours/week of toil."

Enlarge diagramToilEngineering it awayRenewing certificates by handRunning deploys from a laptopResizing disks from ticketscert-manager renews themCI/CD + GitOps deploysAlerts + auto-expansion
Figure 2. Toil is manual, repetitive work that grows with the service. Measure it, cap it, and automate it away.
Swipe sideways to see all of it, or tap to open it full screen.