Opscraft
SRE & incident management · Lesson 3 of 7 · 4 min read

Incident management

Severity levels

Define them in advance, with examples, so nobody debates at 3 a.m.:

Sev Definition Response
SEV1 Critical customer impact: outage of a core flow, data loss, security breach All hands, incident commander, status page, exec updates every 30 min
SEV2 Significant degradation or partial outage On-call + owners, IC, regular updates
SEV3 Minor impact, workaround exists Business hours
SEV4 No customer impact Ticket
Enlarge diagramSEV1core flow down, dataloss, breach: all handsSEV2major degradation:on-call + ownersSEV3minor, workaroundexists: business hoursSEV4no customer impact:ticket
Figure 1. Severity levels decided in advance, so nobody argues at 3 a.m.
Swipe sideways to see all of it, or tap to open it full screen.

Roles (ICS-style)

  • Incident Commander (IC) — coordinates, decides, delegates; does not debug hands-on in big incidents.
  • Operations / tech lead(s) — investigate and mitigate.
  • Communications lead — status page, stakeholders, customer support.
  • Scribe — timeline of actions, decisions and timestamps.
  • Subject matter experts paged as needed.
Enlarge diagramIncidentCommandercoordinates, decidesOps / tech leadinvestigate +mitigateComms leadstatus page,stakeholdersScribetimeline ofdecisionsExpertspaged as needed
Figure 2. Incident roles. The incident commander coordinates and decides; others investigate, communicate and record.
Swipe sideways to see all of it, or tap to open it full screen.

The incident lifecycle

  1. Detect — alert, customer report, synthetic check.
  2. Triage — confirm impact and severity; declare the incident early (it's cheap to downgrade).
  3. Mobilise — incident channel (#inc-2026-10-27-checkout), bridge call, assign roles.
  4. Mitigate first, diagnose later — restore service: roll back the last deploy, fail over, scale up, disable a feature flag, shed load, block bad traffic. Root cause can wait.
  5. Communicate — regular updates at a set cadence (what's affected, what we're doing, next update time); internal and external.
  6. Resolve — confirm with metrics/SLIs that users are fine; monitor for recurrence.
  7. Postmortem — within days, blameless, with tracked actions.
Enlarge diagramDetectalert, customer, syntheticTriageimpact, severity; declare earlyMobilisechannel, bridge, rolesMitigateroll back, fail over, flag offResolveSLIs back to normalPostmortemblameless, owned actions123456
Figure 3. The incident lifecycle. Mitigate first; find the root cause after users are fine.
Swipe sideways to see all of it, or tap to open it full screen.

"What changed?" is the most productive first question: deploys, config/feature-flag changes, infrastructure changes, traffic shifts, dependency incidents, certificate expiry, quota limits. Deployment annotations on dashboards make this fast.

Metrics for incident response

  • MTTD (time to detect), MTTA (to acknowledge), MTTM (to mitigate), MTTR (to restore/resolve), MTBF (between failures).
  • Use them as trends for improvement, not as targets that encourage gaming (e.g. closing incidents early).
Enlarge diagramstartsdetectedackedmitigatedresolvedMTTDMTTAMTTMMTTR
Figure 4. The time metrics of an incident.
Swipe sideways to see all of it, or tap to open it full screen.