The Core of Incident Handling — What Matters Most When Things Break

The SRE intro series closes with the moment the error budget actually burns: incident response. The gap in incident capability is usually not technical but procedural — the difference between the organization where one talented person fixes things alone at 3 a.m., and the one where any on-call responder delivers the same quality. This post lays out the skeleton of that procedure. The user-facing explainer on outages is IT Literacy #4; this is the responder’s view.

Principle one — recovery comes before root cause #

The most common trap mid-incident is digging into “why.” The question during an incident is not “what’s the cause” but “how do we end user impact now?” Recovery often doesn’t need the cause: roll back the last deploy, isolate the bad instance, fall back to cached responses, shift traffic. There’s a reason so many runbooks begin with “what changed recently → undo it.”

Root cause is the postmortem’s job, after recovery. That requires the habit of leaving evidence behind as you go — logs, metric snapshots, a timeline. And when diagnosis itself stalls, the answer is a runbook prepared in advance with a system like the Why Your Server Is Slow series, not deriving one from scratch at 3 a.m.

Role separation — the person fixing doesn’t command #

Beyond a certain size, the most important structure in an incident is role separation.

• Incident Commander (IC) — runs the response: situation, assignments, priorities. Does not touch the keyboard. Whoever is deep in a debugger loses the overall picture.
• Operations — does the actual investigation and recovery, reporting findings and changes to the IC.
• Communications — summarizes status for stakeholders (leadership, support, customer notices when needed) on a cadence. Without this role, responders answer “how’s it going?” every five minutes and recovery slows down.

A small team can keep the principle without three people: with two, one takes IC plus comms and one operates; alone, at minimum keep a one-line status log every ten minutes as your comms channel. The point is knowing that command and communication are jobs that exist.

Severity levels and declaration — make the judgment in advance #

Don’t make someone decide “is this an incident?” at 3 a.m. Define severity levels and declaration criteria in peacetime.

Wire this to burn-rate alerts and declaration becomes natural: fast-burn alerts are SEV1–2 candidates, slow burns are SEV3 tickets. When in doubt, declare high and downgrade later — always cheaper than declaring low and escalating. Over-declaring costs a little embarrassment; under-declaring costs response time.

Postmortems — blameless, but never vague #

After recovery, write the postmortem. Formats vary; four things are non-negotiable: a timeline (detection to recovery, with times), impact (what share of users, for how long, how much budget spent), cause analysis (the direct cause and the structure that allowed it), and action items (prevention work with owners and deadlines).

“Blameless” is routinely misunderstood. It doesn’t mean nobody is accountable — it means replacing “who made the mistake” with “what made the mistake possible.” Stop at “A pushed a bad config to production” and you gain nothing. Reach “config changes had no validation step, and staging and production configs are edited on the same screen,” and the system gets fixed. Aim at people and facts start hiding; nothing can be built on hidden facts. And action items without owners and deadlines are decoration — the quarterly review of unfinished action items is part of the postmortem, not an extra.

On-call — it only works if it’s sustainable #

All of this presumes someone to respond: on-call. Where on-call collapses, the cause is load, not technology.

• Alert quality — pages that need no action (informational, auto-recovering) grind responders down fast. Reserve paging for “a human must act now”; everything else goes to tickets and dashboards. The burn-rate scheme draws this line naturally.
• A load ceiling — track pages per shift, and rule that beyond a set threshold, alert cleanup and root-cause removal outrank feature work. The same principle as part 1’s toil cap.
• Runbooks — recurring alerts get a runbook: “when this fires, look at X, do Y.” An alert without one sends people hunting for whoever knows that system — and that person becomes your single point of failure.

Summary #

• The goal mid-incident is ending user impact, not finding root cause. Reach for rollback, isolation, and fallbacks first.
• Separate command (IC), communications, and operations. When the fixer also commands, both jobs slow down.
• Define severity levels and declaration criteria in peacetime. Declare high and downgrade — always cheaper.
• Postmortems: timeline, impact, structural cause, deadlined action items — asking “what made it possible,” not “who.”
• On-call stays alive through alert quality, load ceilings, and runbooks. No procedure stands on an unsustainable rotation.

That closes the four-part SRE intro: make targets numeric with SLI/SLO, turn the speed-stability tradeoff into rules with error budgets, and protect the budget with incident procedure. Those three, cycling, are the skeleton of SRE.