SRE in Practice #4 On-Call Operations: Rotations, Escalation, Load Management

However good the alerting system is, a human ultimately receives the page. On-call is the most human part of SRE practice — and, run badly, the part most reliably guaranteed to burn people out. This part covers the design elements that make on-call a sustainable institution.

Rotations: start by counting heads #

The first question of any rotation is “how many people does this take?” The answer is bigger than people expect.

• One person covering 24/7 is not a rotation; it is a hostage situation. Accounting for vacations, sick days, and departures, five to six people per rotation is the realistic floor. Below that, on-call weeks come around too fast and burnout begins.
• Small teams below that headcount are better served by honesty than by forcing a rotation: several teams pooling services into a joint rotation, or shrinking what pages outside business hours at all.

Two basic shift structures:

• Weekly shifts (full-day): one person is on call 24 hours a day for a week. Simple, but the night-page load is heavy — suited to teams where night pages are rare.
• Day/night split or follow-the-sun: geographically distributed organizations ideally have each region cover only its own daytime. Night calls disappear entirely.

Either way, run a primary and a secondary. The secondary backs up situations where the primary cannot respond (in transit, overlapping incidents) and doubles as the training path where new members gain experience before taking primary.

Escalation: draw the map before you are lost #

The escalation policy is the advance agreement on “who asks whom for help, and when.”

1. Automatic escalation: if a page goes unacknowledged for N minutes, the paging tool escalates on its own — secondary, then team lead. People sleep through pages and lose signal; this chain must be enforced by tool configuration, not memory.
2. Judgment escalation: acknowledged, but 30 minutes with no progress, or the blast radius is growing — call in another team or the higher-level response structure. One cultural point matters here: escalating is procedure, not failure. Organizations where the vibe is “I should have held on longer alone” have longer outages.
3. Incident command: for high-impact incidents, separate the roles. The incident commander only coordinates and decides; operators only execute; a communicator only shares status. Making the person fixing the problem simultaneously write the status updates ruins both.

Handoffs: shifts change via documents #

Passing a shift with a verbal “all yours” loses the context of everything in flight. Keep handoffs short but structured:

• Ongoing incidents, their current state, and the next check-in time
• A summary of pages during the shift and how they were resolved
• Upcoming risks (major deploys, migrations, traffic events)
• If it was quiet, say “it was quiet” explicitly — no record and no problem are different things

For weekly rotations, a fixed five-minute agenda item in the team meeting is the most sustainable form.

Load management: put a ceiling on pages #

On-call collapses not from the dramatic mega-outage but from the chronic state of modest pages arriving night after night. So manage the load as a number.

• Track pages per shift. When the number persistently exceeds a threshold (say, two per 12-hour shift), treat it as a system problem, not a test of individual endurance. The cause is one of two things: poor alert quality (the false-positive cleanup from part 3) or genuinely large reliability debt (an engineering investment decision).
• Expect no project work during on-call. The output of an on-call shift is response, plus the record of problems discovered while responding. Demand a normal sprint commitment from the on-caller and the organization quietly abandons one or the other.
• Institutionalize follow-up time. Explicitly budget next-sprint time to pay down what the on-call week surfaced — inaccurate runbooks, recurring manual recoveries. Without this, the same page repeats every month.

Compensation is part of load management. In an organization where night and weekend standby is unpaid volunteering, avoiding on-call is not a character flaw but a rational choice. The form varies — on-call pay, time off in lieu — but the organization acknowledging that standby has a cost is what decides whether the institution survives.

The on-caller’s first 30 minutes: why runbooks exist #

The first half hour after a page follows a roughly fixed arc: acknowledge → assess impact (SLI dashboard) → check recent changes (deploys, config) → mitigate (rollback, scale out, shed traffic) → escalate if needed. What the on-caller needs for this arc is not deep knowledge of the entire system but per-alert runbooks and an order of diagnosis.

A runbook needs three things at minimum: what this alert means (which SLI and why), the first checks (dashboard links, the command to list recent deploys), and proven mitigations (rollback procedure, feature flags). Root-causing can wait for daylight. At 3 AM the goal is not diagnosis — it is ending user impact.

Summary #

• Rotations need five to six people as a floor; below that, joint rotations or reduced coverage are the honest answers. Primary/secondary is both backup and training path.
• Design escalation in two layers — automatic (no response) and judgment (no progress) — and build a culture where escalating is procedure, not failure.
• Fix handoffs as a short structured document. “It was quiet” is also a record.
• Cap pages per shift; persistent overruns escalate to an alert-quality or reliability-debt decision. Expect no project work during on-call and institutionalize follow-up time.
• The overnight goal is ending impact, not finding root cause — which is why runbooks are mandatory. Next part: what happens after the incident ends — postmortems.