SRE in Practice #5 Postmortems: Blameless Reviews and Keeping Action Items Alive
Incidents will happen again. What an organization gets to choose is not whether there are incidents, but whether it repeats the same one or meets a new one each time. The institution that decides which path you take is the postmortem. This part is about making postmortems function as an actual learning device rather than a ritual report.
Blamelessness is an information strategy, not kindness #
The first principle of postmortems is blamelessness: the individuals involved in an incident are not named and punished. Read that as “let’s be nice” and you miss the point. The reason is coldly practical.
In an organization that blames, information disappears. If the person who mistyped the command gets disciplined, then in the next incident that person does not say what they know. Timelines go inaccurate, root-cause analysis skims the surface, and the same incident happens again. Blamelessness is the price paid for accurate information.
There is one more logical foundation: that a human error was possible is itself a defect of the system. Stop at “A ran the wrong command in production” and the action item becomes “A will be careful” — which is not prevention. The right questions continue: why was that command runnable in production? Why was there no confirmation step? Why was it hard to undo? Repeating “why” until the answers reach system improvements — permission separation, confirmation prompts, automated rollback — is the method of root-cause analysis.
When to write one: fix the triggers in advance #
Write a postmortem for every issue and the institution becomes a formality; have no criteria and big incidents slip past quietly. Codify the triggers. Common ones:
- Any incident whose user impact consumed more than a set share of the SLO budget (say, 10% of the 28-day budget)
- Every event involving data loss or corruption
- Any incident where on-call escalation reached the higher response structure
- Regardless of impact, any case where the response exposed defects in tools or procedures — including near misses
That last item matters. A near miss with no actual damage is the same lesson at the cheapest possible tuition.
The template: the timeline is the spine #
The standard structure of a postmortem document:
| Section | Contents |
|---|---|
| Summary | What happened, for how long, with what impact — three sentences at most |
| Impact | User impact quantified: failed requests, affected users, error budget consumed |
| Timeline | Chronological record from detection through response, mitigation, and resolution — including decisions and what was known at the time |
| Root-cause analysis | Direct cause and contributing factors; “why” repeated to the system level |
| What went well | Was detection fast? Did the runbook hold up? Record what to keep, too |
| Action items | Improvements with owners and due dates |
The craft of the timeline is separating what was known then from what is known now. Write “22:14 — error-rate spike seen on the dashboard; the connection to the deploy was not yet known at this point,” and later readers move from the hindsight blame of “why didn’t they just roll back?” to the productive question “what would it have taken to see that connection at that moment?”
Also connect the impact section to the error budget. The sentence “this incident consumed 40% of the budget” explains, in numbers, why the follow-up work outranks the roadmap.
Action items: where postmortems live or die #
The death of a postmortem program follows a fixed path: the review is well written, the action items sink to the bottom of the backlog, the same incident recurs six months later, and the old document turns out to contain exactly the unfinished prevention item. The countermeasures are simple but need enforcement.
- Every action item gets one owner and a due date. “The team will look into it” is not an owner.
- Split priorities. Items that directly prevent recurrence (P0) versus nice-to-haves (P1 and below); P0 goes into the sprint immediately.
- Review completion rates periodically. Just putting “last quarter’s postmortem action-item completion rate” on the monthly ops review agenda cuts abandonment dramatically.
- Do not stack action items without limit. Thirty action items from one incident means none will happen. Compress to the few that directly prevent recurrence.
Sharing: a document is only an asset when read #
A finished postmortem does not end its life in the team folder. Collect them in a repository the whole organization can read, and walk through the big ones together in review meetings. The practical payoff of sharing is other teams discovering the same weakness in their own systems from your incident. Some organizations hand new hires a few landmark postmortems as onboarding material — no document shows a system’s real weak points and response culture better.
Summary #
- Blamelessness is the strategy for getting accurate information. Where blame lives, timelines start lying.
- Never stop the analysis at human error. Repeat “why” until you reach the system defect that made the error possible.
- Codify postmortem triggers in advance: budget consumption, data loss, near misses.
- Write timelines from what was known at the time, and quantify impact in error budget.
- Success is decided in action-item tracking: enforce owners, due dates, priorities, and review completion rates.
- Next, the series finale: measuring repetitive manual work — toil — and paying it down with automation.