Error Budgets — The Concept and Math of a Failure Allowance

In the previous post we set an SLO below 100%. The moment you do, something interesting appears: at a 99.9% target, 0.1% worth of failure is allowed — and giving that allowance a name yields the error budget. “Budget” is not a metaphor but the operating model itself: there’s a balance, it gets spent, and when it’s gone, spending gets controlled.

The size of the budget — subtraction and multiplication #

Error budget = 1 − SLO. Multiply by the window and it becomes something you can hold.

Keep both conversions handy. Time conveys the weight of an incident instantly (“that 37-minute outage spent 86% of this month’s budget”); request counts let you price partial outages — an error rate of 5% for two hours spends the same as a 100% outage for 6 minutes, since consumption is error rate × duration.

What the budget changes ① — the conflict becomes a rule #

The budget’s first use is the one previewed in part 1: converting the speed-vs-stability fight from meetings into rules.

• While budget remains, ship, experiment, take risk. Reliability is running above target, and surplus reliability is rationally converted into speed.
• When the budget is spent, the pre-agreed policy fires: typically a feature-deploy freeze (hotfixes and reliability work only) and reliability work jumping the priority queue.

The key is agreeing the policy while the budget is still healthy. Debate “freeze or not” after it’s gone and you’re back to politics. With a written policy, the freeze isn’t someone’s judgment call — it’s the result of a balance inquiry. And the rule must not be a punishment device for developers: while the budget is fat, it’s explicit permission to deploy more boldly. Run it both ways or the practice collapses.

What the budget changes ② — alerts become burn-rate alerts #

Traditional alerting is a threshold (“page on error rate > 1%”). It fails in both directions: it wakes humans for short spikes (a rounding error in budget terms), and it misses the slow bleed running just under the threshold for days.

Error budgets replace this with burn rate — the speed of spending. Burn rate 1 means “at this pace, the budget hits exactly zero as the window ends.” Burn rate 14.4 torches a 30-day budget in two days. The working standard is two tiers:

• Fast burn (say, burn rate ≥ 14 over a 1-hour window) — page immediately. You’re bleeding out right now.
• Slow burn (say, burn rate ≥ 3 over a 6-hour window) — ticket, business-hours response. Left alone, the budget dries up within days.

The structural advantage: the alert’s meaning is self-evident. “Error rate 1%” requires interpretation; “at this pace, the budget is gone in two days” is the interpretation.

The realities you’ll run into #

• Dependencies spend your budget — cloud and third-party outages land on your SLI as your failures. That’s correct in principle: users don’t parse root causes. If a dependency repeatedly burns your budget, read it not as “not our fault” but as a signal you owe defenses — retries, fallbacks, multi-region.
• Planned maintenance — user-affecting planned work spends from the same budget, so schedule maintenance when the balance is healthy. One balance, all spenders.
• Window styles — calendar 30 days (monthly reset) is easy to reason about; rolling 30 days avoids the distortion of a suddenly full budget after each reset. Many teams start calendar, mature into rolling.
• Chronically unspent budget is also a signal — if most of the budget survives every month, the target is looser than your actual capability, and you may be shipping more conservatively than you need to. The budget exists to be spent.

Getting started — small, one service #

With an SLO in hand, the runway is short: convert one core service’s SLO into a budget, put balance and burn rate on a dashboard, wire the two-tier burn alerts, and agree a one-page exhaustion policy with the affected teams. The first month’s goal isn’t perfect rules — it’s the habit of looking at the balance. When incidents start being retrospected in budget language (next post), the practice has taken hold.

Summary #

• Error budget = 1 − SLO. Convert to both time and request counts, and incident weight and partial-outage spending become instant arithmetic.
• Remaining budget buys speed; exhaustion triggers the pre-agreed policy (freeze, reliability first). Agree the policy before it’s needed.
• Alert on burn rate, not thresholds: fast burn pages, slow burn tickets.
• Dependency failures spend your budget too — repeated burns are a signal to build defenses, not an excuse.
• A budget that’s never spent means the target is too loose. Budgets exist to be spent.

Next up: the moments that actually burn the budget — incident handling.