SRE in Practice #3 Alert Design: Symptom-Based Alerting and Multiwindow Burn Rates
Alerting has one purpose: notify a person who can intervene, about a problem that needs intervention, at the moment intervention is needed. Yet in most organizations the alert channel has become a waterfall of notifications nobody reads. This part covers implementing part 2’s burn rates as alerts, and the operations that keep an alerting system trustworthy.
Alert on symptoms, not causes #
Traditional alerts hang off candidate causes: CPU over 80%, disk at 90%, process restarted. This approach fails in both directions.
- False positives: if CPU is at 90% but every user request is being served normally, there is no reason to wake anyone. It might just be a batch job.
- False negatives: if CPU, memory, and disk are all healthy but user requests are failing because of an upstream dependency, cause-based alerts stay silent.
Symptom-based alerts hang off what users experience — the SLIs. Is the error rate over threshold? Is the latency SLI collapsing? Whatever the cause, it fires when users are affected and stays quiet when they are not. This is where part 1’s insistence on user-journey-based SLIs pays off. Cause metrics like CPU move down a level: not alert triggers, but the diagnostic dashboard you open after being alerted.
Pages and tickets: a contract on response time #
Not every notification carries the same urgency. Split into two grades.
| Grade | Meaning | Criterion |
|---|---|---|
| Page | Wake a human now | Left alone, the SLO takes meaningful damage within hours |
| Ticket | Handle in business hours | The problem is real but there are days of slack |
The test is a single question: “receiving this at 3 AM, is there something I can and must do right now?” If not, it is not a page. Let pages that fail this test accumulate, and the response to the pages that matter goes dull.
Multiwindow burn-rate alerts: catching fast and slow burns alike #
The naive implementation of alerting on an SLI is “alert when the 5-minute error rate exceeds X%.” The short window makes it fire on momentary spikes, while low-grade problems that persist for days slip past. The proven fix is the multiwindow burn-rate alert.
The construction, for a 28-day window and a 99.9% SLO, uses this widely adopted combination:
| Purpose | Burn rate | Window | Meaning | Response |
|---|---|---|---|---|
| Fast-burn detection | 14.4x | 1 hour | ~2% of the 28-day budget consumed in 1 hour | Page |
| Mid-burn detection | 6x | 6 hours | ~5% of the budget in 6 hours | Page |
| Slow-burn detection | 1x | 3 days | at this speed the budget dies inside the window | Ticket |
Then combine each with a short confirmation window — for example, fire only when both the 1-hour window and the last 5 minutes show 14.4x. That kills the smoldering-ember alerts that arrive late because a long window’s average is still elevated after the problem has already ended.
The virtue of this construction is that alert severity scales with the size of user impact. A big problem burning budget fast pages within minutes; a slow leak stacks up as tickets for daytime work. Nothing significant escapes, and overnight pages stay minimal.
Alert fatigue: what you do not measure does not shrink #
Alerting systems fail in maintenance, not construction. Numbness creeps in slowly, so look at the numbers on a schedule.
- Weekly alert review: take every page from the past week and sort into three bins. Action was needed (healthy); no action was needed (false positive — tune it); user impact was discovered through some other channel with no alert (false negative — extend coverage).
- Dispose of false positives on the spot. Adjust the threshold, demote to ticket, or delete. The pile of “leave it for now” is exactly what the notification waterfall is made of.
- Track page counts as a metric in themselves. When pages per shift stay above a set line, that is not an individual on-call problem — it is a reliability-debt or alert-quality problem for the system. This number returns in the next part as the on-call load ceiling.
One more rule: every alert carries a runbook link. Making the person paged at 3 AM figure out “what does this alert mean and what do I check in the first 30 minutes” without a document is a design failure. If the first diagnostic step feels foggy, ordering documents like When a REST API Is Slow: The Order of Diagnosis are raw material for runbooks.
Summary #
- Alert on symptoms (SLIs), not causes (CPU). Cause metrics belong on the diagnostic dashboard.
- The page criterion is “is there something to get up and do right now?” Otherwise it is a ticket.
- Multiwindow burn-rate alerts (14.4x/1h and 6x/6h paging, 1x/3d ticketing) are the proven construction that cuts false positives and false negatives together.
- Run a weekly alert review, dispose of false positives immediately, and manage pages-per-shift as a metric.
- Every alert gets a runbook. Next part: the people receiving those pages — on-call operations.