SRE in Practice #6 Reducing Toil: Measuring Manual Work, Automating It, and an Adoption Roadmap

The final topic of the series is unglamorous but decides the team’s future: how do you stop operational manual work from swallowing engineering time? SRE calls this work toil, and left unchecked it traps teams in a spiral — the service grows, the team sinks into operations, and there is no time to automate precisely because everyone is busy doing things by hand.

Identifying toil: not “hard work” but a specific kind of work #

Toil is not a sentimental word for unpleasant work; it has identification criteria. The more of these properties a task has, the more it is toil.

• Manual: a human executes it — including a human running a script by hand.
• Repetitive: not the first time; you have lost count of how many times it has happened.
• Automatable: no essential human judgment; a machine would produce the same result.
• Tactical: creates no lasting value. When it is done, the system is in the same state as yesterday.
• Scales with growth: more users, traffic, or servers means proportionally more of it.

Disk cleanups, clicking approve on manual deploys, processing account-creation requests, running the same query every week for a report — all classic toil. By contrast, judgment during incident response, architecture design, and building automation itself are hard work but not toil. The distinction matters because only toil can be eliminated by automation.

The 50% rule: without a ceiling, it always overflows #

Google SRE popularized the operating principle: operational work (toil + on-call + tickets) must not exceed 50% of a team’s time. The other half goes to engineering that reduces toil — automation, reliability work.

The exact number matters less than the existence of a ceiling. Without one, operational work inevitably expands, because the service keeps growing and toil scales with growth. The ceiling is enforced by measurement.

• Once a quarter, have each team member roughly classify the last two weeks of their time. Precision is unnecessary; “operations: 70%” as a rough number is enough to start the conversation.
• If the team stays above 50%, that is not a personal time-management issue — it is a team investment-allocation decision: schedule an automation sprint, hand a toil-generating service back to its development team, or adjust staffing.

Automation priority: frequency × duration + risk #

With a toil inventory in hand, decide where to start. The base formula is annual time saved = frequency × time per occurrence, weighted by two factors.

1. Error risk: work where a human mistake becomes an incident (manual production-database operations, say) moves up the list even when the time saved is small — the value of automating it is incident prevention, not time.
2. Automation cost: spending three weeks to automate away ten hours a year is a loss. But cost accounting must include the hidden lines: the focus shattered by the interruption, and the work that stops when the one person who knows the procedure is on vacation.

One trap to note: automating a broken process just produces broken results faster. Before auto-processing tickets born of a nonsensical approval flow, removing the approval flow is often the correct first move. Before automating, ask once: why does this task exist at all?

The automation ladder: runbook → script → self-service #

Automation is not all-or-nothing; it climbs in stages.

1. Documentation (runbooks): turn the procedure in someone’s head into a document. The stage before automation, and the stage where the procedure’s absurdities first become visible.
2. Semi-automation (scripts): bundle the runbook’s commands into a script, with a human still deciding when to run it. For most toil, this stage alone slashes time spent.
3. Full automation: humans exit from trigger to execution. Candidates are judgment-free work — disk cleanup, certificate renewal, retrying failed jobs — always paired with safety rails (change limits, halt on anomaly).
4. Self-service: requesters handle it themselves. Instead of receiving “please create an account” tickets, provide a platform where the requester completes it through a form and auto-approval rules. This is the terminus where the toil leaves the team entirely.

If writing that first repetitive script is new territory, the Python Automation series is the toolbox for stage two.

Closing the series: a realistic adoption order #

One structure runs through all six parts: measurement creates agreement, and agreement justifies action. Without SLOs there is no error budget, and without a budget, “we need to invest in reliability now” remains the opinion of whoever talks loudest. The realistic order for introducing this into an organization:

1. Build SLOs for one service, two or three journeys (part 1). Do not announce a company-wide rollout.
2. Agree the budget policy with leadership (part 2). The hardest step and the most important.
3. Move alerting onto SLIs (part 3). This is where the team feels overnight pages drop — and where trust in the program is born.
4. Straighten out on-call and postmortems (part 4, part 5). The response-and-learning loop closes.
5. Measure toil and defend automation time (this part). The force that keeps the loop turning.

Once the loop runs on one service, other teams follow the results. Institutions spread by example, not by decree.

Summary #

• Toil is a term with criteria — manual, repetitive, automatable, tactical, growth-proportional. Not all hard work is toil.
• Cap operational work at 50% and measure roughly each quarter. Persistent overruns escalate to a team-level investment decision.
• Prioritize automation by frequency × duration weighted by error risk, and fix broken processes before automating them.
• Climb the ladder: runbook → script → full automation → self-service.
• Adoption order: SLOs → budget policy → alerting → on-call and postmortems → toil management. One service’s success story is the best propagation mechanism.