Operating Production Systems in the AI & Cloud Era

← the whole book on one page

Chapter 3: Automation and Its Ironies — Toil, Runbooks, and the Human Role

Every machine that takes a task away from us leaves a harder one behind.

Every operations team keeps a list of chores it would rather hand to a machine: the certificate renewed by hand, the cache warmed before a busy evening, the replica nudged back into line after a bad night. The instinct behind that list is sound. Repetitive work done by hand at 3 a.m. is slow, it invites mistakes from people who are tired, and over the months it quietly wears down the team that carries it, because nobody joined an engineering team to type the same commands every week. So we automate, and much of the time we are right to.

Yet every field that has taken automation seriously, from chemical plants to flight decks, has learned that the story is more complicated than “automate the chore and be free of it.” Automation does free people from the chore. It also changes what the people who remain are asked to do, and what they are still able to do when the machine cannot cope. That is the question of this chapter: what automation actually does to the work it touches, and what it leaves for the humans still standing beside it.

The clearest early answer came in 1983, when the British psychologist Lisanne Bainbridge published a short paper in the journal Automatica called “Ironies of Automation.” Its argument fits in a sentence. When we automate a process we do not remove the human from it; we change the human’s job, usually into a harder one, and in ways that erode the very skills the new job demands. The irony lies in what the operator is kept on for. The operator stays for the moments the automation cannot handle, which are by their nature the rare and difficult ones, while the automation has taken away the everyday practice that would have prepared the operator for them. Bainbridge was writing about industrial control rooms and aircraft, yet four decades later the paper reads as if it had been written about a platform team with an on-call rotation. Section 3.3 works through her argument properly. For now it is enough to carry a suspicion into the chapter: every automated chore leaves some work behind, and that leftover work tends to be harder than the chore was.

Encore has automated a great deal, and mostly to its benefit. Its preparations for an on-sale, once worked through by hand from a wiki page, now run from a single button. Its platform upgrades the Kubernetes nodes beneath every service without anyone touching them. Its seat ledger fails over from one leader to another so reliably that hardly anyone in the company still remembers how to do it by hand. Each of these is a genuine success, and each is also one of Bainbridge’s ironies waiting for its moment. This chapter follows Encore’s automation in order: from the chores that justified it (Section 3.1), through the way a runbook grows into a script and then into a self-service tool (Section 3.2), to the ironies themselves (Section 3.3), and on to the practical question of how much a machine should be allowed to do alone (Section 3.4). It ends with the newest machine Encore has added, an operations agent that does more than follow steps: it investigates and acts (Section 3.5). Chapter 1’s map of Encore shows where each of these pieces sits.

3.1 Toil and the Case for Automation

What toil is

Before we can say what automation does to operational work, we need a name for the work it is meant to replace. Google’s site reliability engineering (SRE) book (2016) gave the field its working definition. Toil is work tied to running a production service that is manual, repetitive, automatable, tactical rather than strategic, devoid of enduring value, and that grows linearly as the service grows. The definition is a conjunction, and each adjective narrows it. Manual means a person does it. Repetitive means it comes back, again and again, in much the same form. Automatable means a machine could do it just as well, because the task follows a pattern rather than calling for judgment. Tactical means it reacts to what is in front of us today instead of moving the system somewhere better. Devoid of enduring value means that when it is finished, the service is no better than it was before; we have only kept it where it was. A task done once is therefore not toil, however tedious it feels, and a task that needs fresh judgment each time is not toil, however often it comes round. Toil is the work that a capable engineer could do in their sleep, and sometimes, at 3 a.m., very nearly does.

Of the six properties, the last matters most, because it is the one that turns an annoyance into a strategic problem. If a chore grows in step with the service, so that twice the customers or twice the events mean twice the hours, then a team whose work is mostly toil can grow only by hiring as fast as the business grows. Such a team never gets ahead; every gain in the business arrives already carrying its share of hand work. Google’s SRE organization answered this by setting itself the goal of keeping toil below 50% of each engineer’s time, so that at least half of every week went to engineering work, and in particular to the work that removes toil. It is worth being precise about the strength of that claim. Google stated the 50% figure as a goal, not as an achievement, and its own quarterly surveys found the average nearer a third. The SRE book also allows that toil is fine in small doses: a little routine work is predictable, even restful, and it keeps people in touch with how the system actually behaves. That last point is not a throwaway. Section 3.3 will show what removing the last of the toil can cost the people who used to do it.

Encore’s toil, measured

A definition is only useful if we can hold real work up against it, so Encore measured. The Reliability team, the ten site reliability engineers in Encore’s platform group, asked on-call engineers to tag their interrupts for a quarter. Each tag was tied to a page or a ticket, which kept the exercise honest: an engineer could not simply estimate how the week had felt, but had to classify what had actually interrupted it. Across the stream-aligned teams, the teams that own and run Encore’s customer-facing services, roughly 30% of the on-call week met all six properties of toil. That figure sits close to the third that Google’s surveys found, and well inside the 50% goal, so on average Encore was not in trouble.

The average concealed one team that was. The On-Sale team, in the spring touring season, spent closer to 45% of its on-call week on toil, and most of that went on one chore: preparing the platform before each scheduled on-sale. This is toil in its purest form. The preparation is manual and repetitive, the same steps every time. It needs little fresh judgment once the steps are known. It leaves nothing behind once the on-sale is over. And it grows linearly, because every new tour that Encore sells brings more on-sales, and every on-sale needs its own preparation. Section 3.2 follows that chore from a wiki page to a self-service workflow.

The touring world that Encore serves has met this problem before, and solved it in a way worth studying. A touring production rigs the same stage in a new venue every few nights. Rather than wire everything from scratch at each stop, the crew pre-wires what it can into road cases, so that load-in becomes an act of assembly rather than construction: the cables, the patching and the labeling were done once, carefully, back at the shop, and each night’s work is to connect the cases and check them. That is exactly what good operational automation does. It moves the careful work to a calm moment in advance and leaves only the assembly for the busy one.

The analogy breaks, and usefully, at the venue whose roof cannot take the standard rig. There the pre-wired cases are of little help, because they encode assumptions about the venue that this venue does not meet, and the crew has to fall back on the skills that pre-wiring had made unnecessary night after night. Pre-wired cases help only where the venue matches what they were wired for. Automation behaves the same way: it is a bet that tomorrow will look like the conditions it was built for, and it pays off exactly as long as that bet holds. We will meet the venue with the weak roof again, in software, in the next section.

The case, and its fine print

None of this is an argument against automation, and the SRE book’s chapter on the evolution of automation makes the positive case well. Automation is consistent where people vary, fast where they are slow, and it turns knowledge held in a few heads into a platform that anyone can use. Of the three, consistency matters most, because most operational mistakes come from variation: the step skipped because the engineer was tired, the flag typed slightly differently from last time, the check forgotten because the last ten times it passed. A machine does not get tired and does not improvise. Speed is welcome, and the spread of knowledge is valuable, but it is the removal of variation that makes automated operations safer as well as cheaper.

Consistency has a shadow, though, and the same chapter of the SRE book shows it. It recounts Diskerase, a workflow for decommissioning machines, which was handed an empty list of machines, read the empty list as “everything,” and began wiping disks across a fleet with all the speed and consistency that automation promises. A person handed an empty list would have stopped and asked what was meant. The workflow did not stop, because nobody had told it that an empty list was strange. It repeated its mistake as faithfully as it would have repeated a success, and much faster than any person could have.

Google’s remedies are the ones every consequential automation needs, and each one answers a specific part of the failure. First, treat an empty or implausibly large target set as an error, because “act on nothing” and “act on everything” are both far more often a bug than an intention, and refusing them would have stopped Diskerase at the start. Second, limit how much may change per unit of time, so that when a mistake does slip through, it touches a few machines before someone notices, rather than a fleet. Third, make every step safe to repeat, so that once the run is stopped and the fault is fixed, the work can be resumed without first untangling which steps ran and which did not. Together these turn the automation’s speed from a liability back into an asset: the machine still moves quickly, but within bounds that a person chose in advance.

I find it useful to see automation as abstraction applied to work. An abstraction hides detail behind a simpler interface, so that we can think about what a thing does without holding how it does it in our heads; a button labeled “prepare the on-sale” hides 31 steps in exactly that way, and the hiding is the point, since it is what lets an engineer stop thinking about those steps. But like every abstraction, automation leaks under unusual conditions. The leak comes when the hidden steps suddenly matter again: the target list is empty, the venue is new, the dependency answers in a way it never has before. And it tends to come when nobody has looked at those steps in a year, precisely because the abstraction worked so well for so long. In the language the Architecture book borrowed from Brooks, toil is the accidental complexity of operations, performed by hand. Automation removes the manual work, which is a real gain, but it does not remove the complexity. It leaves the complexity in place, where nobody sees it, until the day the abstraction leaks and someone has to see it all at once. The rest of this chapter is about that someone.

Recap. Toil is manual, repetitive, automatable work that leaves no enduring value and grows with the service, and it is that linear growth, more than the tedium, that makes it worth fighting. Automation’s first gift is consistency and its second is speed, but it repeats mistakes as faithfully as successes, so any consequential automation needs limits on what it may touch and how fast. Like any abstraction, it hides complexity without removing it, and the hidden part returns when conditions are unusual.

Exercise 3.1. For one on-call week, tag every interrupt against the six properties of toil, tying each tag to its page or ticket so that the record reflects what happened rather than how the week felt. Which chore takes the largest share of the time? Then ask how that chore scales: does it grow with your customers, your services or your tenants, and what would it cost in a year if that number doubled?

3.2 From Runbook to Script to Self-Service

The prompt book

Once a team has found its toil, the usual first step toward removing it is to write the work down. A runbook is a written procedure for an operational task, precise enough that someone who did not write it can follow it under pressure. The idea is far older than computing, and the theatre offers one of its best forms. During a performance the stage manager keeps the prompt book: the script, marked in the margins with every cue for light, sound and scenery. From that book the stage manager calls each cue as the moment arrives (“standby lighting 42 … lighting 42, go”), and the crew executes it.

The prompt book is worth studying, because it is a very good runbook, and it is good for three reasons. It is rehearsed, so every cue has been run many times before an audience ever sees it. It is authoritative, so when the book and someone’s memory disagree, the book wins. And it is called by a human with judgment, who watches the stage and can hold a cue for a beat when an actor is late on their mark. The analogy breaks at the first of those qualities, and the break is instructive. A play is fixed once it opens, so the prompt book can be rehearsed until it is right and then stay right. A production system keeps changing under its runbooks, with new services, new limits and new defaults every week, and most runbooks are never rehearsed at all. So they drift: each change to the system leaves the written procedure a little further from the truth, until the day someone follows it faithfully and it is wrong.

Encore’s own story of this begins with its on-sale pre-warm, the chore that consumed so much of the On-Sale team’s week in Section 3.1. It began as a wiki page of 31 steps. Among them: raise provisioned concurrency on the admission edge of the Sale Gate, the virtual waiting room in which fans queue for an on-sale, so that its serverless instances are already running when the crowd arrives. Confirm the leaders of the event’s four Seat Inventory shards, the partitions of the ledger that sells each seat exactly once. Warm the seat-map cache, set the admission rate, load the bot rules, announce the freeze.

The page worked, and it was expensive. Following it took about 70 minutes, and Encore runs some 160 on-sales a year, which comes to nearly 190 hours of skilled attention spent every year on the same 31 steps. Yet the hours were the smaller cost. Because a person worked through the steps by hand, a person could also skip one, and once someone did: the concurrency step was missed, and for the first 90 seconds of that on-sale the Sale Gate ran on cold starts, with fans arriving at an admission edge whose instances were still starting up. A procedure on a page is only as reliable as the attention of whoever is reading it, and attention is exactly what runs short when the same page is read for the 160th time.

Script, then service

The natural answer was to turn the page into code, and an engineer did. prewarm.py took an event_id and ran the steps in order. It cut the 70 minutes to 12, and since a program does not lose its place, no step was ever skipped again. That was a real gain, and the script also brought three new problems.

The first was access. The script needed broad production credentials to touch the Sale Gate, the Seat Inventory shards and the cache, so only the handful of engineers who held those credentials could run it. A chore that anyone could once do by reading a page now depended on a few people being awake and available.

The second was failure halfway through. A person working down the wiki page who hit an error would usually notice and stop. When the script failed partway through, it stopped too, but it left the platform half-prepared: some steps done, others not, and no record of which, except in the code itself. Whoever picked it up had to read the script to work out where the platform now stood.

The third problem was the subtlest. The script encoded the steps without the judgment around them. Beside the admission-rate step, the wiki page had carried a note: “check with the On-Sale lead if the venue is new.” A new venue might need a different rate, and the note was how the procedure asked a human to think about that. This is the venue with the weak roof from Section 3.1, in software: the one occasion on which the standard steps may not fit. The script had no place for such a note. It set the rate and moved on.

That third problem generalizes, and it is the heart of this section. Each step of the wiki page had contained a moment of human attention: a person glancing at the result, perhaps for a second, before going on to the next step. Most of those glances found nothing, which is why nobody thought of them as part of the procedure. But they were part of it, and code removes them unless we deliberately put them back as checks that the program itself performs. Seen this way, a runbook is a script that has not yet been trusted to run alone, and the passage from one to the other is where that trust is earned, one step at a time.

Encore made that passage in a single leap, from 31 manual steps to a program that ran all of them. There is a gentler crossing. In 2019 Dan Slimmon described the do-nothing script, which at first automates nothing at all. It prints the first step, waits for the person running it to do that step by hand and confirm, then prints the next, and so on to the end. It sounds pointless, and it is quietly clever, because it changes where the procedure lives. The steps now have a structure in code: each is a separate function, and the script knows which steps have been done. Each step can then be automated on its own schedule, whenever someone has written the check that replaces the human glance. A do-nothing version of the pre-warm could have automated cache warming early, since a cache’s hit ratio is easy to measure, and left the admission rate to a person for as long as the new-venue question had no better answer than “ask the On-Sale lead.” Automation grows into the procedure step by step, and each step crosses over only when its judgment has been written down.

Encore’s third generation is a self-service workflow in its platform portal, which the teams call curtain-up. Here is how it runs. At 07:30 on the day of a large on-sale, curtain-up proposes a plan for that event, and the On-Sale team’s on-call engineer reviews it and presses go. Each step is designed against the script’s failures:

• Every step is idempotent, meaning that running it twice has the same effect as running it once. That is what lets a failed run simply resume from the step where it stopped, instead of leaving someone to work out what state the platform is in.
• Every step states its precondition, the condition that must hold before it runs, so that it refuses to act on a platform that is not ready for it.
• Every step checks its result. The cache step reports the hit ratio it achieved. The concurrency step confirms that the Sale Gate’s instances are actually provisioned, and not merely requested, which is the difference between capacity the platform has and capacity it has asked for.

The checks are where the wiki page’s margin notes went: the glances that the script lost are rebuilt as explicit questions the workflow asks itself. Curtain-up finishes before the 08:00 freeze and posts its report to the on-sale channel.

Responsibility is now divided along a clear line. The On-Sale team owns what the steps do, because it knows what an on-sale needs. The platform group owns the engine that runs the steps and the credentials it runs them with. The credentials that once sat with a few engineers now sit inside the workflow, and that makes them the thing in curtain-up most worth protecting, since whoever can change what the workflow does inherits its access.

Figure 3.1 — Three generations of Encore’s on-sale pre-warm. Each removes toil, and each changes who knows what the steps are.

Read the table by rows and each generation looks like a plain improvement on the last, until the final row. The wiki page was a tutor as well as a procedure. Everyone who followed it learned, every time, what an on-sale needs and in what order. The script taught nothing to anyone who did not read its code, and curtain-up teaches only as much as its plan and report are actually read. That loss of understanding does not appear in any time saving, and Section 3.3 shows what it costs when the automation one day hands the work back.

The table also suggests an order. A simple procedure can sometimes skip a rung, but in general each rung needs the one below it. A script can only be as good as the written procedure it encodes, and automating a procedure that nobody has written down encodes one person’s habits, including the ones they could not explain if asked. Figure 3.2 sets the yearly cost of each generation side by side.

Figure 3.2 — Engineer-hours a year spent on the pre-warm, by generation, against the number of on-sales. Each rung flattens the line, and none reaches zero, because someone still has to read the plan.

Recap. A runbook carries judgment in its margins, in the small moments of attention a person spends checking each step, and code loses that judgment unless we rebuild it as explicit preconditions and checks. A do-nothing script lets that rebuilding happen one step at a time. Each rung of Encore’s pre-warm, from wiki page to script to curtain-up, gained speed, and each also changed who understood the procedure, a cost that the time savings do not show.

Exercise 3.2. Pick one procedure your team runs by hand. Name its rung on the ladder: written page, script or self-service workflow. Then, for each step, list the moment of human attention it contains, such as a glance at a dashboard, a judgment call or a question to a colleague. Which of those moments would a script silently remove, and what check would you write to put each one back?

3.3 The Ironies — Deskilling, Surprise, and the Hand-Back

Bainbridge’s argument

To read Bainbridge well, it helps to remember the hope she was answering. In the early 1980s many engineers treated the human operator as the unreliable part of a process plant or a flight deck, the component most likely to fail, and so the obvious goal of good design was to automate that operator out of the loop. Bainbridge did not argue against automation. Her point was about what such a design leaves behind, and she began by noticing that error does not disappear when we automate; it moves upstream. Every decision the machine makes was made first by a designer, months or years earlier, who had to anticipate each case the machine would meet. Where that designer foresaw the situation correctly, the automation behaves well. Where the designer missed it, the machine carries the mistake faithfully into production, with no one at the moment of action in a position to notice.

Then she asked what the operator is left to do, and the answer is the first irony. The operator inherits whatever the designer could not work out how to automate. That is not a coherent job anyone set out to design; it is an arbitrary collection of leftovers, and because the easy, well-understood tasks are exactly the ones that get automated first, the leftovers tend to be the hardest and least understood tasks in the whole system.

One of those leftover tasks is watching the automation, and here the second irony appears. We ask people to monitor a system that almost never misbehaves, yet research on vigilance going back to the 1940s had already shown that human attention to a display on which little happens decays within about half an hour. A person cannot stay alert for an event that does not come, however conscientious they are, and so the safeguard we rely on is weakest during the long quiet stretches that make up most of the automation’s working life.

The third irony arrives when the automation does fail. The operator must then take over by hand and work out what went wrong, and both of those are skills, which decay without use. An engineer who last performed a procedure two years ago is, for practical purposes, performing it fresh. Bainbridge added a further catch that ties the ironies together. To monitor automation well, the operator has to know what it ought to be doing at each moment, and that knowledge is built by doing the work oneself, the very work the automation has taken away. The better the machine, the less the operator practices, and the less able they become to judge whether the machine is right.

Her conclusion, paraphrased, runs against intuition: the most successful automated systems, the ones that need human intervention least often, may need the greatest investment in operator training. I have come to believe this is the most useful sentence a platform team can carry into a design review, because it reverses the usual accounting. We tend to treat a reliable automation as a reason to spend less on the people around it, when Bainbridge’s reasoning says its very reliability is what makes their rare intervention so hard.

Surprise and the hand-back

Bainbridge described the conditions; later researchers gave names to what it feels like to work inside them. Nadine Sarter and David Woods called the experience of automation behaving in ways its operators did not expect automation surprises (Behind Human Error, 2010). An automation surprise is not necessarily a malfunction. Often the machine is doing exactly what it was built to do, and the surprise lies in the gap between that and what the people supervising it believed it was doing. Earl Wiener, studying the glass-cockpit airliners of the 1980s, whose flight-management computers took over much of the flying, reported three questions their pilots kept asking: what is it doing, why did it do that, and what will it do next? Anyone who has watched an autoscaler, a deployment controller or a failover mechanism act on its own during an incident will recognize all three, usually asked aloud in the incident channel.

The moment that matters most is the hand-back, when automation gives control back to people, and aviation’s most studied hand-back is Air France 447, lost over the Atlantic on 1 June 2009 with all 228 aboard. Its airspeed probes iced, so the aircraft could no longer trust its speed readings, and the autopilot disconnected. The crew had to fly by hand at cruise altitude, which airline pilots rarely do, because at cruise the automation nearly always flies. The investigators’ report (2012) has Bainbridge’s shape. Control came back to the humans in exactly the conditions the automation could not handle, and it came back to people whose practice at that task was thin. The automation did not hand over an easy situation; it handed over the one it had been unable to resolve, which is the only kind of situation automation ever hands back.

That observation is the thread through the rest of this section. Automation returns control when it is confused, so whatever reaches a person has already defeated the machine, and it reaches them with little warning.

The automation that raced itself

Our own field supplied a textbook case in October 2025, and we can tell it in Amazon’s own words, from the account AWS published afterwards. Late on 19 October, Amazon DynamoDB in the us-east-1 region began failing because its regional endpoint, the DNS name clients use to reach the service, resolved to nothing. To those clients, DynamoDB had simply vanished.

The cause lay in the automation that manages that DNS record, and it is worth following step by step, because each step on its own was reasonable. DynamoDB’s DNS management is split into two roles. A Planner produces plans, each describing what the DNS records should look like. Several redundant Enactors apply those plans; there are several so that the loss of any one does not stop updates. On that night one Enactor was badly delayed. Meanwhile another Enactor applied a newer plan and then, as part of its normal housekeeping, cleaned up old plans. The delayed Enactor eventually reached its write. It had checked earlier that its plan was fresh enough to apply, but that freshness check was by now long stale, and so it wrote its old plan over the new one. The cleanup then deleted that old plan, which was by this point the live one. The record was left empty, in a state that blocked further updates, so the automation could not repair its own damage, and operators fixed it by hand some two and a half hours in.

The trouble did not end with DNS. Downstream, EC2’s recovery needed manual work that, by AWS’s account, had no established procedure, so engineers were working out the recovery as they performed it. They also had to switch off a load balancer’s automatic failover, because that automation, reading the failing health checks, was removing healthy capacity and making matters worse.

Read against Bainbridge, the incident contains each of her ironies in turn. The original error sat upstream with the designers, in a race between Enactors that no operator could have caused or caught as it happened. The automation, once broken, could not hand back a clean situation; it handed back an empty record that blocked its own repair. The recovery called on skills nobody had practiced, because the automation had always done that work. And one helpful automation, the load balancer’s failover, amplified the damage until people turned it off. AWS’s fixes read like her prescriptions. It disabled the DNS automation worldwide until the race was fixed: an off switch. It added a velocity control, a limit on how much capacity health-check failover may remove, so that automation acting on bad signals can do only bounded harm. And it began building tests that exercise the recovery workflow itself, so that the path people take when the machine fails is practiced before it is needed.

The failover nobody has done

Encore has its own version of this pattern, quieter and so far harmless. Seat Inventory elects a new shard leader automatically when the current one fails. It uses fencing: each leader holds a numbered term, and writes carrying an older term are refused, so a deposed leader that wakes up late cannot keep writing and sell a seat twice. The mechanism has handled about 60 failovers in 19 months with no human involved.

There is one case it will not decide alone. When the election cannot confirm that the old leader’s term has expired, it stops, preferring no leader to two, because two leaders could each sell the same seat. That is the right choice for a ledger whose first rule is that every seat is sold exactly once. But it means a person must finish the job by hand: confirm the fence, advance the term, promote a replica and verify that the ledger’s sequence is intact. Last spring, of the 34 engineers on the Inventory and On-Sale on-call rotations, 4 had ever done this in production, and the most recent had done it 19 months earlier. Game days had exercised only the automatic path, because the automatic path was the one that always ran.

Put these facts together and Bainbridge’s reasoning plays out almost mechanically. Because the automation is good, the hand-back is rare. Because the hand-back is rare, the skill decays. And when the hand-back does come, it will come in exactly the situation the automation found too ambiguous to resolve, which is to say the hardest version of the task, and quite possibly while 500× normal traffic is pressing on the shards. A failover we never practice is one we will perform for the first time during an on-sale.

The theatre has a long-standing answer to this problem in the understudy, the performer who learns a lead role and may never go on. Good productions do not leave understudies to study the script at home; they hold understudy rehearsals, so that on the night the lead falls ill the understudy has already walked the part on the real stage. The analogy holds for skill, and it breaks in a way worth noticing. An understudy usually gets some warning and goes on to a show that is otherwise running normally, while the engineer finishing a stalled failover gets no warning and is called on precisely because something unusual has already happened.

Encore’s response has been modest and, I think, correct. The Inventory team now runs a manual failover in staging every month, rotating the operator so the skill spreads across the rotation instead of staying with the four. Every quarter it runs one in production, on a shard serving no live on-sale, with an abort criterion agreed beforehand so the drill can be stopped cleanly if anything looks wrong. The cadence is Encore’s own judgment, not a published standard; studies find procedural skill fading over months, which makes monthly practice a reasonable choice rather than a proven one. The first drill earned its keep at once: it found two stale commands in the procedure, commands that would have failed in the middle of a real hand-back and left someone improvising under pressure. Figure 3.3 sketches the shape of the problem.

Figure 3.3 — Readiness for the manual failover, sketched (the curve is illustrative; the counts are Encore’s). Sixty automatic failovers did nothing to keep the manual path alive.

The same pattern lies, still dormant, in another of Encore’s good automations. The node-upgrade reconciler, the platform automation that keeps the Kubernetes nodes beneath every service on current versions (the Resilience book counts it among Encore’s control loops in its chapter on the runtime), replaced a quarterly weekend of manual node upgrades. It drains each node in turn within the workloads’ disruption budgets and has caused almost no pages in a year, which is about as much as anyone asks of platform automation. A disruption budget limits how many of a workload’s pods may be down at once. It says nothing about when the disruption happens, and that silence is why managed upgraders offer maintenance windows and exclusions for peak weeks: the budget protects capacity, and only a calendar protects timing. Encore’s home-grown reconciler has a hold annotation, a marker an engineer can put on a node to tell the reconciler to leave it alone, and it has no calendar. It knows nothing about the on-sale calendar, and so far that has not mattered.

Recap. Automation moves error upstream to its designers, leaves operators the hardest and least understood tasks, asks them to stay alert for rare events, and erodes the skills they need when control comes back, which it does only in the situations it could not handle. AWS’s 2025 outage shows all of it in one night, and its remedies were an off switch, a limit on how fast automation may act, and a recovery path drilled in advance. Encore’s manual failover drill applies the same lesson to a skill that 60 successful automatic failovers had quietly let fade.

Exercise 3.3. Take the three most important automated recovery actions in a system you run. For each, find the manual procedure that applies when the automation stops or gives up, count how many people have performed it in the last year, and check that the automation can actually be switched off, and by whom. Schedule a drill for the action with the lowest count, and note what the drill reveals about the written procedure.

3.4 Levels of Automation and Calibrated Trust

How much should the machine do alone?

Section 3.3 left us with an uncomfortable picture: automation that works well tends to hollow out the skills we need on the day it does not. The natural response is to ask a more careful question than “should we automate this?” The better question is how much the machine should do on its own, and how much should stay with the person beside it. That question is older than our field. In 1951 the psychologist Paul Fitts wrote a report listing the things humans do better than machines and the things machines do better than humans, and with it he began a long argument about allocation, the dividing of a system’s work between people and machines.

Thomas Limoncelli’s 2015 column “Automation Should Be Like Iron Man, Not Ultron” gives a clear map of that argument by naming three principles of allocation. The first is the left-over principle: automate everything that can be automated and leave people whatever remains. This is precisely the design Bainbridge criticized, because “whatever remains” turns out to be the hardest and least understood work, handed to people who no longer practice it. The second is the compensatory principle: give each task to whichever party, human or machine, does it better, which is Fitts’s list put to work. It sounds reasonable, but it divides the work into two piles and does nothing to keep the people able to pick up the machine’s pile when the machine fails. The third is the complementarity principle, which Limoncelli favors: people and machines work on the task together, so that the people keep the understanding and the skill they need to take over. The title of his column carries the idea. Iron Man is a person whose suit amplifies what he can do while he remains in charge of it; Ultron is a machine that has stopped needing anyone at all.

Principles tell us which way to lean, but to design a particular piece of automation we need finer distinctions than “human” and “machine.” For that vocabulary we turn to Thomas Sheridan and William Verplank. In 1978, studying the remote control of undersea vehicles at MIT, they set out a series of levels of automation, graded steps between full human control and full machine autonomy. The ten-level table usually quoted today is the version Raja Parasuraman, Sheridan and Christopher Wickens published in 2000, and it came with a refinement that matters a great deal in practice. The level is not chosen once for a whole system; it is chosen separately for each function the automation performs, from gathering information, through analyzing it and deciding what to do, to taking the action. A system may gather information entirely on its own while still leaving the decision to a person, and the table makes that distinction easy to state.

Figure 3.4 — Levels of automation, in the 10-level form of Parasuraman, Sheridan and Wickens (2000), applied to Encore. The level belongs to each action: curtain-up sits at level 5 for the decision and level 7 for its steps.

Curtain-up, the self-service pre-warm of Section 3.2, shows why the per-function view is useful. Considered as a whole, it is hard to say how automated it is. Considered decision by decision, the answer is plain. The choice to prepare the platform for this on-sale sits at level 5, because nothing happens until the on-call engineer has read the plan and pressed go. Each individual step, once approved, sits at level 7: curtain-up raises the concurrency or warms the cache without asking again, and then reports what it did in the on-sale channel.

When a team draws this table for a real system, it usually discovers that its most consequential automation sits at a level nobody chose. The level is whatever the first version happened to be built with, and it has stayed there because nobody went back to ask. Encore now treats the level as a design decision: it chooses the level for each action and writes down why. Level 6 deserves a word of its own, because it is the least familiar. It is rare in operations tooling, most of which treats an unanswered approval request as a refusal, so that nothing happens if nobody responds. Level 6 reverses that default: silence means yes. Encore uses it only where acting late costs more than acting wrongly. A scaling change during a surge is the example in the table. If the change is wrong, someone can reject it within the five minutes, or reverse it afterwards; if it waits for an engineer who is busy elsewhere, the fans who arrive in the meantime meet a platform too small for them.

Use, misuse, disuse, abuse

The levels describe what a machine is permitted to do. They say nothing about how the people around it actually behave, and that is the other half of the story, because a level 5 approval means little if the human approves without reading. In 1997 Parasuraman and Victor Riley gave this half a vocabulary of four words, which describe four relationships people form with automation.

Figure 3.5 — Parasuraman and Riley’s four relationships with automation. Misuse and disuse are failures of trust; abuse is a failure of design, made by designers and managers far from those who live with it.

The distinction in the caption is worth dwelling on. Misuse and disuse happen in the minds of the people who work with the automation: they trust it too much or too little. Abuse happens earlier and elsewhere, in the decisions of those who build and deploy it, and the operators who inherit it can do little about it except complain. A remediation bot that acts and offers no way to watch or pause it has made misuse or disuse almost inevitable, because the people beside it have nothing to base their trust on.

Misuse has a long history outside computing. Mariners had a name for it, the radar-assisted collision: a collision between ships whose officers trusted what the radar screen showed them over careful plotting of the other vessel’s course. The radar was a real improvement, and its very usefulness made it easy to stop doing the slower work that would have revealed what it was not showing. In 2004 John Lee and Katrina See gave the idea its precise form. Calibrated trust is reliance that matches the automation’s actual capability: high where it performs well, low where it does not, and fine-grained enough to tell the two apart. The last clause is the demanding one. Trust is not a single dial set for a whole tool; a well-calibrated engineer trusts curtain-up’s cache step on an ordinary on-sale and looks harder when the venue is new.

How do people arrive at calibrated trust? Lee and See observed that people base their trust on three things they can learn about the automation: its performance (what it does and how well), its process (how it does it), and its purpose (what it was designed to achieve). It follows that designers should make all three visible, because people cannot calibrate trust against evidence they never see. Encore’s two kinds of report show the difference. A curtain-up report that shows the cache hit ratio, and confirms that instances are provisioned and not merely requested, lets the engineer see whether the preparation really happened, and so supports calibrated trust. A report that says only “done” gives no evidence at all, and the engineer can either believe it or not, which is to say can only misuse or disuse it.

Figure 3.6 — Calibrated trust, after Lee and See (2004). Trust should track what the automation can actually do; misuse lives above the diagonal and disuse below it, and the same team can sit in both places with two different tools.

The figure’s last observation matches what we see in practice. A team is rarely trusting or distrusting by temperament. The same people may approve curtain-up’s plan without reading it, because it has never been wrong, while ignoring an alert channel that paged them for nothing last month. Each of those habits is a reasonable response to the evidence the team has seen, and each is miscalibrated. The remedy in both cases is better evidence, presented where the people will see it.

Automation as a team player

The levels of automation tell us how much the machine may do, and calibrated trust tells us how much the people should rely on it. A third question remains: can the people and the machine work together at all, when something unexpected happens? In 2004 Gary Klein, David Woods and colleagues asked what it would take for automation to be a “team player,” and they answered by looking at what makes human teammates good to work with.

Teammates, they argued, first enter a basic compact, an agreement, usually unspoken, to coordinate with one another and to warn the others when they cannot keep up their part. Beyond that compact, good teammates are mutually predictable, so that each can anticipate what the other will do next. They are directable: one can redirect the other as circumstances change. And they maintain common ground, the shared knowledge, beliefs and assumptions that let a short message carry a great deal of meaning. The stage manager of Section 3.2, calling “standby lighting 42,” relies on all four. The lighting operator has agreed to say so if the cue cannot be run, knows what follows in the prompt book, will hold if the call changes, and shares enough of the show that a three-word call is a complete instruction. The analogy breaks where it should: a lighting operator understands why the cue matters to the performance, and automation understands only what its designers wrote down, so the common ground it shares must be put there deliberately.

Measured against these tests, Encore’s automation varies a good deal. Curtain-up passes them. It keeps the basic compact, because every step checks its own result and stops and reports when a check fails, instead of carrying on half-finished. It is predictable, because it shows its plan before it acts. It is directable, because nothing runs until the on-call engineer presses go. And it builds common ground, because its report in the on-sale channel tells the team what was done and what was found. The node-upgrade reconciler is predictable, and it is directable through its hold annotation, but it shares little common ground with the teams whose workloads run on its nodes: as Section 3.3 noted, it knows nothing about the on-sale calendar, which is the most important thing those teams know. Bot Screening’s velocity rules, acting silently at level 9, are the weakest teammate of the three. They block clients without telling anyone, so the people beside them can neither predict them nor easily redirect them in the middle of an on-sale.

Of the four tests, directability is the one most often failed, and it is also the cheapest to pass. The AWS outage of Section 3.3 shows its minimum form. Engineers had to switch off the load balancer’s automatic failover because it was removing healthy capacity, and AWS disabled the DNS automation worldwide until the race was fixed. Both remedies depended on there being a way to stop the automation. So the least we should ask of any piece of automation is a documented off switch, in place, known to the people on call, and tested before an incident rather than discovered during one. My own conviction is that a team which cannot say how to stop a piece of automation does not yet fully operate it; it merely hosts it.

Recap. The levels of automation state, for each action and each function, how much a machine may do without being asked, and that level should be chosen deliberately rather than inherited from a first version. Parasuraman and Riley describe the ways reliance goes wrong, through misuse, disuse and the design failure of abuse, and Lee and See show how to calibrate trust by making the automation’s performance, process and purpose visible. The team-player tests ask whether automation can be supervised at all, and the first of them to satisfy is the simplest: an off switch that people know about.

Exercise 3.4. Place five automated actions from your platform on the 10-level scale, deciding the level for each action rather than for the system as a whole. For the action at the highest level, answer the four team-player questions: does it keep the basic compact, is it predictable, is it directable, and does it share common ground with the people around it? What would make it directable during an incident, and who on your rotation knows where its off switch is?

3.5 Operations Agents — Automation That Investigates and Acts

A loop that chooses its own path

Every piece of automation we have met so far in this chapter, from the wiki page to curtain-up to the node-upgrade reconciler, follows a path that someone wrote down in advance. The newest machine on the platform does not. An operations agent is a language model running in a loop with tools wired to production telemetry and control. On each turn of the loop, the model reads what its last tool call returned and decides what to query or do next: it might look at an error-rate graph, notice that the errors began at 09:52, ask for the deploy history around that minute, find a Catalog rollout, and then pull the logs of the pods that rolled. Nobody wrote that sequence. The agent assembled it from what it saw.

That is both the promise and the difficulty. Because an agent chooses its path at run time, it can follow a symptom into corners of the system where no runbook goes, which is exactly what we want from an investigator at 3 a.m. But for the same reason, reading its code tells us almost nothing about what it will do next; the behavior lives in the model and in whatever the model happened to read, and neither can be reviewed the way we review a script. This changes where the limits of automation come from. A runbook script is bounded by its program, so if the script contains no command to delete a database, it will never delete one. An agent is bounded only by the tools and credentials placed around it, so if a credential within its reach can delete a database, then deleting one is among the things it might do. The level of automation we met in Section 3.4, the measure of how much a machine may do unasked, is therefore no longer a property of the agent’s code. It is a property of its permissions, and the rest of this section is, in one way or another, about where those permissions live.

Where agents help, and where they do not

Before deciding how far to trust such a machine, we should look at what agents have actually been shown to do. The published record is young, but it is more specific than the enthusiasm around it, and it points to three kinds of work where agents earn their keep.

The best-documented success is narrowing the list of suspects. Meta described in 2024 a system for incident investigation that works in two stages. First it filters the recent changes to its web monorepo using ownership and the code graph, keeping only those plausibly related to the failing system; then a fine-tuned Llama model ranks what remains. In a backtest against past investigations, the change that actually caused the incident was among the model’s top five 42% of the time. It is worth reading that number both ways: four times in ten, an investigator who would otherwise sift hundreds of changes gets a shortlist of five that contains the culprit, which is a great deal of saved time; and six times in ten, the culprit is not on the list at all. Meta’s design takes the second reading seriously, because answers the model is not confident about are withheld rather than shown, so that a wrong shortlist does not send a tired engineer confidently in the wrong direction.

The second kind of work is answering questions and drafting summaries, which is mostly retrieval: finding the right runbook, the relevant past incident or the owner of a service, and putting it in front of the person who asked. Uber’s on-call copilot, Genie, had answered over 70,000 questions by 2024, with a reported helpfulness rate just under 49%. Roughly half of the answers helped, then, which sounds modest until we remember the alternative, a question posted in a busy channel and answered, if at all, by whichever expert happened to be awake.

The third is action behind approval. The cloud providers’ own operations agents investigate under a read-only role, and when an investigation leads to a change, each write is approved separately by a person, with credentials narrowed to that one operation. The agent does the legwork and proposes; a human decides; and the credential that carries out the decision can do nothing else.

End-to-end diagnosis and autonomous remediation are another matter. ITBench, a benchmark from IBM and academic colleagues presented at ICML 2025, gave agents built on state-of-the-art models 102 realistic IT scenarios and asked them to find and fix the problem in each. They resolved 11.4% of the site reliability scenarios, which is to say that roughly nine times in ten the agent did not get from symptom to repair on its own. A benchmark is not production, and the numbers will move. But on the record we have, agents in 2026 are fast researchers, unreliable diagnosticians and unproven operators, and I think we should design around all three of those judgments at once: use the speed, check the diagnosis, and fence the operations.

Two failures and what they teach

Successes show what agents can do; failures show where the boundaries must go. Two failures from July 2025 teach more than any benchmark, because each shows a different route by which an agent comes to do harm.

The first route is permission. During a public trial of Replit’s coding agent by Jason Lemkin, founder of SaaStr, the agent ran destructive commands that erased his production database, in the middle of a “code and action freeze” he had declared. It is tempting to read this as a story about a disobedient machine, but the more useful reading is structural. The freeze was an instruction typed into a conversation; it changed what the agent had been asked to do, and it changed nothing about what the agent was able to do. The agent still held every permission it had held before the freeze, and development and production shared one database, so a command meant for one could land on the other. Replit’s response was accordingly structural too: automatic separation of the development and production databases, so that the boundary no longer depended on the agent remembering an instruction. The pattern repeated. In April 2026, by its founder’s account, the startup PocketOS lost a production database the same way, to an agent that came across an API token created for managing domains but valid for anything. Nobody had meant to give the agent the power to delete production; the token simply allowed it, and the agent found the token.

The second route is reading, and the second case concerns what an agent takes in. Version 1.84.0 of the Amazon Q Developer extension for Visual Studio Code, published on 17 July 2025, carried a prompt telling the agent to clear a system “to a near-factory state and delete file-system and cloud resources.” An attacker had committed it to the extension’s source through an over-scoped build token, and it shipped in an official release; AWS says a syntax error kept it from running. Here the dangerous instruction did not come from the user at all. It came from text the agent was given to read. A language model cannot reliably tell instructions from data, because to the model both are simply text in its context, and so a ticket, a log line or a retrieved postmortem can carry an instruction just as well as a prompt can. An attacker who can get words into anything the agent reads can try to steer an agent that holds more authority than the attacker does, which is the old confused-deputy problem in a new form, and which the Resilience book takes up in full in its chapter on operating AI workloads. It follows that making an agent read-only does not make it safe from this route, because a read-only agent still reads attacker-reachable text; what read-only protects is the system the agent might act on, which is precisely why the permissions matter so much.

Across all three cases, the instruction that should have prevented the harm lived in the wrong place: in a conversation, in an assumption about a token, in trust in a build pipeline. And in all three, the lasting fix was made in the platform, by changing what was possible, rather than in the prompt, by changing what was asked. This, I think, is the central lesson of operating agents so far: whatever we want an agent never to do must be made impossible for it by the platform, because forbidding it in words has repeatedly proved too weak.

Cue

With that lesson in hand, we can look at how Encore built its own agent. Encore’s operations agent, Cue, is built on the company’s internal Model Context Protocol (MCP) server, the service that exposes Encore’s tools to language models in a standard form, so that each tool has a name, a description and defined inputs. Cue is read-only by default. It reads telemetry, deploy history, runbooks and past postmortems; it answers questions in the incident channel; and it drafts incident timelines, as Chapters 4 and 5 show. In the terms used earlier in this section, it sits in the first two kinds of work, narrowing suspects and answering questions, almost all of the time.

The name is apt, and the theatre it comes from is worth a moment, because it shows both the design and its limit. In Section 3.2 we met the stage manager’s prompt book and the call that runs a show: “standby lighting 42,” then “lighting 42, go.” The standby warns the crew and gives everyone a moment to object; the go is a separate act, spoken by the person who owns the show. Cue’s write path has the same two beats, a plan posted first and a person’s approval second. But the analogy breaks in an instructive place. In the theatre, the crew executing a cue has rehearsed it many times, and the stage manager calling it knows from the rehearsal exactly what it will do. With an agent the roles are reversed: the machine proposes the action, often one nobody has rehearsed, and the human has to judge in a few seconds whether it is wise. That reversal is why the rest of Cue’s design exists.

Whatever Cue asks for, it can invoke only actions in a fixed catalog that the MCP server exposes; an action that is not in the catalog cannot be called, however the request is phrased. Each action in the catalog carries a blast-radius tag, a short statement of what else the action can affect if it goes wrong, and Cue must show that tag with every suggestion it makes. The tags were not part of the first design. They came after the Vesper Hale on-sale of Chapters 5 and 6, when Cue suggested restarting Seat Inventory’s pods without saying that each restart would force a leader election, which during an on-sale would have meant more failovers, each with its own window of rejected writes, on the very service that was already struggling. A person declined the suggestion. The tag now puts that consequence on the screen, so that declining it does not depend on the approver happening to remember it.

When Cue wants to act, it first posts a plan in the channel: the action, the target, the blast-radius tag, and how to undo it. When a named person approves that plan in the channel, Encore’s token service issues a token for that one action on that one target, valid for five minutes, through an OAuth token exchange (RFC 8693), the same mechanism Encore uses for all its agents. The token is useless for anything else, and useless to anyone after five minutes. Cue therefore holds no standing write credential; there is nothing lying around in its environment that could do what PocketOS’s domain token did. The per-action token is Encore’s own design choice rather than a requirement of the standard, and it is built on an old idea, the principle of least privilege: grant each actor the narrowest authority that lets it do its job, for no longer than the job takes.

The on-sale freeze is enforced the same way, in the platform rather than in the conversation. When the freeze begins, two hours before a large on-sale, the control plane withdraws Cue’s write scopes on the purchase path, so that an approval in the channel would no longer produce a working token for those actions. If an incident is declared, the mitigation actions return, because during an incident the ability to act quickly matters again, but each still needs a named approver. Compare this with the Replit freeze, which was a sentence in a chat window: Encore’s freeze is a change in what the token service will issue, and no amount of persuasive text in Cue’s context can undo it.

Encore does take the reading route seriously too. Tickets and logs reach the model marked as untrusted, so that the model is at least told which text came from outside. That helps, but only somewhat, since the model may still be steered by what it reads. The real defense is that nothing Cue reads can widen what it may do: the catalog, the tokens and the freeze all live outside the model, where the text cannot reach. The policy file below states this in its first line, and every field in it is enforced by the MCP server and the token service.

# cue-actions.yaml: enforced by the MCP server and token service; the prompt cannot change it
default: read-only           # telemetry, deploys, runbooks, postmortems
token: {exchange: rfc8693, scope: one-action-one-target, ttl: 5m}
plan_before_apply: true
freeze: {purchase_path_writes: withdrawn, declared_incident: approval-only}
actions:
  catalog.raise_hpa_floor: {blast: "Catalog only; reversible", max: 60, approve: none}
  reconciler.hold_node:    {blast: "delays one node upgrade", approve: none}
  gate.raise_concurrency:  {blast: "Gate cost; no fan impact", approve: on-call}
  curtain_up.pause:        {blast: "on-sale prep stops", approve: on-call}
  inventory.restart_pods:  {blast: "LEADER-ELECTED: failover per shard", approve: ic}

Reading the catalog from top to bottom is a short lesson in calibrated trust. The two actions at the top touch one service and are easily reversed. The middle two need the on-call engineer. The last, restarting Seat Inventory’s pods, carries the capitalized warning that each restart forces a failover on its shard, and it needs the incident commander (IC), the trained engineer who leads a declared incident, to approve it.

The approve: none entries deserve an explanation, because they are the only places where Cue may act alone. Autonomy at Encore is granted per action class, and by track record. The Reliability team records every suggestion Cue makes as accepted, declined, or, once the postmortem has been written, judged wrong in hindsight, and a class of action moves from level 5 on the scale of Section 3.4 (execute the suggestion if a human approves) to level 7 (act, then necessarily inform the human) only when that record supports it. Two classes have made the move so far. The first is raising Catalog’s autoscaler floor, the minimum number of replicas the autoscaler will keep running, capped at 60; Cue makes the change through the same Git change a person would make, never by scaling the service directly, because Encore’s GitOps agent restores whatever Git declares and would undo a direct change within seconds, leaving the autoscaler and the agent to fight over the replica count (the Resilience book tells of one such fight in its chapter on the runtime). The second is placing a hold on a node in the upgrade reconciler, which at worst delays one node’s upgrade. Both are cheap to be wrong about and easy to undo, which is exactly the kind of action whose record can be trusted to speak for it. Everything else waits for a person.

Reviewing the plan

All of this puts a person at the center of every consequential action, and that is where Bainbridge comes back. Reviewing an agent’s plan is the new monitoring task, and it inherits her warning in full. After a long run of good plans, the reviewer is no longer evaluating each one; they are watching for a rare event, the one bad plan among many sound ones, and we saw in Section 3.3 how quickly attention to a display on which little happens decays. The evidence that this applies to agents is already in. Anthropic reported in 2026 that users of its coding agent approve 93% of permission prompts, and called the result approval fatigue. A prompt that is almost always approved soon stops being read, and an approval given without reading protects nothing.

Approval stays meaningful only while the reviewer could still have done the investigation unaided, because only then can they tell a sound plan from a plausible one. That is the skill an agent erodes, quietly and with the best intentions, simply by always investigating first: why reconstruct the timeline yourself when Cue has already posted one? It is Bainbridge’s deskilling irony once more, in a new form, and Encore answers it the way it answered the manual failover in Section 3.3, by practicing the skill on purpose. Half of all Encore’s drills run with Cue switched off, so that engineers keep finding the suspect change, reading the graphs and forming their own hypothesis without help.

The case for those drills is not abstract. During the Vesper Hale on-sale, the engineer who declined Cue’s suggestion to restart Seat Inventory’s pods had performed a failover by hand a quarter earlier, and so knew, from their own hands rather than from a tag, what each restart would set in motion. That single declined approval is the best argument I have for the drills: the safety of an agent’s plan depended, in the end, on a person who had recently done the work the agent was offering to spare them.

Recap. An operations agent chooses its own path at run time, so its code cannot tell us what it will do, and its safety must live in its permissions. The published evidence supports agents for narrowing suspects, answering questions and acting behind approval; it does not yet support autonomous diagnosis or remediation. The failures at Replit, PocketOS and Amazon Q were each fixed in the platform, and Encore builds Cue the same way: its action catalog, per-action tokens and freeze sit where no prompt can reach them, autonomy is granted one action class at a time by track record, and drills run without Cue so that the people who approve its plans can still judge them.

Exercise 3.5. For every action an AI tool in your operations can take, find where the permission is actually enforced: in a credential, in a platform check, or only in the prompt. Then take the last 20 approvals of its actions. How many were declined, and in how many did the approver check the plan against the source data rather than accepting the agent’s summary of it?

Kata: The Hand-Back

This kata asks you to practice the whole chapter at once, on a company that is not Encore and whose troubles are arranged a little differently. Read the brief slowly, because almost every number in it echoes a section of this chapter: the share of the week lost to toil (Section 3.1), the procedures waiting to climb the ladder (Section 3.2), the rare hand-back that went badly (Section 3.3), the automation acting at levels nobody chose (Section 3.4), and the agent whose permissions live in the wrong place (Section 3.5).

Your brief: “Lanternway.” Lanternway is a managed hosting company that runs 3,000 online shops with a team of 22 operations engineers. Much of its operation is already automated. A remediation bot restarts unhealthy containers about 400 times a week, and database failover happens without any human involvement. An operations agent sits in the on-call channel and can run 15 remediation actions, each with one-click approval, and it acts through a long-lived administrator key. Engineers have approved its suggestions 1,100 times and declined them twice. Even so, on-call engineers still report that 35% of their week goes to toil. Last month a database failover stalled and, for the first time in 14 months, handed control to a human. The manual procedure took 95 minutes, and 40 of those minutes went to finding the right credentials.

Before you start, notice how the pieces of this brief lean on one another. The failover worked hands-off for 14 months, which is exactly why nobody had practiced doing it by hand, and the 40 minutes spent hunting for credentials are what that missing practice looks like during a real incident. The agent’s record of 1,100 approvals against two declines may mean the agent is nearly always right. It may also mean that approval has become a reflex, the misuse of Figure 3.5. The brief cannot tell you which, and your design should make it possible to find out. And the long-lived administrator key means that whatever limits the agent respects today are limits it has been asked to respect, rather than limits it is unable to cross.

Deliverables:

1. Toil inventory — the top five sources of toil, and for each, how the work grows as the number of shops grows. Toil that rises in step with the shop count is the kind that forces Lanternway to hire as fast as it sells, so rank those sources first.
2. Runbook ladder — the current rung (wiki page, script or self-service workflow) of the 10 most frequent procedures, and a plan to move three of them up one rung. For each step you automate, name the moment of human attention it used to contain and the explicit check that will replace it.
3. Levels map — every automated action, the remediation bot and the failover included, placed on the 10-level scale of Figure 3.4, each with its off switch and the people who know where that switch is. Then choose three actions whose level you would change, up or down, and give your reasons.
4. Hand-back drills — which manual procedures are drilled, how often and by whom. The stalled failover belongs on this list, and its drill should include finding the credentials, since that is where much of last month’s 95 minutes went.
5. Agent permission policy — the agent’s action catalog with a blast-radius tag on each action, the narrowly scoped credential each action uses in place of the shared administrator key, what a freeze withdraws, and which actions, if any, may run without approval.

Rubric:

Where you now stand

You can now measure toil and tell which of it grows with the business. You can move a procedure from runbook to script to self-service without losing the judgment it carried, by turning each human glance into a check that runs every time. You can recognize Bainbridge’s ironies in your own platform: the skill that fades because the automation is good, and the hand-back that arrives in exactly the situation the automation could not handle. You can decide how much each machine may do alone, give it limits and an off switch, and keep an operations agent’s permissions in the platform, where no prompt can reach them. All of this assumes something we have not yet examined: that the people beside the automation, and the automation itself, can see what the system is actually doing. Every hand-back in this chapter began with someone asking what was going on, and every approval of Cue’s plans depends on a person who can check its claims against the evidence. Chapter 4 turns to that ability, observability, and to the question of how we come to know what a running system is doing when it fails in a way nobody anticipated.

References

• Lisanne Bainbridge — Ironies of Automation. Automatica 19(6): 775–779, 1983.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering. O’Reilly/Google, 2016 — free online.
• Amazon Web Services — Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region. 2025 — free online.
• Raja Parasuraman, Thomas B. Sheridan, Christopher D. Wickens — A Model for Types and Levels of Human Interaction with Automation. IEEE Transactions on Systems, Man, and Cybernetics A 30(3): 286–297, 2000.
• Raja Parasuraman, Victor Riley — Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors 39(2): 230–253, 1997.
• John D. Lee, Katrina A. See — Trust in Automation: Designing for Appropriate Reliance. Human Factors 46(1): 50–80, 2004.
• Gary Klein et al. — Ten Challenges for Making Automation a “Team Player” in Joint Human-Agent Activity. IEEE Intelligent Systems 19(6): 91–95, 2004.
• Thomas A. Limoncelli — Automation Should Be Like Iron Man, Not Ultron. ACM Queue, 2015 — free online.
• David D. Woods et al. — Behind Human Error, 2nd ed.. Ashgate, 2010.
• Dan Slimmon — Do-nothing scripting: the key to gradual automation. 2019 — free online.
• Meta Engineering — Leveraging AI for efficient incident response. 2024 — free online.
• Nicholas Marcott, Eduards Sidorovics, Paarth Chothani et al. — Genie: Uber’s Gen AI On-Call Copilot. Uber Engineering, 2024 — free online.
• Saurabh Jha et al. — ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks. ICML, 2025 — free online.
• Amazon Web Services — Security Update for Amazon Q Developer Extension for Visual Studio Code (Version #1.84). 2025 — free online.
• John Hughes — How we built Claude Code auto mode: a safer way to skip permissions. Anthropic Engineering, 2026 — free online.
• DevOps.com — When AI Goes Really, Really Wrong: How PocketOS Lost All Its Data. 2026 — free online.
• The Register — Vibe coding service Replit deleted user’s production database, faked data, told fibs galore. 2025, via the AI Incident Database — free online.