Operating Production Systems in the AI & Cloud Era

← the whole book on one page

Chapter 5: Incident Response — Command, Communication, and On-Call

The fire does not read the org chart.

Everything in this book so far has happened in the calm: in reviews, drills, dashboards and the replay of an investigation on a quiet afternoon, when there was room to think before acting. An incident is the moment design time ends. Users are being hurt now, and the people who must help them were, a minute ago, drinking coffee, sitting in a meeting or asleep. In the space of a few minutes those people have to become a team. They have to agree on what is happening, and then change a running system that is under load, usually before anyone understands why it is failing. What makes the first 10 minutes so hard is rarely a shortage of technical skill. The hard problems are problems of coordination: who decides, who acts on the system, who speaks to the people outside the room, and who writes down what was done and when. A response that leaves any of those questions open will spend its best people colliding with one another, however talented they are.

This chapter’s home touchstone comes from a field where far more is at stake than a ticket sale. In the autumn of 1970 a cluster of wildfires burned across Southern California for nearly two weeks. Sixteen people were killed and more than 700 homes destroyed. When the fire services looked back afterward, they found that the limiting factors had been neither courage nor equipment. They had been organizational. Agencies used different words for the same things, so an order could mean one thing to the crew that gave it and another to the crew that received it. They ran on different radio frequencies, so crews fighting the same fire could not hear one another. Nobody could reliably say who was in charge of a given flank of a fire. And there was no common way to grow a response smoothly from a single engine to a hundred, so every expansion had to be improvised by people who had never planned it together. Congress funded a cooperative program to fix this, called FIRESCOPE (Firefighting Resources of California Organized for Potential Emergencies), and out of it, through the 1970s, came the Incident Command System (ICS): a standard way to organize an emergency response of any size among people who may never have worked together before. In 2004 ICS became the backbone of the United States’ National Incident Management System.

It is worth pausing on what that history teaches before we borrow from it. The fire services learned that a response can fail even when every individual in it is brave and competent, because the failure lives in the spaces between people: in the vocabulary they share or lack, in the channels they can or cannot hear, and in the unanswered question of who has authority. Software teams tend to learn the same lesson the hard way, often after an outage in which several capable engineers each changed a different thing at the same moment and nobody knew which change had helped. The analogy has limits, and we will meet them as the chapter goes on. A failing checkout rarely puts anyone’s life at risk, and a software change can often be rolled back in a way a burned hillside cannot. But the shape of the coordination problem is the same, and so the shape of the solution carries over remarkably well.

Software did borrow it, a generation later and with less ceremony. The incident management described in Site Reliability Engineering (2016) is explicitly modeled on ICS, and PagerDuty’s published response process uses the same roles. Encore is about to need every one of those roles. It is Friday, a few minutes before 10:00 CET, and 1.1 million fans are waiting in the Sale Gate, Encore’s virtual waiting room, for the on-sale of Vesper Hale’s 12-date European tour. Readers of the Resilience book left Encore at exactly this moment, with the preparations finished and the readiness review signed off. (Chapter 1’s map of Encore shows where each piece of the system sits.) We will follow the next 47 minutes section by section, watching the incident being detected, declared, commanded, communicated and brought under control. The question of why it happened at all we leave to Chapter 6, because that question deserves a calmer room than the one we are about to enter.

5.1 Anatomy of an Incident

What makes something an incident

Before we can respond well to an incident, we have to agree on what one is, and the answer is less obvious than it looks. An incident is an unplanned disruption, or a credible threat of one, that needs a coordinated response beyond the routine work of a single on-call engineer. The interesting word in that definition is coordinated. We might expect an incident to be defined by how much damage is being done, but the definition turns on how many people must act together to stop it. Consider a disk filling up on a database replica at 3 a.m. One engineer can see it, understand it and fix it, and so it is a task, however annoying. Now consider the same disk filling up on a seat-ledger replica during an on-sale. Someone must free the space, the On-Sale team must decide whether to slow the Gate, someone must tell organizers what fans are seeing, and all of those choices must fit together. That is an incident, because several teams must act at once and someone must keep them pointed in the same direction.

A hospital draws the same distinction between a routine admission and a trauma call. A broken wrist is serious to its owner, but one doctor can treat it; a patient with several injuries arriving by ambulance sets off a call that brings a surgeon, an anesthetist and nurses to the same bay, each with an assigned part. What changes is the shape of the work, which suddenly needs more than one pair of hands and someone to direct them. The analogy holds well up to a point. Where it breaks is that a hospital rarely doubts that a trauma patient has arrived, while in software the hardest moment is often deciding that what we are looking at is an incident at all.

That is why a definition is not enough. A definition tells us what an incident is; a tired engineer at 3 a.m. needs to know when to call one, and for that we need a trigger, a test simple enough to apply without deliberation. Google’s SRE book supplies one that has held up well. Declare an incident if any of three things is true: you need a second team to help; customers can see the problem; or the problem is still unsolved after an hour of focused work. Any one of the three is enough. Each criterion is a concrete sign that the work has outgrown one person: a second team means coordination across groups, customer visibility means someone must communicate outward, and an hour without progress means the problem is harder than it first looked. Encore adopted the test and added a clause of its own: during an on-sale, any symptom on the purchase path counts by itself. The reason is plain arithmetic. An on-sale’s busiest stretch is measured in minutes, and a rule that lets a problem run for an hour would allow the whole sale to fail before the third criterion fired.

Once declared, an incident is on a path that every incident follows, through five phases, and naming them helps us know where we are. First, it is detected, by an alert, by a customer, or by an engineer who notices something odd. Second, it is declared: someone judges its severity and says out loud that this is an incident, which gives it a name, a channel and a commander. Third, it is mitigated, meaning its harm is stopped or reduced; this usually happens before anyone understands the cause, a point Section 5.3 develops. Fourth, it is resolved, which we can define practically as the moment nobody needs to watch the system by hand any more. Fifth, it is followed up, through the review that is the subject of Chapter 6. Two things do not fit neatly into this sequence, and it is worth saying so. Diagnosis, the work of understanding why, runs underneath every phase rather than occupying one of its own. And severity is not fixed at declaration; it is revised as the picture sharpens.

Severity as a shared vocabulary

The act of declaring an incident depends on words that everyone has agreed on before anyone needs them. A severity level is such a word. Its purpose is economy: one short label, spoken in a channel, sets a whole response in motion, so that nobody has to negotiate in the middle of a crisis who should be woken, who should be told and how quickly. Encore’s levels are deliberately short, and they are written in terms of harm to fans and organizers. Components are never mentioned. A definition such as “the inventory database is down” would force the person declaring to diagnose first; a definition such as “fans cannot buy during an on-sale” can be judged from the symptom alone, which is all that anyone has in the first minutes.

Figure 5.1 — Encore’s severity levels. Each level is defined by harm and answered by a response.

Read the table from left to right and it becomes a kind of contract. The middle column describes what the person declaring can observe, and the right-hand column describes what the organization promises to do about it. A SEV1 brings a trained incident commander from the shared rotation within 5 minutes, a communications lead, a public status page within 15 minutes and word to leadership. A SEV4 earns a ticket and nothing more. The person holding the pager only has to match what they see to a row; the table does the rest.

Google’s and PagerDuty’s guidance both insist on two habits in using such a scale. The first is to declare early and downgrade freely, taking the higher level whenever we are unsure. The reasoning rests on an asymmetry of costs. A needless declaration costs perhaps 20 minutes of mild embarrassment, as people join a channel and then quietly leave it. A missing declaration costs the first 20 minutes of the response, which are usually the minutes that matter most. The second habit is to treat severity as provisional. The number exists to summon the right people, and once they have arrived, an argument about whether this is “really” a SEV1 or a SEV2 only delays the harder work of stopping the harm. We can settle the label afterward, in the calm of the review.

Friday, 10:04

With that vocabulary in hand, we can return to Encore. At 10:00 the Gate opened and began admitting fans at 1,500 per second per event, the rate the On-Sale team had raised it to on Wednesday. At 10:04 the burn-rate alert on the on-sale-window service level objective (SLO), which asks that 99% of admitted fans finish without an error page in the first hour of a sale, paged Rui, the On-Sale team’s primary on-call. A burn-rate alert fires when the error budget is being spent much faster than the objective allows, so the page told Rui that fans were already failing in numbers that mattered. When he looked, he saw failures in several places at once: “seat unavailable” errors on holds, rising checkout latency, and a retry rate that climbed while he watched.

Rui’s first move was not to touch anything. He spent 60 seconds judging the scope of the problem, and the reason for that pause deserves care, because it is easy to misread. Had the errors begun right after a deploy by his own team, the right move would have been to roll that deploy back first, without waiting to judge anything; the pause is for sizing the problem, and it should never delay an obvious, safe mitigation. But nothing of his had changed. The errors spanned every shard, which meant this was not one bad component misbehaving. And two of the declaration criteria were already met. At 10:06 he paged the incident commander rotation. At 10:07 Marta, a staff engineer from Orders & Payments who was on the rotation that week, declared a SEV1 and took command. Seven minutes after the Gate opened, the incident had a name, a channel and one person whose only job was to run it. Section 5.2 takes up what that job is, and where it came from.

Recap. An incident is defined by the need for coordination rather than by the amount of damage, and a concrete test (a second team, customers who can see it, an hour without a fix) turns that definition into something a tired person can use. Severity is a vocabulary of harm agreed in advance, so that one word summons the right response. We declare early, downgrade freely, and save the argument about the level for afterward.

Exercise 5.1. Take your team’s last three significant alerts. For each one, note when it was detected, when (if ever) it was declared, and at what minute the three declaration criteria were first met. Where declaration came late, ask what the person holding the pager was waiting for.

5.2 Roles, Command, and Communication

What the firefighters learned

At 10:07 Marta took command, and the obvious next question is what, exactly, she had taken. To answer it we go back to the fire services, because the Incident Command System is a set of answers to specific failures, and each of its ideas makes more sense once we can see the failure it was built to prevent. There are four of them, and a fifth that sits behind them all.

The first is common terminology: every agency calls the same resource by the same name. It answers the first of the 1970 failures described at the start of the chapter: when agencies used different words for the same things, a request for help could be misunderstood, or answered with the wrong resource, by the very people it was meant for. When people who have never worked together must act within minutes, shared words are the cheapest coordination there is. The second is modular organization: a small fire and a large one are run in the same shape, and a response grows by adding standard units to that shape rather than by improvising a new structure halfway through. Nobody has to learn a different organization at the moment the fire gets bigger.

The third is span of control, the number of people one supervisor directs. ICS holds it to the number a person can actually keep track of, conventionally three to seven, with five as the target. When a response outgrows that span, it adds a layer of supervisors instead of asking one person to watch twelve. The fourth is unified command: when several agencies each hold jurisdiction over part of an incident, as a county, a state and a federal forest service might on a fire that crosses their boundaries, they command it jointly and work from one plan, so that no crew receives two incompatible sets of orders.

Behind all four sits the idea that matters most, and the one software teams most often get wrong. Command is a role. It is assigned at the start of an incident, it is handed over explicitly when it changes hands, and it has nothing to do with rank. A more senior officer who arrives at the scene does not become commander simply by being senior; command passes only when it is deliberately transferred. I have come to believe this is the single most useful idea software has borrowed from the fire services, because it lets the person best placed to run the response run it, and it gives everyone else a clear answer to the question “who decides?”

Software did not have to work out the translation alone. Rob Schnepp, Ron Vidal and Chris Hawley, veterans of the fire service and of incident command, mapped these ideas onto software operations in Incident Management for Operations (2017). Most of the mapping carries over directly, but one idea needs adjusting. A software company rarely shares jurisdiction over its own systems the way three fire agencies share a hillside, so unified command in its full form seldom applies. The case that comes closest is an incident involving a partner. When the payment service provider (PSP) is in the same incident, each side commands its own systems, and the two responses meet through ICS’s liaison officer, the role that connects one organization’s response to another’s. The two sides then agree on joint objectives without either one giving orders inside the other’s systems.

Figure 5.2 sets out the whole mapping: each ICS idea, the failure of 1970 it answered, the form it takes in a software incident, and the role that fills it at Encore.

Figure 5.2 — From wildfire to on-sale. The roles are ICS’s, scaled down; the reasons are the same.

Read down the third column and a pattern appears. Every role exists to keep one kind of work from crowding out another. The ops lead directs the hands-on changes to production. The comms lead speaks to everyone outside the response. The scribe keeps the written record of what is known and what has been decided, with Cue, Encore’s operations agent, drafting a timeline the scribe can correct. And the incident commander holds the whole picture and makes the decisions. Separating these roles matters because each of them, done properly, takes all of one person’s attention.

The commander’s role is the least intuitive of the four, and for Encore the best way into it is through the theater. A stage manager “calls the show.” From a desk in the wings, wearing a headset and with the prompt book open, the stage manager gives every cue: lights up, sound in, fly the backdrop out. Yet the stage manager operates none of those things. The electricians, the sound engineers and the fly crew do, each at their own station, waiting for the call. The stage manager’s value lies in watching the whole stage at once and timing everything against everything else, which nobody with a hand on a fader can do. The incident commander is the stage manager. PagerDuty’s response documentation says it bluntly: the commander is not a resolver. So Marta would not be the one running kubectl at 10:07. Her hands were free so that her attention could be.

The analogy breaks, and where it breaks is instructive. A stage manager has a script. Every cue was written and rehearsed weeks before opening night, and the job during the performance is to execute it with perfect timing. An incident commander has no script. Marta had to work out the cues while calling them, deciding what the team should try next while also keeping track of what it had already tried. That is a harder job than the stage manager’s, which is all the more reason to give it to someone with nothing else to do.

Handoffs and the collapse of roles

The roles only help if everyone knows who holds them at every moment, and that is harder than it sounds, because command changes hands more often than people expect. The first responder hands to the rotation’s commander, as Rui did to Marta. A long incident outlasts one person’s alertness, a commander in one time zone goes off shift, or a more experienced colleague arrives. Each handoff is a gap through which the team’s shared picture of the incident can fall: the new commander may not know what has been tried, and the team may not be sure who is now deciding.

The ICS answer, which Google’s Site Reliability Engineering book adopts almost word for word, has two parts. First comes a short spoken briefing that gives the incoming commander the state of the incident. Then comes an explicit transfer, a sentence such as “you’re now the incident commander, okay?”, which the outgoing commander does not consider complete until they hear it acknowledged. The acknowledgment is the important part. Without it, there is a window in which each of two people believes the other is in charge, and in that window nobody is.

Rui’s briefing to Marta at 10:07 took under a minute: admission at 1,500 per second, errors on holds, retries rising, nothing changed yet. It is worth noticing how much those four items carry. They tell the new commander the load on the system, the symptom fans are seeing, the direction things are heading, and, perhaps most usefully, that no one has yet made a change whose effects she would need to untangle. A good briefing is short because it has been chosen, and its last item is often the most valuable.

What happens when roles are not held is the subject of one of the most searching studies in the literature on organizations. In 1993 Karl Weick re-examined the Mann Gulch fire of 1949, in which 13 firefighters died in Montana. Weick argued that the crew’s ability to make sense of their situation collapsed along with the structure of their roles. As the fire overtook them, the crew stopped functioning as an organized crew with a leader, and so, when the foreman lit an escape fire and told them to lie down in its burned-out ground, none followed. His instruction was sound, but it no longer came from a role the others recognized. Weick’s point was that roles do more than divide up work: they hold together a group’s shared understanding of what is happening, and when the roles dissolve, the understanding tends to go with them.

The stakes in software are far lower, but the pattern is familiar to anyone who has watched eight senior engineers in one incident channel, each pursuing a private theory. Every one of them is capable, and each is doing something reasonable, yet nobody is assembling the theories into one picture or deciding which to test first. The channel fills with fragments and the response stalls. The remedy is the one ICS prescribes: somebody holds command, and everyone else knows it.

Communication: inward, upward, outward

Holding the roles keeps the team coordinated, but an incident also has to be explained to people outside the team, and there are three audiences, each with different needs.

The first audience is the responders themselves. They need one channel where facts are stated and decisions are recorded, so that nobody has to reconstruct the situation from side conversations. Once three or more people are working the incident, they also need a live document beside the channel, saying what is known, who owns which line of work and what has been decided. A channel is a stream; it records everything in order, and the important facts scroll away under the chatter. The document is the place where the current state lives. The second audience is stakeholders inside the company: leadership, Support, and the account managers who look after Encore’s tenants. They need plain summaries that tell them what is happening without making them read the channel, and that keep them from wandering into it to ask. The third audience is the people affected, here the fans, and they need an honest public statement of what is wrong and when they will hear more.

Where that public statement lives matters more than it first appears. When Amazon S3 failed in the us-east-1 region in February 2017, AWS could not update its own status dashboard for the first part of the outage, because the dashboard itself depended on S3. The tool for announcing the failure had failed along with the thing it was meant to report on. Encore took the lesson. It rents a hosted status page from a provider that shares none of Encore’s DNS, content delivery or cloud regions, so that whatever breaks inside Encore, the page that says so can still be reached.

At 10:14 Elif, the comms lead from the Reliability team, posted the first update. It said three things: some fans were seeing errors when selecting seats; places in the queue were safe; and the next update would follow within 15 minutes. Each of those statements did a job. The first was honest about the harm. The second answered the question every fan in a queue is actually asking, which is whether to stay. The third told them when to look again, so they did not need to keep refreshing. Stakeholders inside Encore received summaries on the same cadence.

There is a discipline in that third statement worth making explicit. Every message gave the time of the next update, and none gave a time for the fix. The reason is that only the first is a promise the comms lead can keep. Elif could guarantee that she would post again by 10:29; nobody in the channel could guarantee when the errors would stop. A promised fix time that slips damages trust twice, once when it is missed and again in every later message, which readers now discount. A promised update time that is kept builds trust, even while the news itself is still bad.

Communication also runs in the other direction, inward from the outside, and that was where Encore’s incident turned. At 10:24 Elif relayed something from Support that no dashboard had shown: fans were writing in to say the site had told them they were bots. Nothing in the telemetry the responders were watching would have revealed that, because the fans being turned away were, from the system’s point of view, a security control working as designed. Support, at that moment, was acting as a sensor, one that measured what fans were experiencing rather than what the system reported. And the comms lead was the only role positioned to connect that sensor to the response, because she was the one person in regular contact with both Support and the incident channel.

Figure 5.3 — The Vesper Hale response in ICS shape: command at the top, three roles beneath it, and the comms lead standing at the channel’s edge. Notice the one line that runs inward; it carried the fact no dashboard had shown.

Recap. The Incident Command System gives us four roles: a commander who decides and does not operate, an ops lead who directs the changes, a comms lead who speaks for the response, and a scribe who keeps the record. Around them sit span of control, which splits a growing response into workstreams; explicit, acknowledged handoffs, so that someone is always in charge; and a liaison for partners who command their own systems. Communication flows outward to three audiences on a promised cadence, never promising a fix time, and it also flows back in, often carrying the very fact the responders were missing.

Exercise 5.2. Reconstruct the roles in your last significant incident. Who was commanding, who was operating, who was communicating and who was recording? If one person held two of those roles, find a moment when that cost something: a decision delayed while they typed, an update that went out late, or a fact that reached the channel but never reached the person deciding.

5.3 Mitigate First — Situational Awareness Under Pressure

Three levels of seeing

By 10:07 Encore’s response had a commander, a channel and a shape. Structure is necessary, but it does not tell anyone what is actually happening. The roles of Section 5.2 exist so that a group of people can come to see the situation together and act on what they see, and to talk sensibly about that seeing we need a word for it. Mica Endsley supplied one in her 1995 theory of situation awareness, and it names precisely what an incident team is trying to build.

Situation awareness is the perception of the elements of an environment, the comprehension of what they mean, and the projection of what they will do next. Endsley’s three levels are worth taking one at a time, because each asks something different of the people in the channel. Perception is noticing the signals at all: at Encore that morning, the “seat unavailable” errors on holds, the rising checkout latency and the climbing retry rate that Rui saw at 10:04. Comprehension is fitting those signals together into a single account of what is going on, so that three alarming graphs become one problem with a shape. Projection is running that account forward a few minutes: if this continues, what happens next, and what will our own actions set in motion?

The levels are cumulative, and a team can stall at any of them. It can perceive every alert and still fail to comprehend them together, treating each graph as its own emergency. Or it can comprehend the present correctly and still fail to project, missing, say, that the queue will empty in four minutes and send a second wave of fans into a system that has not recovered from the first. Each level rests on the one beneath it, and none comes for free.

This is where it helps to be honest about what our tools do. Incident tooling, from dashboards and alerts to traces and log search, serves mostly the first level. It is very good at putting signals in front of people. Comprehension and projection happen in people’s heads, and heads under pressure are unreliable instruments. That is why a shared picture matters so much: the live incident document, the scribe’s log, the commander’s habit of restating what is known. When one person’s comprehension is written where others can see it, it can be checked against someone else’s, and a wrong account has a chance of being caught before the team acts on it.

Fixation

The characteristic failure of situation awareness under pressure is fixation: a team locks onto one explanation and stops seeing evidence that contradicts it. Fixation is not carelessness. It usually starts with an explanation that is true, or partly true, and it grows because a good explanation is a relief: it gives an anxious group something definite to work on, and once the work has started, the attention that would have noticed contrary evidence is busy elsewhere.

Aviation learned this at terrible cost. On 29 December 1972 the crew of Eastern Air Lines Flight 401, approaching Miami, became absorbed in a burned-out landing-gear indicator light. While they worked the problem, the autopilot’s altitude hold was inadvertently disengaged, and the aircraft descended slowly into the Everglades, killing 101 people. The bulb itself was harmless. The danger lay in how absorbing it was: every member of the crew was attending to the puzzle, and nobody had been assigned to fly the airplane. What makes the story so instructive for us is that nothing about the bulb was a mistake to investigate. The mistake was letting the investigation take everyone’s attention, so that the one question that mattered, is the aircraft still where we think it is?, went unasked.

Encore’s fixation began at 10:11, and it began, as fixations often do, with something true. Cue, the operations agent, posted that the platform’s upgrade reconciler had drained a node at 10:02, and that at that same moment Seat Inventory shard 3, one of the four partitions selling the tour’s seats, had failed over its leader. Both facts were correct, and they arrived as a tidy story: a node went away, a shard lost its leader, writes were rejected, fans saw errors. The channel’s attention gathered around shard 3.

The trouble was that the story explained the past and not the present. Jonas, Seat Inventory’s on-call, confirmed the failover and its 40 s of rejected writes, but that window had closed at about 10:03, and errors were still rising, not on shard 3 alone but on all four shards. A failover that had finished eight minutes earlier on one shard could not account for a problem that was growing on every shard now. The evidence against the shard 3 story was already on the dashboards; the difficulty was that a team busy confirming a story is poorly placed to notice it.

Marta’s question at 10:16 broke the fixation: “What is still broken now, and is it getting better or worse?” It is worth noticing what kind of question that is. It asks for no theory and does not argue with Cue’s or Jonas’s findings. It pulls the team back from explanation to perception and projection, to what the signals say at this minute and which way they are heading. This is the incident commander’s version of flying the airplane, and it is one reason the commander keeps their hands off the keyboard: someone has to stay free to ask it. Once the question was asked, the answer was plain. The failover explained how the trouble had started; something else was keeping it going.

Mitigation before diagnosis

Marta’s question leads directly to a principle that is old in medicine and easily forgotten in engineering: stop the bleeding before you look for its cause. An emergency physician restores circulation in a patient in shock before asking why the patient is in shock, because a correct diagnosis delivered too late helps no one. The physician is not indifferent to the cause; she knows that the cause can be pursued only while there is still a patient to pursue it in.

In an incident, mitigation is any action that reduces user harm without requiring that we understand why the harm is happening. Google’s SREs call the best of them generic mitigations: shedding load, rolling back, failing over, disabling a feature. What makes these generic is that they work across many causes. We do not need to know why the system is struggling to know that asking less of it will usually help, or that returning to the last version that worked will usually help if a change is to blame. The best generic mitigations are cheap to try, easy to reverse, and dependable only if they have been rehearsed. That last condition is the one teams most often neglect. A mitigation nobody has practiced is a fresh experiment run on production at the worst possible moment.

The medical analogy holds well, but it breaks in two places worth noticing. First, we think of stabilizing a patient as preserving the evidence of what went wrong, whereas some software mitigations destroy it: a restart clears the very state an engineer might have inspected, and a rollback can hide the change it undoes. Stopping the bleeding is still right, but a good responder notes what the system looked like before the mitigation changed it, so that the diagnosis can continue afterward, in the review of Chapter 6. Second, a software mitigation is itself a change to a running system, and it can carry costs of its own. Both of Encore’s mitigations that morning show the second point.

The first came at 10:19. On Rui’s proposal and with Marta’s decision, the Gate’s admission rate was lowered from 1,500 to 900 fans per second. The reasoning did not depend on knowing the cause. Fewer fans admitted each second meant fewer seat holds attempted each second, which meant less load on Seat Inventory; fewer failed holds meant fewer fans retrying, which eased the retry storm that was feeding on those failures. The cost was small and understood: fans would wait a little longer in a queue they already expected to wait in. And the move was reversible in seconds, because the rate is a single setting on the Gate. Errors on holds began to fall within two minutes. Nothing was fixed, but the team had bought something valuable. It now had time to think.

The second mitigation came from the fact Elif had brought in from Support at 10:24, that fans were being told they were bots. Noor, Bot Screening’s on-call, joined at 10:27 and confirmed it. The velocity rule, which flags rapid repeated requests as bot behavior, had been firing on real fans who were retrying failed seat selections, exactly the behavior the first wave of errors had provoked, and it had blocked about 30,000 of them. The rule predated the policy, adopted that spring after an error-budget dispute between the On-Sale and Bot Screening teams (the Resilience book tells it in its chapter on service level objectives), that every block must offer a challenge and a way back; it had never been reviewed against that policy, so it still blocked outright, with no way for a wrongly blocked fan to prove otherwise.

At 10:31, with the IC’s approval, Noor disabled the rule, and the Gate re-admitted the blocked fans with their original queue positions restored. This was a conscious trade, and it is the clearest example in the incident of a mitigation with a price. Turning off a bot rule in the middle of a major on-sale opened a short window in which bots would meet less resistance. Against that stood certain harm to tens of thousands of real fans who had done nothing wrong except retry. The commander weighed a possible cost against a present one, chose, and the choice was recorded in the channel where everyone could see it.

By 10:47 the checkout burn rate had been within budget for several minutes, and Marta declared the incident recovered. Admission was raised back to 1,200 fans per second at 11:10, under watch, and the incident was resolved at noon. Note that the rate came back to 1,200, the level that had served before Wednesday’s change, not to the 1,500 at which the morning began; Chapter 6 explains why that difference matters. Throughout, no seat was sold twice. That fact is easy to pass over, but it is what made the morning’s boldness possible. The team could shed load, disable a rule and re-admit 30,000 fans quickly because it knew that fencing and idempotency would hold underneath whatever it did. Aggressive mitigation is affordable only on top of guarantees we trust.

Figure 5.4 — The on-sale as the Sale Gate saw it: admission rate per event, the 47 degraded minutes, and the response’s events beneath. Both mitigations were cheap moves on one dial and one switch, made long before anyone understood the cause.

The timeline below gathers the morning in one place, as the scribe’s log recorded it. Read the Phase column and you will see the shape this section has argued for: diagnosis ran alongside everything, but the two entries marked Mitigate came well before anyone could have said why the incident was happening. The first row, the Catalog deploy at 09:52, was noticed only afterward; nobody in the channel knew of it while the incident ran.

Figure 5.5 — The Vesper Hale on-sale as the scribe’s log recorded it. 47 minutes of degraded checkout; roughly 9% of admitted fans hit at least one error; no seat sold twice. Chapter 6 asks why each of these things was possible.

Recap. Situation awareness has three levels (perceiving the signals, comprehending what they mean together, and projecting where they lead), and our tooling helps mostly with the first; the other two live in people’s heads, which is why a shared picture matters. Fixation is the characteristic failure under pressure, and it usually begins with an explanation that is true but no longer sufficient. The remedy is a role whose job is to keep asking what is broken now and which way it is heading. Mitigation reduces harm before the cause is understood, and the best mitigations are generic, cheap, reversible and rehearsed, and safe to use boldly only on top of guarantees we trust.

Exercise 5.3. List the generic mitigations available for your most important service: what you can shed, roll back, fail over or switch off. For each, note how long it takes, who may authorize it, what it costs users while it is in effect, and when it was last practiced.

5.4 Tooling for Response — Runbooks, ChatOps, and AI Assistance

Runbooks and the channel

Everything in the last two sections was about people: who commands, who acts, who speaks, who keeps the record, and how they keep from fixing on the wrong story. Tools come into it only now, and that order is deliberate, because the tooling for incidents should serve those roles. A tool that asks the commander to operate it, or that scatters the record across five consoles, makes the work harder on the very morning it was bought for.

The oldest of these tools is the runbook, and a good incident runbook is written differently from the reference documentation most teams keep. It starts from a symptom, because a symptom is what the on-call engineer actually has at 10:04 (“errors on seat holds are rising”); the cause comes much later, if it comes at all. It offers the generic mitigations of Section 5.3 before any diagnostic steps, so that the first page of the runbook helps reduce harm while the later pages help explain it. And it names who may authorize each action, so that nobody has to stop in the middle of an outage and work out whose decision this is. All three properties showed on Friday. When Rui proposed “lower admission at the Gate”, the move was already written down under the symptom the team was seeing, and the runbook said the incident commander could approve it. Proposal and approval took under a minute, because every question the team might have argued over had been settled calmly, weeks earlier, by people who were not under pressure.

A runbook says what may be done. The incident channel is where it is done, and that brings us to the second tool. ChatOps is the practice of running operational work through commands typed into a shared chat channel, so that each action and its result are visible to everyone and recorded as they happen. Its value in an incident is less about convenience than about witness. A command typed in the channel is seen by the incident commander as it is issued and logged by the scribe without anyone having to remember it afterward. Compare a change made from an engineer’s own terminal: it may be the right change, but the commander learns of it late or not at all, and the timeline later has a hole exactly where the most important decision went. Encore’s incident commands look like this:

/incident declare sev1 "Vesper Hale on-sale: checkout errors"
/incident role ic @marta
/incident role comms @elif
/gate admission set event=vh-2026 rate=900   # requires IC approval
/statuspage draft "Some fans are seeing errors when selecting seats..."

Each line does a piece of the work Section 5.2 assigned to a role. The first two declare the incident and name its commander, the third names the voice that speaks to stakeholders, and the fourth is the mitigation itself, guarded so that it runs only with the IC’s approval. The last line is the most instructive: it drafts a status-page update and does not publish it. As in PagerDuty’s process, a public post needs the commander’s approval, because what Encore says to 1.1 million waiting fans is a decision about the incident and belongs to the person running it. The tool makes the comms lead faster; it does not let anyone skip the commander.

Cue at 10:11

The newest tool in Encore’s channel is not a command but a participant. Cue, the operations agent first introduced in Section 3.5, whose 10:11 findings we have already watched the channel fix upon, joins every SEV1 and SEV2 channel. It reads telemetry, deploy history, runbooks and past postmortems, and it answers questions in the channel the way a well-read colleague might.

What matters most about Cue is where its limits come from. What it may do is fixed by the platform, not by the instructions in its prompt, and that distinction carries the whole design. Instructions can be argued with, misread or overridden by a cleverly worded input; a platform control cannot. So Cue’s only actions come from a catalog that the platform enforces. When Cue wants to act, it posts a plan in the channel, a named person approves it there, and only then does the platform issue a token that lasts five minutes and is scoped to that one action on that one target. Cue holds no standing permission to change anything. Just two small actions may run without approval: raising Catalog’s autoscaler floor and holding a node in the upgrade reconciler. A restart of Seat Inventory was not one of them, which is why what follows was a suggestion rather than an event.

At 10:11 Cue did two things. They are worth looking at separately, because they line up closely with what tools of this kind have and have not been shown to do.

The first was genuinely helpful. In about four minutes, Cue had correlated the alert with the platform’s change log and with the shards’ leader history, and it posted what it found: the reconciler had drained a node at 10:02, and Seat Inventory shard 3 had failed over its leader at that moment. Correlating alerts with recent change is an old and proven move. Most outages follow a change, so the first useful question in almost any incident is “what changed?”, and on-call engineers have been asking it of deploy logs since long before language models existed. What is new is asking a language model to do the correlating, and here the published accounts are real but modest. Meta reported in 2024 that its ranker put the responsible change among its top five candidates in 42% of the investigations it was tested on. Read that number carefully: it is a shortlist of five that contained the right answer a little less than half the time. That is useful to a human investigator and far from an oracle. Microsoft’s RCACopilot (EuroSys 2024) was evaluated on a year of incidents and deployed with one team. Cue showed the limit of the technique too. The Catalog deploy at 09:52, which had emptied the seat-map geometry cache, fell outside the time window Cue searched, so it never appeared in Cue’s answer. Cue was right about what it found and blind to what it did not look for, and Section 5.3 showed what came of the channel’s attention gathering around shard 3. Chapter 6 measures that blind spot.

The second thing Cue did was suggest a fix: restart the Seat Inventory pods to clear what it called an unhealthy state. It had learned the move from two old postmortems, both about stateless services, where a restart is cheap and often works, because a fresh pod picks up a clean state and loses nothing. Seat Inventory is not stateless. Each of its shards has a leader, and restarting the pods would have forced leader elections on every shard, each election with its own window of rejected writes, in the middle of the busiest on-sale of the year. The suggestion was plausible, and it was wrong. Jonas, one of the few engineers who had run a shard failover by hand (at a game day the quarter before), declined it in one line.

This was Bainbridge’s irony, from Chapter 3, at full strength. Automation leaves people with the hardest part of the job, and with less practice at it. The safeguard against a fluent agent turned out to be a person skilled enough to judge its suggestion, at exactly the moment when the agent’s speed and confidence made it tempting to go along. Encore was lucky that Jonas had done a manual failover recently enough to know what a restart would cost; most of his colleagues had never done one. The lesson I draw is a narrow one, and I think it is durable: an assistant can be right about the facts and still wrong about the action. Cue’s correlation was accurate. Its recommendation took accurate facts and applied a pattern from a different kind of system.

The approval gate that held at 10:11 is not only Encore’s caution; there is production practice behind it too. Google’s SREs described in early 2026 how Gemini CLI works through outages using typed tools that carry risk metadata, with a human confirmation for every change and an audit trail of what was done. Their worked example was a simulated outage, and the practice is young, but its shape is the same as Encore’s: the agent proposes, a person decides, and the system records. The other approach, letting an agent remediate unattended, has no credible production account behind it. In IBM’s ITBench benchmark (ICML 2025), agents resolved only 11.4% of the site reliability engineering scenarios they were given. So Encore keeps Cue to suggestions and to approved catalog actions. Since that Friday each action in the catalog also carries a blast-radius tag, which tells the approver, before anything runs, how much of the system the action could disturb. The tag puts in front of every approver the question Jonas answered from experience: if this goes wrong, how much goes wrong with it?

Cue’s most valuable work that morning drew the least attention. It drafted the incident timeline as events unfolded, which the scribe corrected into Figure 5.5, and it drafted stakeholder summaries for Elif to edit and send. This is where the evidence is firmest. Google’s security team reported in 2024 that incident summaries drafted by a model were written 51% faster and rated higher in quality, with a person approving each one, and Datadog describes the same draft-then-correct pattern for postmortems. The reason it works is not mysterious. Drafting a summary is slow, tedious work for a tired person, while checking a draft is quick and plays to what the person knows. A wrong sentence in a draft costs a correction, but a wrong action against a live shard costs an outage. That difference in the cost of error explains a good deal about where these assistants have so far earned their place.

Recap. Good runbooks start from a symptom, lead with generic mitigations and name who may authorize each action, and ChatOps makes every action visible to the commander and part of the record. AI assistance in incidents is young, but it has real accounts behind it for correlating alerts with change and for drafting timelines and summaries that a person corrects. Its actions belong in a catalog the platform enforces, each one approved by a named person and scoped to one target; autonomous remediation remains unproven.

Exercise 5.4. Pick one action your tooling or assistant could take during an incident: a restart, a rollback or a scale change. Write down the conditions under which that action would make things worse. Then check two things: whether the approval step forces a human to consider those conditions before approving, and whether anything other than the approval stops the tool from acting.

5.5 On-Call That People Can Sustain

Load and its measures

Everything we have watched so far in this chapter, the page at 10:04, the briefing, the command, the mitigation, depended on a person who was awake, reachable and clear-headed when the phone rang. That person is the subject of this last section. On-call is where incident response stops being a question of process and becomes a question of labor: how much of someone’s life the pager may claim, how often it may wake them, and what the organization owes them in return. The roles of Section 5.2 can be drawn on a whiteboard in an afternoon, but they are only as good as the rested, willing people who fill them week after week, and a rotation that wears those people down will eventually fail at exactly the moment it is needed.

The Google SRE book offers two reference points that have aged well. The first is that no more than 25% of an engineer’s time should be spent on call. The second is that a 12-hour shift should bring about two incidents at most, because each real incident carries hours of follow-up beyond the page itself: investigating, writing it up, fixing the alert or the code, and catching up on the work it interrupted. Two incidents in a shift, each with its tail of follow-up, already fill that shift. Other teams will reasonably choose their own targets, but whatever the targets, the load should be measured with the same seriousness we give an SLO. Four measures do most of the work: pages per shift; pages that arrived outside working hours; the share of pages that needed someone to act; and time to acknowledge. Each is best read as a distribution over many shifts, because a single team’s averages are noisy, and one brutal week can hide inside a comfortable mean.

The third of those measures deserves a moment of its own. A page that needed no action is a defect in the alerting. It woke someone, or interrupted them, in order to tell them something they could do nothing about, and each such page teaches the person carrying the pager that pages can be ignored, which is the most dangerous lesson an alerting system can teach. The cure is the paging discipline of Chapter 4: page only on symptoms that a human must act on now, and send everything else to a ticket or a dashboard.

The 25% cap sounds like a policy, but it carries an arithmetic consequence that shapes how teams are organized. Suppose a rotation has a primary and a secondary each week, so two people are on call at any moment. For each engineer to be on call no more than a quarter of the time, that is two weeks in every eight, the rotation needs at least eight people. When two sites split the day between them, each site holds the pager for only half of every day, and the SRE book puts the minimum at six people per site. Now consider a typical stream-aligned team of five engineers. On its own it cannot meet the cap: with two people on call every week, each of the five would carry the pager for two weeks out of every five, which is 40% of their time. The numbers leave only two honest choices, which are to relax the cap or to share the rotation with someone else.

Encore chose to share. Each team runs a weekly primary and secondary, but small teams are paired with neighboring teams whose systems they understand, so that no rotation has fewer than eight people. Alongside these team rotations, Encore keeps a separate incident commander rotation of about 20 trained engineers, most of them senior, drawn from across the company. Every rotation answers to an escalation policy, the configuration that decides who is notified for an alert, in what order, and how urgently. It does two jobs without anyone having to remember them. It moves an unacknowledged page on to the next person by itself, and it decides, from the alert alone, whether a phone may ring at night:

escalation_policy: on-sale
steps:
  - notify: primary
    escalate_after: 5m        # unacknowledged
  - notify: secondary
    escalate_after: 10m
  - notify: on-sale-engineering-manager
urgency:
  high: [sev1-symptom, sev2-symptom]   # any hour
  low: [everything-else]               # working hours only

Read from the top, the policy says that the primary has five minutes to acknowledge; if they do not, the secondary is paged, and if the secondary is also silent for ten minutes, the On-Sale engineering manager is paged. The urgency block says that only alerts on SEV1 and SEV2 symptoms may wake anyone, and that everything else waits for working hours.

Even with a policy like this, the numbers can drift without anyone noticing. Rui had been paged twice in the small hours that week, mostly by pre-warm alerts, the kind that fire around the preparations for a scheduled on-sale. He responded well on Friday. That says something about Rui, and very little about whether the rotation was healthy, because a rotation is judged by its load over months, and a good performance on one morning tells us nothing about how many more such weeks he can absorb.

Follow-the-sun, compensation and handoffs

The night is the expensive part of on-call, and the most elegant way to remove it is geography. Organizations with two adequately staffed sites in distant time zones can run follow-the-sun rotations, in which each region holds the pager during its own daytime and hands it to the other region as its evening arrives, so that nobody is paged at night. The price of removing the night is that every shift boundary becomes a handoff. Each handoff needs the spoken briefing of Section 5.2 and a written note, because anything in flight, an open incident, a suspicious graph, a change half rolled out, must cross the boundary intact, and Section 5.2 showed how easily a shared picture falls through the gap between two people.

Encore cannot use this arrangement, because its engineers all work near CET; there is no second site whose daytime covers Encore’s night. It does the next best thing, which is to make the night as narrow as possible. Night pages are kept for SEV1 and SEV2 symptoms only, and that rule is set in advance as an urgency on each alert, as in the policy above. It cannot be left to the severity of the incident, because severity is assigned by people after someone has already been woken up to look; by then it is too late to decide whether the phone should have rung. And after any night page, the engineer who answered it is given time off.

Compensation is part of sustainability too, and here the law has something to say, though less than is often claimed. In the Matzak case (2018) the Court of Justice of the European Union considered a Belgian volunteer firefighter who, while on standby at home, had to be able to reach his station within eight minutes. The Court held that his standby was working time. The reasoning is easy to follow: a man who must be at the station within eight minutes cannot really go anywhere or do much with that time, so it is not his own.

Matzak marks the extreme, and it is often quoted as if it covered every on-call arrangement. It does not. In two 2021 rulings the Court narrowed it: standby counts as working time only where the employer’s constraints very significantly limit the worker’s ability to use their free time, and even then the classification does not settle what the worker must be paid. Few software rotations come near that line. An engineer with a laptop and half an hour to respond can go to dinner, visit family or sit in a cinema with the phone on vibrate, which is a different life from waiting by the door for the fire siren. What is settled is narrower and more practical. Answering a page is working time, and the 11 hours of daily rest that the Working Time Directive requires are owed after it, so an engineer who works through the night on an incident is not expected at their desk the next morning.

Encore pays a standby allowance on top of what the law requires. That is its own choice, and I think the right one, because a rotation that depends on goodwill is already failing. Goodwill is real, and most engineers give it freely the first time and the fifth, but it is spent each time it is drawn on, and an organization that treats it as an inexhaustible resource discovers its limit only when experienced people quietly leave the rotation, or the company.

Fatigue

Fatigue matters for a reason more specific than kindness: it degrades exactly the faculties incident response depends on. Section 5.3 described the first two. Comprehension is understanding what the signals mean. Projection is seeing where the system is heading. And beneath both lies the patience Rui showed in Section 5.1, spending 60 seconds establishing the scope of a problem before acting on it. A tired engineer can still type commands and read graphs. What goes first is the judgment that connects them, and the tired engineer is often the last to notice it has gone.

Touring crews know this well. The crew who loads out a show after midnight and drives through the night to the next city is not the crew you want rigging lights over an audience the next afternoon, and good tour managers build rest into the schedule as carefully as they build the route. The analogy holds as far as the schedule; it breaks at predictability. A tour knows its load-outs months in advance, while an incident arrives when it likes and lasts as long as it lasts, so on-call rest has to be granted after the fact and command has to be rotated during the event.

Long incidents therefore rotate command. In one incident recounted in the SRE Workbook, PagerDuty rotated its responders and its incident commander every four hours, and Encore adopted the same four-hour limit for the incident commander. The handoff at each rotation is the briefing of Section 5.2, which is one more reason to practice it when nothing is on fire.

Rest also matters once an incident is over. After Vesper Hale, Marta sent Rui and Jonas home at lunchtime and asked the On-Sale team’s secondary to take the pager for the weekend. Rui had been woken twice that week and then carried the opening minutes of a SEV1; Jonas had held his nerve when a fluent agent suggested a plausible restart. Neither needed to be told they had done well as much as they needed to sleep. It was a small decision, and I have come to believe that decisions of exactly that size, made consistently by the people who lead incidents, are what keep engineers on a rotation for years.

Recap. On-call load deserves to be measured as carefully as any SLO, through pages per shift, night pages, the share that needed action and time to acknowledge, read as distributions. The 25% cap implies rotations of at least eight people, or six per site across two sites, which small teams reach only by pooling with their neighbors. The law asks less of standby than is often claimed, but answering a page is work and the rest after it is owed; and because fatigue erodes judgment before it erodes skill, long incidents rotate command and the people who carried them are sent to rest.

Exercise 5.5. Pull the last quarter’s pages for your rotation. Compute pages per shift, the share that arrived outside working hours and the share that required action, and check the rotation’s size against the 25% arithmetic: how many people would it need, with its current pattern of primaries and secondaries, to keep each one on call no more than a quarter of the time? Then ask which single alert, removed or demoted to working hours, would most improve the numbers.

Kata: Storm Night

Your brief: “Brightwater Energy.” Brightwater is a regional electricity utility, and you are inheriting its customer platform. The platform serves 1.1 million accounts. It publishes an outage map that customers check when their lights go out. It listens to 800,000 smart meters, each of which sends a “last gasp” signal in the moment it loses power, so the platform can know about an outage before anyone phones it in. And it runs dispatch for 300 field crews, the people who drive out in the weather to restore the lines.

A winter storm brings 25 times the normal traffic. Brightwater does not face a storm alone: the grid operator’s control room and two vendors who supply field crews take part in every storm incident, so each one is shared among several organizations from its first minute.

Twenty-four engineers in four teams run the platform from two offices an hour apart. They share a single on-call rotation, which took 310 pages last quarter. That is between three and four pages a day, every day; 55% of them arrived at night, and most of them needed no action at all.

Last January, during a storm, the outage map froze for four hours. Throughout those hours it showed 2,000 customers without power, when the true number was 90,000. Nobody declared an incident for 70 minutes. Three engineers made changes to the platform that conflicted with one another, and customers learned when their power would be back from local radio, not from Brightwater.

Brightwater’s January is Encore’s Friday morning without the scaffolding this chapter built, and setting the two side by side is the best way into the kata. Encore declared within minutes, because Rui had a concrete test and a severity table that told him what a purchase-path symptom during an on-sale was worth; Brightwater waited 70 minutes, presumably because nobody had agreed what a frozen map was worth. Encore had one incident commander who did not touch production and an ops lead who did; Brightwater had three engineers each acting on a private theory, which is the collapse of roles that Weick described at Mann Gulch, at lower stakes but in the same shape. Encore’s comms lead posted to a status page that shared none of Encore’s infrastructure and promised only the time of the next update; Brightwater’s customers turned to the radio, which is what people do when the organization that ought to inform them falls silent. And Rui, paged twice in the small hours that week, still answered well, whereas Brightwater’s engineers have been answering pages at night for a whole quarter, most of them for nothing, and the storm will find them already tired.

One difference cuts the other way, and it is worth noticing before you begin. Section 5.2 observed that a software company rarely shares jurisdiction, so Encore met its payment provider through a liaison, each side commanding its own systems. A utility in a storm is much closer to the wildfire that gave birth to ICS. The control room decides what happens on the grid, the crew vendors decide where their trucks go, and Brightwater decides what customers see and how dispatch works. These are separate jurisdictions over one emergency, which is exactly the situation unified command was invented for. Your design has to say which decisions belong to Brightwater alone and which must be made jointly, and it has to say so before the next storm, because nobody negotiates a command structure well at two in the morning with the wind rising.

Deliverables:

1. Severity matrix and declaration test. Define four severity levels by the harm done, never by the component that failed. The harm that matters most falls on customers, and above all on those on the medical-dependency register, whose equipment needs electricity; it also falls on the crews, who rely on the platform to know where to go. For each level, state the response it sets in motion, as Figure 5.1 does for Encore, and write the concrete criteria by which a tired engineer decides to declare.
2. Role cards. Write one page each for the incident commander (IC), the ops lead, the comms lead and the scribe. Each card says what the role does, what it must not do, and how it hands over: include the spoken briefing for a handoff. Then describe how decisions are shared with the control room and the crew vendors, since they belong to the incident too.
3. Communication plan. Set the cadence and write the templates for each audience: the incident channel; the control room and the call center; the status page and the banner on the outage map; and SMS to customers. Say where the status page is hosted, and check that it cannot fail for the same reason the platform fails, remembering what happened to AWS’s own status dashboard when S3 went down in 2017.
4. Mitigation runbook. Write symptom-first steps for “the outage map is stale or wrong.” Start from what a responder can observe, not from a guess at the cause. Lead with generic mitigations, the ones that reduce harm whatever the cause turns out to be, and name who may authorize each step.
5. On-call redesign. Set rotation sizes and an escalation policy, choose targets for page load, list the alerts you would delete or demote to working hours, and write a compensation and rest policy. Section 5.5 gives you the arithmetic and the legal floor.
6. Tabletop script. Write a 45-minute exercise that replays last January’s storm with the new roles in place, so the team can rehearse the response before it needs it.

Rubric:

Three of these questions repay some thought before you start writing. The first is mitigation first. When Brightwater’s map froze, the team’s instinct was presumably to find out why, and every minute spent diagnosing was a minute in which 88,000 customers without power were told, in effect, that they had power. A generic mitigation does not need the cause. A banner saying the map is behind and pointing customers to another way to report an outage, or a fallback view built from the meters’ last-gasp signals, would make the map honest while the diagnosis continues; Encore restored fans’ queue positions in the same spirit while it was still working out what had gone wrong. Such an option is useful only if someone has practiced switching it on, which is why the rubric asks whether it has been rehearsed.

The second is honest communication, and here the radio is a fair judge. The radio stations had no special knowledge of Brightwater’s systems; they were simply the first to say something. Encore’s practice of promising the time of the next update, and never the time of the fix, is what lets a comms lead speak early without saying something that later proves false.

The third is sustainable load, and the numbers in the brief already tell much of the story. Three or four pages a day, more than half of them at night and most of them needing nothing, means the engineers who must declare quickly and think clearly on storm night have been trained by a whole quarter of pages to expect that a page is noise. Fatigue, as Section 5.5 put it, erodes judgment first. Before you decide how many people the rotation needs, decide how many of those 310 pages should ever have reached a person; a smaller and more honest stream of pages may turn out to matter more than a larger rotation.

Where you now stand

You can now declare an incident by a concrete test, so that a tired engineer does not have to decide alone whether something is serious enough. You can staff a response with the roles the fire services refined over half a century: a commander who does not operate, an ops lead who does, a voice for everyone outside the channel and a scribe who keeps the record. You can recognize fixation and break it with a plain question about what is still broken now. You can mitigate before you diagnose, trading a known small cost for relief from a large and certain harm. You can give an agent only the part of the work that has been approved and that the evidence supports. And you can size a rotation that people can carry for years, not just survive for a quarter. Encore’s 47 minutes ended with no seat sold twice, and with some 30,000 fans wrongly called bots before the rule that blocked them was switched off. Chapter 6 takes up the harder question that this chapter deliberately set aside: how a sensible rate change, a routine deploy, a trusted reconciler and an old bot rule came to meet at 10:02 on the most important morning of the year.

References

• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering. O’Reilly/Google, 2016 — free online.
• Betsy Beyer, Niall Richard Murphy, David K. Rensin, Kent Kawahara, Stephen Thorne (eds.) — The Site Reliability Workbook. O’Reilly/Google, 2018 — free online.
• PagerDuty — PagerDuty Incident Response Documentation. PagerDuty — free online.
• Rob Schnepp, Ron Vidal, Chris Hawley — Incident Management for Operations. O’Reilly, 2017.
• Amazon Web Services — Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region. AWS, 2017 — free online.
• Mica R. Endsley — Toward a Theory of Situation Awareness in Dynamic Systems. Human Factors, 1995.
• Karl E. Weick — The Collapse of Sensemaking in Organizations: The Mann Gulch Disaster. Administrative Science Quarterly, 1993.
• Meta Engineering — Leveraging AI for Efficient Incident Response. Engineering at Meta, 2024 — free online.
• Yinfang Chen et al. — Automatic Root Cause Analysis via Large Language Models for Cloud Incidents. EuroSys, 2024.
• Google Security Blog — Accelerating Incident Response Using Generative AI. Google, 2024 — free online.
• Riccardo Carlesso, Ramón Medrano Llamas — How Google SREs Use Gemini CLI to Solve Real-World Outages. Google Cloud Blog, 2026 — free online.
• Saurabh Jha et al. — ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks. ICML, 2025 — free online.
• Court of Justice of the European Union — Case C-518/15, Ville de Nivelles v Matzak (2018), and Cases C-344/19 and C-580/19 (2021). CURIA — free online.