Operating Production Systems in the AI & Cloud Era

← the whole book on one page

Chapter 7: The Production Organization and the Capstone

No one flies the mission alone, and no one learns to fly it on the day.

For six chapters we have treated production as a system of parts: automation and telemetry, incident command and postmortems here, and in the Resilience book beside them service level objectives, capacity and runtimes, pipelines and rings, caches and ledgers. Again and again we said that each of those parts was held in place by people, and again and again we moved on to the next part. This last chapter turns to the people directly. Its question is how to arrange an organization so that many specialists, and a growing number of machines, can run one system together when none of them holds the whole of it in their head. That is the oldest problem in our field wearing working clothes. We master a complex program by decomposing it into parts that each fit in one mind, and an organization that runs a complex system has to be decomposed the same way, with the same care about where the boundaries fall and what crosses them.

The best answer I know to that question was worked out in a room in Houston in the 1960s. NASA’s Mission Control put a spaceflight in the hands of flight controllers, each of whom owned one system or one discipline of the flight. Behind each controller stood a back room of specialists, joined to the console by voice loops, the shared audio channels on which each group talked and listened. The arrangement meant that the person at the console carried a summary of the system while the full detail lived one room away, ready to be asked for. Above all of them sat the flight director, who by long-standing rule could take whatever action the crew’s safety and the mission’s success required. Seen with a software engineer’s eyes, this was decomposition and hierarchy built out of people. Each console was a module with a narrow interface, each back room was an implementation hidden behind that interface, and one role existed whose whole job was the whole. The analogy with software is close, though it breaks in an instructive place: a module never gets tired, frightened or distracted, and Mission Control’s designers knew their components would be all three, which is why so much of their effort went into training rather than into the diagram.

That arrangement was tested at the worst possible moment. On 20 July 1969, during Apollo 11’s descent to the Moon, the lunar module’s guidance computer raised program alarms, first 1202 and then 1201. Both meant that the computer was overloaded and was shedding lower-priority work to keep the essential work running. The guidance officer, Steve Bales, had seconds to decide whether to abort the landing. In his back room, Jack Garman judged that the alarms were safe so long as they did not persist, and on that judgment Bales called “go”. They could make that call in seconds because they had already faced it once, when nothing was at stake. A training simulation on 5 July, about two weeks earlier, had thrown a 1201 alarm at the team, and the team had aborted when it did not need to. Being wrong in a simulation cost nothing, and the team used the error well. After that simulation the flight director, Gene Kranz, had every alarm code reviewed, and Garman wrote out what each one meant, a list he kept at his console for the flight. So when the real alarm came, the knowledge was already in the room, written down and within reach of the one person who needed it. The landing depended on three things working together: a structure that put the right specialist one voice loop away, a culture in which a back-room engineer’s judgment was asked for and trusted, and a rehearsal that had surfaced the gap while it was still cheap. Structure, culture and rehearsal are this chapter’s subject.

Encore, in this final chapter, stands where Mission Control stood after a hard flight. The Vesper Hale postmortem of Chapter 6 has produced its changes, and those changes are now in place. The promoter has announced four added stadium dates for the tour, going on sale on a Friday six weeks from now, which gives Encore both a reason to prove that the changes work and enough time to prepare properly. The chapter follows that preparation in the order an organization would think about it. The first four sections examine how Encore’s organization is shaped, how ownership of its services is divided, how its people are trained, and what culture lets information reach the people who need it. The fifth section works through the on-sale day itself, end to end, drawing on every earlier chapter of this book and on the engineering practices of the Resilience book.

7.1 SRE Team Models — Embedded, Central, and Platform

Where reliability expertise lives

Every organization that runs software in production has to decide, sooner or later and often without noticing that it has decided, where the people who know most about reliability will sit. An SRE team model is the decision about where reliability expertise sits relative to the services it protects, and therefore about who notices trouble, who decides what to do about it and who is woken when it happens. The choice looks like a matter of org charts, but it is really a choice about the flow of knowledge. Put the experts close to one service and they will understand it intimately; spread them across many and they will see patterns that no single team can see. Mission Control faced the same question and answered it with consoles and back rooms: the specialist knowledge lived next to the system it watched, while one role kept the whole in view.

Site reliability engineering (SRE) did not arrive with this question already answered. Chapter 1 told how SRE grew from the seven-engineer production team Ben Treynor took over at Google in 2003. Only later did it settle into the form most people now picture, in which an SRE team takes over a service’s pager only after a production readiness review has found the service ready for it. A production readiness review is a structured examination of a service before reliability engineers accept operational responsibility for it: its objectives, its alerts, its runbooks, its capacity and its failure modes. The review matters because it turns “will you look after this for us?” from a favor into a contract, with conditions the owning team must meet first. Once that gate existed, the question of where SRE should sit became a real design choice, because there was now something definite to hand over and someone definite to receive it.

The choices have since been catalogued. Ryan Franco and Brian Brown’s survey for Google Cloud (2019) names six team types. Figure 7.1 folds two of them, the infrastructure team and the tools team, into one platform-aligned row, since both serve every product team through shared machinery rather than through any one service, and it adds the newer hybrid.

Figure 7.1 — SRE team models and how each tends to fail. Most organizations end up hybrid; the useful question is which failure mode they have chosen to watch for.

The table is best read from its last column, because each model’s strength casts a matching shadow. The central team gathers lessons in one place, but that same concentration makes it a queue: every other team must wait its turn, and the reliability engineers slowly become the people who do the operations work nobody else wants. Embedded engineers gain context and can shape a design before it hardens, but cut off from their peers they lose the shared practice that made them reliability engineers, and the product team’s backlog quietly turns them into extra feature developers. Consultants reach many teams cheaply, yet because they own nothing, their advice carries no consequence for them, and over time they lose the feel for what a 3 a.m. page is actually like. A platform-aligned team gains leverage, since a fix to shared infrastructure serves every team at once, but it stands far from the failure modes that live in one product’s own logic. None of these failures is a mistake of execution. Each is the natural drift of a model left to itself, and so choosing a team model is, I think, mostly a matter of choosing which drift you are prepared to watch for.

The commonest of these drifts has a name. David Blank-Edelman’s Becoming SRE calls it title flipping: renaming the operations team and changing nothing else. Title flipping is the central model’s failure in its purest form, because the work, the queue and the relationship with product teams all stay exactly as they were, and only the job titles change. Google’s second edition of Site Reliability Engineering, published this year, describes hybrid models and reliability champions placed inside product teams. A reliability champion is an engineer inside a product team who carries reliability practice for that team, as part of their normal role rather than as a separate job, and who acts as the team’s link to the central reliability engineers. That idea is young, and we should hold it with the caution young ideas deserve. Its nearest relative is the security champion, and security champion programs report precisely the failure Figure 7.1 names: champions who have the title but neither the time nor the recognition to do the work. A champion whose reliability work must be squeezed in after the sprint’s real commitments is a champion in name only.

The models differ along two dimensions that pull against each other. Context is how deeply the reliability engineers understand a particular service: its design, its users and its quirks. Reach is how many teams and services their work touches. Embedded engineers have a great deal of context and little reach; consultants and platform teams have reach and little context. Figure 7.2 draws this trade of context against reach.

Figure 7.2 — SRE team models placed by context and reach. The pure models sit near one trade-off line; a hybrid with champions tries to get above it, which is where Encore now aims.

The trade-off line in the figure is the price of having a fixed number of reliability engineers: every hour spent deep inside one team is an hour not spent across the others. A hybrid with champions tries to rise above that line by drawing on people outside the reliability team. The champions bring the context, because they already live inside their teams, and the small central team brings the reach, because it sets the shared practice and tooling that every champion uses. Mission Control’s back rooms worked in a similar way, with the specialist close to the detail and the shared loops carrying what mattered to everyone, but the analogy breaks at one point that matters here: a back-room specialist did nothing else, while a champion is also a full member of a product team with a product team’s deadlines. Whether the hybrid succeeds therefore depends almost entirely on whether the champions are given real time, which is where Encore’s story ends.

Encore’s arrangement was not designed in one sitting; it grew in steps, each forced by the company outgrowing the step before. With nine engineers and a modular monolith there was no team model to speak of. Everyone carried the pager and knew every part, so context and reach were the same thing, held in nine heads. Extracting the on-sale path drew the first real boundary, the first place where one engineer could say of a component “that is not mine”. At 70 engineers in nine teams, the boundaries had multiplied and ownership had to be written down, because it could no longer be assumed. At 400 engineers in 40 teams, the platform group and its Reliability team appeared, because nobody could see the whole any more. This is the story of most growing organizations, told in the language of complexity: when a system outgrows any one mind we decompose it, and then we need a structure that keeps the pieces in touch.

The Reliability team today is 10 engineers led by a staff SRE, working from a one-page charter. Franco and Brown recommend a charter as the defense against the kitchen-sink failure, because a team that has not written down what it will not do will slowly be handed everything. Encore’s charter therefore opens with a refusal: its first line says that the team does not run other teams’ services. Within that boundary the team works in three ways, which match three rows of Figure 7.1. As a platform-aligned team, it holds the pager only for platform planes that have passed a readiness review. As a consulting team, it serves the 40 stream-aligned teams through the service level objective (SLO) tooling, the incident process, the readiness review and the game days, all of which it owns on everyone’s behalf. As an embedded team, it sends two of its engineers into a product team for each big launch; they help that team improve its reliability without taking over its service, and they leave when the launch ships. The departure date is the point. Embedding for one launch captures the embedded model’s strength, context at the moment design decisions are made, while a fixed end keeps away its typical failure, the slow drift into becoming permanent feature developers.

The last piece came after Vesper Hale. Encore named a reliability champion in every stream-aligned team and protected a tenth of each champion’s time in planning, which means the time appears in the team’s plan as committed work rather than something to be found in the margins. No published standard sets that figure. Encore chose a tenth because it is large enough to appear as a real line in a plan and small enough that nobody resents it. Whether it is the right amount is something Encore can learn only by watching, and that kind of continuing judgment is what the champion model asks of any organization that adopts it.

Recap. Team models trade context for reach: embedded engineers know one system deeply, while central, consulting and platform teams touch many systems and know each less well. Every model has a characteristic drift, so mature organizations combine models so that each covers another’s weakness, gate what reliability engineers carry with a readiness review, and write a charter that says what they will not do. Reliability champions are a young addition that promises context and reach together, and that fails, as security champions have, when their time is not protected.

Exercise 7.1. Place your organization in Figure 7.1, honestly rather than by its org chart: where do the people who know most about reliability actually spend their days, and who actually answers the pager? Name that row’s typical failure mode, and find one piece of evidence that it is, or is not, happening to you.

7.2 Ownership — You Build It, You Run It, and Its Limits

What “you run it” asks of a team

Section 7.1 settled where reliability expertise lives. This section asks the question underneath it: once a service is running, who answers for it? The most familiar answer is Vogels’s “you build it, you run it”, which the Architecture book took as a principle of operable design. Its virtue is that it closes the loop between the people who change a system and the people who feel the consequences of those changes. When the same team writes the code and answers the page, a careless deploy costs its own authors a night’s sleep, and that cost teaches faster than any review board. It is the right default. But a principle repeated for 20 years starts to sound like a law of nature, and principles that sound like laws stop being examined, so it is worth stating plainly what this one asks of a team.

Operational ownership is a named team’s accountability for a service’s behavior in production, together with the authority to change it. The accountability covers the service’s objectives, its pager, its runbooks and, eventually, its retirement. The authority is the other half of the bargain, and the definition does not work without it. A team that is accountable but lacks authority, answering for a service it is not allowed to change, can only absorb blame for decisions made elsewhere. A team with authority but no accountability, free to change a service whose failures land on someone else, has rebuilt the old wall between development and operations under a new name. Ownership in the useful sense means that both halves sit with the same people.

The principle has three limits, and each of them is, at bottom, a limit on how much complexity one group of people can master. The first is cognitive load: how much of a system a team must hold in its heads in order to run it well. Matthew Skelton and Manuel Pais’s Team Topologies (2019; second edition 2025) argued that a team’s software should be sized to fit what the team can understand, because whatever lies beyond that size will be run badly however willing the team is. Their remedy is a platform offered as a service. If the platform takes on the clusters, the pipelines and the shared tooling, then what remains for a product team is mostly the essential complexity of its own domain, the seats, orders and queues of the business it serves, rather than the accidental complexity of the machinery underneath. The evidence for this is case studies rather than controlled studies, and we should hold it with that much confidence. Still, Encore has already shown us what happens when ownership outruns understanding. The Notification rewrite of Chapter 1 left a service of 40 files that nobody had read in full. The team owned the service on paper, its name was on the service and its pager would ring, but in practice it was a stranger to the code it answered for. Ownership in that state is a label, and a label debugs nothing at 3 a.m.

The second limit is arithmetic. Chapter 5 set out the caps that keep an on-call rotation humane: no more than a quarter of anyone’s time on call, about two incidents at most per shift, and compensation for the hours someone is tied to a pager. Those caps have a consequence for headcount. If a rotation puts one primary and one secondary on call each week, then each engineer serves two weeks in every eight, a quarter of the time, only if the rotation has at least eight people. That is how Google’s SRE book derives its minimum of eight people for a single-site rotation, and it means a team of five cannot meet the cap alone however hard it tries. Encore pairs its smallest teams with a sibling team for out-of-hours paging, so that the combined rotation reaches eight. The pairing holds only because of what stands behind it: the sibling team has the same runbooks, the same access and a written escalation path back to the owners. Without those three things, pairing would mean waking someone who can do nothing except wake someone else.

Incident command follows a different logic. Encore’s incident commander rotation draws trained engineers, most of them senior, from across all the teams rather than from any one owner. What qualifies them is the training. In PagerDuty’s published practice and in Google’s, a commander need be neither the most senior person present nor an expert on the failing system, because the commander’s job is to coordinate the people who do know the system, and to keep the whole in view while they work on the parts. It is the flight director’s role from the chapter’s opening, set apart from the consoles for the same reason Houston set it apart.

Google’s engagement model adds a feature that the slogan hides: ownership can move back. When SRE carries a service’s pager and the service falls below the standards both sides agreed, SRE can return excess operational work to the developers and, in the extreme, the pager itself, until the service meets those standards again. Joint work to fix the service is the usual remedy, and a full hand-back is the last resort. But the mere existence of the option changes what ownership is. It stops being a permanent assignment and becomes a contract with terms, and a contract can be renegotiated when one side stops keeping them.

The third limit is the seam between owners, and it is the subtlest of the three. During Vesper Hale, every component involved had an owner, and each owner had done something reasonable within its own territory. The hazard lived in none of those territories. It lived in the interaction between them: a reconciler that drained a node without knowing an on-sale was under way, a calendar that the reconciler never consulted, a cache that had just gone cold, and a bot rule that read frightened fans’ retries as an attack. No owner’s dashboard showed that interaction, because no owner’s territory contained it. A map of ownership can be complete and still leave the most dangerous ground unclaimed. Encore’s response was to give the seam an owner and a contract. The calendar became a published signal owned by the On-Sale team, and any automation that changes the purchase path must declare that it reads the signal, a requirement the platform’s scorecard enforces. The mechanism is Encore’s own, but its precedents are not. Managed Kubernetes offers maintenance exclusions, so that a provider’s upgrades stay clear of a customer’s peak periods, and Argo CD offers sync windows that deny automated syncs during a chosen time range. Both tools encode the same lesson Encore learned the hard way: an automation must know when not to act.

A touring production shows this kind of arrangement at its best. When a show comes to a city, the touring crew runs the show itself, while the venue’s house crew owns the building’s rigging, its power and its fire curtain. Between them sits the technical rider, the document the tour sends ahead, which among much else says who owns which cable and who may touch which hoist. Two crews with different employers share one stage safely because the boundary between them was written down before load-in. The analogy fits Encore’s teams sharing a platform closely, but it fails on one point that deserves our attention: time. No venue rebuilds its rigging during a performance. The house crew’s work happens before the doors open or after the audience has gone, and everyone understands that without needing to write it down. Encore’s platform drained a node in the middle of an on-sale, because nothing told it that the performance had begun. A contract between owners must therefore cover time as well as territory: it must say when each party may act, and not only what each party owns. I spent years on the platform side of exactly this boundary, inside a large bank, and I have come to believe that most operational trouble between teams begins with a clause about time that nobody wrote.

Contracts that live only in people’s memories decay, so Encore keeps its ownership where it can be read and checked. Its service registry is a software catalog of the kind Spotify’s Backstage made common. Each entry names an owning team, a rotation, objectives and runbooks, and the scorecard flags any entry where one of these is missing. The first sweep for gaps found 11 components with no owner at all. Most were scheduled jobs, the small useful things someone writes once and nobody thinks of as a service, and one was a dashboard that three teams relied on and none had ever claimed. Each was a seam in miniature: something many depended on and nobody answered for. A registry shows who owns what, but it cannot show whether the owners are coping. For that, the Reliability team reviews each rotation’s pages per shift and its out-of-hours pages every month, because a rotation shows strain long before anyone resigns. Tired engineers usually keep answering the pager, out of pride or duty, and the numbers are often the first place the strain becomes visible to anyone outside the team.

Recap. “You build it, you run it” is the right default, and an incomplete one. Ownership needs authority as well as accountability. It must fit within what a team can understand, and within the arithmetic and load caps that keep a rotation humane. And the seams between owned components, where the dangerous interactions live, need owners of their own. A contract between teams should say when each may act, as well as what each owns.

Exercise 7.2. List the services your team owns and the people on its rotation. Compute the weeks on call per person per year, and the pages per shift over the last quarter. Then name one interaction between your service and another team’s that nobody owns, and propose who should own it.

7.3 Skills, Training, and Operational Maturity

Rehearsing the day before it comes

The previous two sections were about where people sit and what they own. A structure, however well drawn, only helps if the people in it know what to do when the moment comes, and so we turn now to how that knowledge is made. Mission Control did not discover its competence; it built it on purpose, and it built most of it through simulation. A team led by the simulation supervisor, known in Houston as the SimSup, designed full-scale rehearsals of a flight and quietly injected failures into them, so that the controllers met trouble at their consoles long before they met it in space. The 1201 alarm that drew an unnecessary abort on 5 July 1969 was one of those injected failures. Just as important as the run itself was what came after it: the debrief, in which the team went over what had happened and why they had decided as they did. That simulation produced Garman’s written list of alarm codes, and the list was on his console for the landing. Nearly everything we have learned about operational skill since then builds on the same loop, and it is worth naming its four steps plainly: realistic practice, an honest debrief, a written artifact that captures what was learned, and then practice again.

We need that loop more than Houston did, and the reason is the irony Chapter 3 described. When we automate a process, the automation takes over the routine work, which is exactly the work in which people used to acquire their skill, and hands back to people only the rare and difficult cases, which are exactly the cases in which skill is most needed. The better the automation, the fewer chances anyone gets to practice, and the harder the problem they face on the day it gives up. Encore shows this precisely. Its shard failover is automatic, and it works; in normal operation, when a node goes away, a new leader is elected without anyone lifting a finger. That reliability is the very reason that, before Vesper Hale, few of Encore’s engineers had ever performed a shard failover by hand. Nobody had neglected anything. The system had simply never given them the occasion.

The good news is that the practices which keep skill alive are cheap, and each of them is a version of the SimSup’s rehearsal scaled down for an ordinary team. The first is the Wheel of Misfortune, described in Google’s first SRE book (2016): a role-play in which a game master narrates a past incident, step by step, and an engineer works through it aloud, saying what they would check, what they would conclude and what they would do, while the game master reveals what the system shows in response. Google runs it weekly. Nothing in production is touched, which is what makes it cheap; the incident is real, because it already happened, which is what makes it realistic. The second is the game day, a planned exercise in which failures are injected into a running system on purpose, which the Resilience book treats in its chapter on designing for failure as an experiment on the system. The same exercise becomes training when we measure the people as well as the software: how long it took to declare an incident, whether a commander was named, whether the status page told the truth, and whether the follow-up actions were actually finished afterward. The third is shadowing, in which a new engineer sits beside an experienced one through a shift on call, watching how the veteran reads alerts and makes decisions. Reverse shadowing swaps the seats, so that the newcomer leads while the veteran watches and steps in only if needed. The value of the pairing is in the order: first the newcomer sees the work done, then does it with a safety net, and only then does it alone. PagerDuty’s training path for incident commanders follows this order, asking for a week of shadowing and a week of reverse shadowing, then several simulated incidents in command.

Medicine adds a lesson of its own, about checklists and who owns them. A surgical checklist built around pause points, defined moments at which the whole team stops and confirms the few things that kill a patient when forgotten, was the core of the World Health Organization’s surgical safety checklist. In 2009 Alex Haynes, Atul Gawande and colleagues reported that across eight hospitals, deaths after surgery fell from 1.5% to 0.8% once the checklist was introduced, nearly halving them. That seemed to settle the matter, until in 2014 David Urbach and colleagues studied Ontario, where the province had made the checklist mandatory in its hospitals, and found no significant change in deaths or complications. The same list, in two settings, gave two different results. The usual reading of the difference is ownership. The teams in the original study had chosen their checklist and believed in it, so the pauses were real pauses in which people actually looked. A mandate, by contrast, produced compliance from people who had not chosen the list, and a checklist that is ticked without being believed becomes a form to fill in rather than a moment of attention. Encore’s on-sale readiness review, held on the Wednesday before each big on-sale, is designed as this kind of pause point. It is short, so that people give it attention rather than endure it. It is owned by the team that runs it, so it is their list and nobody else’s. And every item on it can be traced to an incident or a near miss, so that no one has to wonder why a line is there.

With those practices in view, we can look at what Encore actually does. Its training program, rebuilt after Vesper Hale, has five parts, and each one answers a specific weakness that the incident exposed. First, each team runs a monthly Wheel of Misfortune drawn from its own postmortems. That is lighter than Google’s weekly cadence, but it is a cadence the teams can sustain, and a lighter practice that continues does more good than an ambitious one that quietly stops. Second, every quarter an engineer who has never done it before performs a manual shard failover in a staging cell, which turns the skill that automation had let fade into one that someone on each rotation has recently exercised. Third, new members of an on-call rotation do two weeks of shadowing and one week of reverse shadowing before they carry the pager alone. Fourth, candidates for the incident commander rotation must command two simulated on-sales before they command a real one. Fifth, in half of all drills Cue, Encore’s operations agent, is switched off, so that the team keeps using its own judgment rather than learning to wait for the agent’s suggestion. That last rule needs a caveat. It has no published precedent in software, and the figure of 50% is Encore’s own choice rather than a number anyone has validated. Its closest analogue comes from aviation: in 2013 the Federal Aviation Administration advised airlines to give their pilots regular opportunities to fly by hand, for the same reason, so that the skill the autopilot spares them is still there on the day the autopilot hands the aircraft back.

All of this raises a natural question: how do we know whether it is working? The tempting answer is a maturity model, a ladder of numbered levels against which an organization grades itself, climbing one rung at a time as it adds practices. Accelerate argued against that approach and in favor of capabilities that can be measured and improved, for a reason worth dwelling on: a level implies a finish line, a point at which the organization has arrived and may stop, and production has no such point, because the system, the traffic and the people keep changing underneath it. A more useful frame is the hierarchy Mikey Dickerson drew for Google’s first SRE book. It stacks the work of reliability in layers, with monitoring at the base, then incident response, then postmortems, then testing and release, then capacity planning, then development, and finally product at the top. The hierarchy has never been tested empirically, so we should not treat it as a scorecard. It reads best as an order of construction, in the way a builder lays a foundation before raising walls: each layer depends on the ones beneath it. A team that cannot yet see what its system is doing has no business building elaborate release automation, because when a release goes wrong it will have no way to notice, and no way to learn from the incident it cannot see.

Recap. Operational skill is produced by rehearsal under realistic pressure followed by an honest debrief, and automation makes that rehearsal more necessary, not less, because it removes the everyday practice in which skill used to grow. Wheels of Misfortune, game days that measure the people as well as the system, shadowing and reverse shadowing, and checklists whose users own them keep skills alive at little cost. Maturity frameworks are most useful as an order of construction, telling a team what to build first, rather than as a ladder with a top.

Exercise 7.3. Name the three operations your team would most need to perform by hand in a real incident. For each, find the last date on which someone on today’s rotation actually performed it. Then schedule a drill for the one with the oldest date.

7.4 Culture — Generative Organizations and Psychological Safety

How information flows

The first three sections of this chapter were about arrangements we can draw and schedule: where reliability expertise sits, who owns which service, how often people rehearse. Structure and training decide what an organization can do under pressure. Culture decides something subtler and, in my experience, more decisive: what the organization will actually hear when someone has bad news. A team can have a flawless rotation and a superb drill program and still fail, because the one engineer who noticed the problem kept quiet, or spoke and was ignored. To reason about that, we need a way to describe culture that is more precise than “good” or “toxic”. The sociologist Ron Westrum gave us one. He classified organizations by how they handle information, and his typology, in the form most often cited today, was published in 2004.

Figure 7.3 — Westrum’s typology of organizational cultures. Each column describes what happens to an inconvenient fact.

The table is worth reading one column at a time, as the story of a single unwelcome fact. In a pathological organization, oriented toward power, the person who brings bad news is punished for it, so people learn to stop bringing it. In a bureaucratic organization, oriented toward rules, the messenger is not punished but simply ignored: the fact is filed, routed to the team whose remit it falls under, and left there. In a generative organization, oriented toward the mission’s performance, the messenger is trained and valued, because the organization understands that its next failure is probably already known to someone in it.

Westrum’s central insight follows from that reading. Safety depends on information reaching the people who can act on it, and reaching them in time; culture is the thing that speeds that information along or blocks it. This is why culture belongs in a book about running production systems rather than in a book about management. A monitoring pipeline that drops alerts is a reliability defect, and so is an organization that drops warnings. The difference is only that we know how to test the pipeline.

For a long time, claims like Westrum’s rested on case studies and argument. The book Accelerate (2018) put them to a larger test: it turned his typology into survey questions, put them to thousands of practitioners, and found that, in that data, a generative culture predicts both software delivery performance and wider organizational performance. It found something more hopeful as well. Teams that adopted continuous delivery and lean management practices reported more generative cultures, which suggests that culture is not only a cause of good practice but can also be one of its results. We should hold that finding at its true strength. The data is self-reported and the analysis correlational, so it shows a strong association and a plausible direction rather than proof. Even so, it remains our best large-scale evidence that a team can change its culture, and that it does so the way most durable change happens, one practice at a time. DORA’s 2025 report on AI-assisted development found the same pattern with new tools: AI amplifies what a team already has, its strengths and its dysfunctions alike. A generative team with an AI assistant becomes more generative; a team whose messengers are neglected now has a faster way to neglect them.

Westrum describes the organization as a whole. The second strand of research looks closer, at the small team, and asks what makes an individual willing to be the messenger in the first place. In 1999 Amy Edmondson defined psychological safety as a shared belief that the team is safe for interpersonal risk taking. The definition is precise in a way worth noticing. It describes a belief, so it lives in people’s heads rather than in a policy; it is shared, so it is a property of the team rather than of one confident member; and its subject is interpersonal risk, the risk of looking ignorant, incompetent or disruptive in front of colleagues. Edmondson showed that teams holding this belief did more of the behaviors through which teams learn, such as asking for help, admitting errors and raising concerns, and that they performed better as a result. Later meta-analyses confirmed the link. Google’s internal study, Project Aristotle, reached a similar conclusion about its own teams, though it was never peer-reviewed and should be weighed accordingly.

It helps to say plainly what psychological safety is not. It is not comfort, and it is not a promise that nobody will disagree with you. Its point is candor: the confidence that saying “I think the freeze is drifting” in a planning meeting will be met with interest and a follow-up question, not with a sigh or a remark about negativity. Readers of Chapter 6 will recognize that sentence. Encore’s on-sale freeze had in fact been loosening for a year before Vesper Hale, one reasonable exception at a time, and the question a culture must answer is whether the person who noticed that drift would have felt able to say so, and whether anyone would have acted.

Westrum’s typology and Edmondson’s definition can sound abstract until we watch them working under pressure. Apollo 13 is the finest demonstration I know of generative information flow in a crisis, and it comes from the same Mission Control that opened this chapter. After an oxygen tank exploded in April 1970, the crew sheltered in the lunar module. That spacecraft’s carbon dioxide scrubbers could not last the trip home, and the command module’s spare scrubber canisters, which would have extended them, were square where the lunar module’s sockets were round. Engineers on the ground built an adapter from bags, cardboard, suit hoses and tape, tested it, and read the instructions up to the crew step by step. Look at the path the idea traveled. It came from engineers well outside the control room, it reached the flight director, it was trusted, and it became a procedure while there was still time to use it. Every link in that chain is a column of Figure 7.3 choosing “generative”: the messengers were heard, responsibility for the risk was shared across teams, and a novel idea was implemented rather than treated as a problem.

The analogy has a limit, and the limit is instructive. A crisis concentrates attention and suspends ordinary politics; almost any organization becomes more generative for a few days when the stakes are obvious and everyone is watching. An organization can be generative in a crisis and pathological in an ordinary week, and it is the ordinary week in which most warnings are raised and lost. Karl Weick and Kathleen Sutcliffe, studying organizations that operate hazardous systems with few accidents, named one of their defining habits deference to expertise: letting a decision travel to whoever knows most about the problem at hand, whatever their rank. The discipline lies in practicing that deference on ordinary days too, in a Wednesday readiness review as well as on the night of an explosion, so that the habit is already in place when the stakes arrive.

At Encore, the practices that came out of the Vesper Hale review try to build exactly that ordinary-week habit, and to measure it. Westrum’s six survey items now sit in the quarterly engineering survey. The results are reported per team and, as DORA advises, are never used to rank teams, because a culture score that can cost a team its standing will quickly stop measuring anything except the team’s ability to answer surveys. Readiness reviews now open with a single question: “What are we worried about?” It is a cousin of Gary Klein’s premortem, in which a team imagines that the plan has already failed and explains why, a framing that makes voicing a doubt feel like contributing rather than complaining. The first answer to the question is thanked before it is examined, a small ritual whose purpose is to tell everyone in the room what happens to the messenger here.

Psychological safety also has a new object, and Vesper Hale showed it. During the incident, Cue confidently suggested restarting Seat Inventory pods, and Jonas declined the suggestion in front of a busy incident channel. He was right to: as Chapter 5 described, a restart would have forced leader elections on every shard. Automation bias, the tendency to accept a machine’s recommendation over one’s own judgment, has been documented in cockpits and clinics for decades. What is new is a machine that argues in the engineers’ own language, fluently, with the vocabulary and cadence of a capable colleague, and whose suggestions arrive in the same channel and in the same form as a senior engineer’s. Declining it can feel less like overriding a tool and more like contradicting a confident peer in public. So I have come to believe that the safety to contradict a fluent machine now matters as much as the safety to contradict a senior colleague. Encore had the first only because Jonas happened to have run a failover by hand; how to build it deliberately, for everyone on a rotation, is something we are still learning.

Recap. Westrum’s typology describes a culture by what happens to inconvenient information: it is punished, neglected, or welcomed and acted upon. DORA’s survey data associates generative culture with performance and suggests that adopting better practices can move a culture, while its AI findings say that new tools amplify whatever culture they land in. Psychological safety, the shared belief that interpersonal risk is safe, is the condition under which a team learns; it rests on candor, it must hold in ordinary weeks as well as in crises, and it must now include the safety to disagree with the machine.

Exercise 7.4. Recall the last time someone in your organization raised an unwelcome risk. Which column of Figure 7.3 describes what happened next: was the messenger punished, neglected, or heard and acted upon? Write down one practice that would move your organization one column to the right, and name who would have to adopt it.

7.5 Capstone — Operating One System End to End

The added dates

Four added stadium dates for Vesper Hale go on sale at 10:00 CET on a Friday, and Encore expects about 1.4 million fans in the waiting room across the four events. This is the same artist, the same kind of crowd and much the same system that failed in Chapter 5, and so the question this section asks is a practical one: what does it look like when everything the book has described is applied, in order, to one known peak?

The shape of the preparation was not Encore’s invention. Large engineering organizations that face predictable peaks have converged on a similar rhythm. Google built a dedicated function, launch coordination engineering, to shepherd big launches; AWS runs operational readiness reviews before a service or a major change goes live; Shopify spends months on game days and scale tests before Black Friday. The details differ, but the pattern underneath is the same, and it has four parts. First, set objectives for the event itself, because a day this important deserves its own definition of success. Second, test load past the forecast, so that the forecast is never the edge of what you know. Third, rehearse the likeliest failures, so that the first time people meet them is not the real day. Fourth, hold a review a few days out, close enough to see the final state and far enough to still change it. Figure 7.4 lays Encore’s version of that rhythm across the six weeks.

Figure 7.4 — The added dates, end to end, with the chapter each step comes from in this book or in the Resilience book. Read down the right-hand column and the two books appear in order of use rather than order of writing.

A few rows deserve a word of explanation, because each one answers something specific. The on-sale-window SLO exists because Encore’s ordinary objectives are measured over 28 days, and a bad hour barely moves a 28-day number; an objective for the first 60 minutes makes the hour that matters most visible on its own terms, and the burn-rate alerts had to be re-derived so that they fire at a pace that makes sense inside one hour. The load tests run with a cold cache, at both the admission rate Encore intends to use and the higher rate to which it was raised during Vesper Hale, because the Chapter 6 review found that the earlier test had been run warm and so had proved headroom that did not exist. Admission stays at 1,200 fans/s per event. Because each event’s writes are pinned to a home region, assigning two events to each region spreads the write load instead of stacking all four on one. And the pre-warm step now checks the cache hit ratio rather than assuming the warm-up worked.

Two other rows close hazards that belonged to Vesper Hale specifically. The first is the bot rule. A bot rule is configuration, not code, and it is tempting to treat configuration as lighter and safer to change. History says otherwise: in July 2024 a single content update to CrowdStrike’s security sensor, pushed to every online Windows sensor at once, crashed millions of Windows computers around the world (the Resilience book tells the story in its chapter on shipping change), and it showed that configuration which reaches everyone at once is as dangerous as code. So the retuned rule travels in stages. It first runs count-only against recorded traffic, where it can be measured but blocks nobody; then it is enforced for one small on-sale; only then does it go everywhere. Behind those stages sits a second safeguard: Bot Screening’s loader validates every rule set it receives, and if a new set fails validation it keeps running the last good one.

The second hazard is what we might call the reconciler’s gap. During Vesper Hale, the platform’s node-upgrade reconciler, which counted as low-risk automation and was exempt from the freeze, drained a node in the middle of the sale and set off a shard failover. The automation was doing its job correctly; it simply did not know that this was the worst hour of the year to do it. Encore’s answer was to make the on-sale calendar something machines can read. A calendar controller reads the calendar and writes a deny window into the Argo CD project that holds the purchase-path applications, so that GitOps stops syncing changes to them for the duration. The node-upgrade reconciler reads the same calendar and holds drains on the same node pools. Two independent pieces of automation now share one source of truth about when to keep their hands off. For the added dates, the window looks like this:

# AppProject "purchase-path", written by the calendar controller
spec:
  syncWindows:
    - kind: deny                 # no automated syncs in this window
      schedule: "0 8 27 11 *"    # Friday 08:00, two hours before the Gate opens
      duration: 6h
      timeZone: Europe/Paris
      applications: ["sale-gate-*", "bot-screening-*", "seat-inventory-*", "orders-*", "catalog-*"]
      manualSync: true           # a registered exception can still be synced by hand

The most interesting line is the last one. A freeze that nobody can ever break is brittle, because sooner or later a real emergency needs a change, and people will find a way around a rule that offers no legitimate door. The manualSync line is that door, and it writes the exception register into configuration. The machines stop on their own when the window opens. A person who has a written reason, and whose exception has been logged with an approver, can still act by hand. Automation is restrained by default, and human judgment keeps a narrow, recorded path through.

The game day, held on the Tuesday before, did what a good rehearsal should: it found the weakest step. During Vesper Hale, at 10:31, engineers had re-admitted fans that Bot Screening wrongly blocked, restoring each fan’s place in the queue. That fix had been applied by hand, under pressure, for the first time. In rehearsal it took 14 minutes, which is a long time to leave legitimate fans locked out of a sale that sells out within the hour. So Encore turned it into a scripted workflow, in the self-service style Chapter 3 described, and added it to Cue’s action catalog, the short, platform-enforced list of operations that Cue is allowed to propose. Only two entries in that catalog may run without a person’s approval, both granted on track record: raising Catalog’s autoscaling floor through Git, within a cap, and holding a node in the upgrade reconciler. Re-admitting fans is not one of them. Cue may find the fans and propose the action, but it may never perform this one alone. I should be candid that agent-proposed, human-approved action is young practice. The published accounts, which come from the cloud providers’ own operations agents, share two features: credentials issued per action rather than held standing, and a named person who approves. Encore’s entry follows that shape:

action: readmit-blocked-fans
proposed_by: cue                  # with the query that found the fans
approved_by: [incident-commander, operations-lead]   # one named person, in the channel
scope: one event; fans blocked by Bot Screening in the last 60 min
preview: fan count and sample sessions shown before approval
limit: 50000 fans per run
credentials: issued for this run only, expire after 10 min
audit: proposer, approver and result posted to the incident log

Each line of the entry narrows something. The scope confines the action to one event and to recent blocks, so a mistake cannot spread across the platform. The preview lets the approver see how many fans and which sessions are involved before saying yes, so approval is a judgment and not a rubber stamp. The limit caps the damage a wrong query could do in one run. The credentials exist only for this run and expire quickly, so there is nothing standing around to be misused later. And the audit line means that afterward anyone can see who proposed, who approved and what happened.

The day itself was dull, which is exactly what six weeks of preparation had been for. At 10:09, card authentication at one Dutch issuing bank slowed down. Of the window’s burn-rate alerts, the slower one fired, the one meant to catch a steady leak of the error budget rather than a sudden collapse. The incident commander asked Cue for a breakdown and confirmed that the slowdown sat with the bank itself, behind both PSPs alike. That finding mattered, because it ruled out the obvious-looking remedies. The primary PSP’s circuit breaker had rightly stayed closed: shifting new payments to the secondary PSP would have sent them to the same slow bank, and so would have achieved nothing. Nor did any seats go into pending payment, the hold Encore uses when a payment route is degraded, because neither PSP was degraded; only one bank behind them was. The commander chose not to act beyond posting a status-page note for Dutch fans, and logged that decision, which is worth noticing: deciding not to intervene, and writing down why, is a decision like any other. About 22% of the window’s error budget was spent. Since the objective allowed 1% of admitted fans to see an error page, that means roughly one admitted fan in 450 did. All four dates sold out within the hour (Figure 7.5).

Figure 7.5 — The Friday of the added dates, with the morning before the Gate opened compressed. The red ink is the error budget, spent mostly after 10:09; the 09:40 line marks the hand-run check that the later review would find.

Learning from a good day

Encore held a review anyway. That may seem odd, since nothing went wrong, but Chapter 6 had set a rule of proportion for reviews, and a good day has a place in it. A major incident earns a full CAST analysis of its control structure; a lesser incident earns a lighter review of contributing factors; and a good day earns the lightest form of all. That form is borrowed from the US Army’s after-action review, which follows every exercise whatever the outcome, and asks what was supposed to happen, what did, and why.

This one found an adaptation nobody had designed. At 09:40, the pre-warm check had timed out on its first run. An engineer reran it by hand, read the cache hit ratio from a dashboard, saw that it was fine, and moved on without telling anyone. Nothing about the act was wrong; it was sensible, quick and correct. But it meant the success of the morning depended, in one small place, on a person quietly filling a gap that the procedure did not know it had. The pattern is the one Chapter 6 found behind Vesper Hale: a platform engineer who paused the reconciler by hand before big on-sales, in no runbook, until she moved teams and the protection left with her. This is Safety-II in miniature, the view from Chapter 2 that safety lives partly in the everyday adaptations by which people make work succeed. The useful response is to make the adaptation visible and then build it in, not to scold anyone, and so the check now retries on its own and has an explicit failure path that tells people when it has not passed. Reviewing success in this way is still emerging practice in software, and in my experience it survives only when its time is protected; a review of a day that went well is the first meeting to be cancelled when the calendar fills.

Looking back across the six weeks, what strikes me is how little of it was heroic, and how much of it had been written down in advance by people who would not be in the room on the day: the SLO, the load test, the staged rule, the sync window, the catalog entry. The arrangement is Mission Control’s, rebuilt in software and teams. There are consoles with narrow interfaces, in the form of teams that own one service each. There are back rooms behind them, in the form of the specialists and the embedded Reliability engineers who carry the detail. There is one role whose job is the whole, the incident commander. And there was a simulation before the day, the game day that found the slowest fix while it was still cheap to find.

Recap. The capstone places the whole book in time against a known peak: SLOs for the event, capacity tested cold and past the forecast, a second payment route, configuration rolled out in stages, automation that reads the calendar, rehearsal, named roles, an agent limited to a scoped catalog, and a review sized to the day. A dull day is what the preparation was for, and a short review after a success finds the quiet adaptations that made it so, so that they can be built in before the people who perform them move on.

Exercise 7.5. Choose the next predictable peak your system faces. Build your own version of Figure 7.4, with a “From” column pointing to the practices you already have. Each blank cell in that column is a gap, and together the blank cells are your preparation plan.

Kata: The Capstone Kata — Results Day

Your brief: “Quorum.” Quorum runs the online admissions platform for universities in four EU countries. It serves 1.8 million applicants a year, and 300 institutions use it as tenants. Two days of the year matter more than all the others. On deadline day, submissions rise 40× in the final six hours, as applicants who have been polishing their essays for weeks all press “submit” at once. On results day, 600,000 applicants sign in within the first hour after 08:00 to learn which offers they have received, and each of them is waiting for news that may decide the next several years of a life.

The organization behind those two days is thin in the places that matter. Quorum has 95 engineers in 12 teams, but a central operations team of six holds every pager, and two of its members left last year. The applications database is a single primary with one replica in the same region, so a regional failure would take both copies at once, and nobody has ever restored a full backup to learn whether it works or how long it takes. An AI assistant answers applicants’ questions about entry requirements; last spring it told some applicants a deadline one day later than the real one. And last year’s results day had 52 minutes of errors, because a scheduled certificate rotation restarted the authentication service at 08:05, five minutes into the busiest hour of the year. Nothing in that rotation was broken. It simply did not know what day it was.

The board’s request is short: make results day boring.

This Kata asks you to do for Quorum what Section 7.5 did for Encore’s added dates: to run one system end to end through a predictable peak, drawing on every chapter at once. Each deliverable below corresponds to a part of this book or of the Resilience book, and together they form the preparation plan for both of Quorum’s critical days.

Deliverables:

1. Operating model — choose a team model from Figure 7.1 and write a one-page charter for it. Map an owner to every service and to every seam between services, since incidents so often begin in the seams nobody owns. Work out the on-call arithmetic and set a load cap for each rotation, and decide what becomes of the central operations team of six: whether it becomes a platform team, an embedded group, the core of a hybrid, or something else, and why.
2. Objectives and signals — define service level objectives (SLOs) measured over the results-day and deadline-day windows themselves, rather than over a month in which those few hours disappear into the average. Scale the burn-rate alerts to those short windows. Then specify the traces and log fields that would let an on-call engineer answer, within minutes, the question “which institution’s applicants are failing, and since when?”
3. Capacity and runtime plan — do the arithmetic for the first hour of results day, starting from 600,000 sign-ins. Plan tests for a cold start and a cold cache, state the scaling limits you will hit first, and calculate the cost per applicant so that the board can see what “boring” will cost.
4. Change and resilience plan — set a change freeze around both days, with an exception register that records every change allowed through it. Make every piece of automation calendar-aware, certificate rotation included. Roll out configuration changes in stages, just as you would code. Finally, plan a game day with a named SimSup, the person who designs and injects the failures, as in Section 7.3.
5. Data and security plan — state the recovery time objective (how long a restore may take) and the recovery point objective (how much recent data may be lost), then run a restore drill and time it against both. Give secret and certificate rotation a named owner, and show where each country’s applicant data resides.
6. Incident and learning plan — staff the incident roles for both days by name. Write status-page templates in advance, so that nobody composes them under pressure. Review last year’s certificate incident with an effort sized to its impact, and commit to an after-action review of this year’s results day however it goes.
7. AI operations plan — build evals on a golden set of questions about deadlines and entry requirements, so that the assistant’s answers are checked against known-correct ones before applicants see them. Define what the assistant is permitted to do and say, and how it degrades when it is unsure: for instance, by pointing the applicant to the authoritative source instead of guessing a date.
8. Ninety-day training and culture plan — set up shadowing and reverse shadowing for every new member of a rotation, run a Wheel of Misfortune built from last year’s certificate incident, and take a Westrum baseline survey so that you can tell, a year from now, whether the culture has moved.

Rubric:

Where you now stand

You began this book with an operator at a console, and you end it in a room full of them, some of them machines, all working one mission. Along the way you have learned to read failure as safety science reads it, to decide how much a machine should do alone and keep the skills it lets fade, and to see a system through its telemetry. You have learned to command an incident while it is happening and to learn from it afterwards without punishing anyone or excusing anyone, and to shape the organization that does all of this. The Resilience book supplies the other half of the craft, the objectives, capacity, delivery and recovery engineered into the system itself; this one has been about the people and machines that run it. The individual techniques will keep changing, as they always have, and many of the tools named in these pages will be replaced. What lasts is a habit of mind: treating reliability as daily work that people and machines produce together, and investing in the people as seriously as in the machines. I have come to believe that this habit, more than any tool, is what makes a results day boring.

References

• Matthew Skelton, Manuel Pais — Team Topologies, 2nd ed.. IT Revolution, 2025.
• Betsy Beyer, Chris Jones, Christof Leng, David Huska, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering, 2nd ed.. O’Reilly, 2026.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering. O’Reilly/Google, 2016 — free online.
• Betsy Beyer, Niall Richard Murphy, David K. Rensin, Kent Kawahara, Stephen Thorne (eds.) — The Site Reliability Workbook. O’Reilly/Google, 2018 — free online.
• Ryan Franco, Brian Brown — How SRE teams are organized, and how to get started. Google Cloud, 2019 — free online.
• David N. Blank-Edelman — Becoming SRE. O’Reilly, 2024.
• Ron Westrum — A Typology of Organisational Cultures. Quality and Safety in Health Care, 2004.
• Nicole Forsgren, Jez Humble, Gene Kim — Accelerate. IT Revolution, 2018.
• DORA — 2025 State of AI-assisted Software Development. Google Cloud, 2025 — free online.
• Amy Edmondson — Psychological Safety and Learning Behavior in Work Teams. Administrative Science Quarterly, 1999.
• Alex B. Haynes et al. — A Surgical Safety Checklist to Reduce Morbidity and Mortality in a Global Population. New England Journal of Medicine, 2009.
• David R. Urbach et al. — Introduction of Surgical Safety Checklists in Ontario, Canada. New England Journal of Medicine, 2014.
• Karl E. Weick, Kathleen M. Sutcliffe — Managing the Unexpected, 3rd ed.. Wiley, 2015.
• Kyle Petroski, Matthew Frail — How we prepare Shopify for BFCM. Shopify Engineering, 2025 — free online.
• Argo Project — Sync Windows. Argo CD documentation — free online.