Building Resilient Systems in the AI & Cloud Era

Production Engineering

© 2026 Oleksiy Pylypenko · Licensed under CC BY-NC-SA 4.0 — share and adapt freely, non-commercially, with attribution. Source on GitHub. Also in the series: Software Architecture in the AI & Cloud Era and Operating Production Systems in the AI & Cloud Era.

Preface

A system that works on the day it ships has passed only the first and easiest test. The harder test begins the morning after, when the system meets production: real users in numbers nobody quite forecast, dependencies that slow down without warning, changes made by forty teams at once, adversaries who probe it every night, and a calendar that sooner or later delivers the busiest hour of the year. Some systems bend under that pressure and recover; others break, and break in ways that spread. The difference is rarely luck. It is usually the accumulated result of many engineering decisions, most of them made long before the bad hour, about what to promise, how much capacity to buy, how to change the system safely, where to draw the walls that contain failure, how to keep data that cannot be redeployed, and how to defend the whole against people who wish it harm.

I call that body of decisions production engineering: the work of designing, measuring, changing and protecting a system so that it withstands production. This book is about that work. It begins with reliability as a product decision, the choice of how much unreliability users can absorb, expressed as service level objectives and spent as an error budget. It turns to performance, capacity and cost, the arithmetic of how much system a promise requires. It follows the runtime and its control plane, the crowd of loops that keep a cloud system in the shape we asked for, and then the delivery pipeline through which every change must pass. It takes up the stability patterns and experiments that keep a failure small, the data and state that make recovery hard, and the security operations that keep defenses current. And it ends with AI workloads, the newest components in production and in some ways the strangest, before a capstone that brings every chapter to bear on one system at once.

None of these techniques is as new as its tooling. Erlang worked out in the 1910s how many lines a telephone exchange needs; Maxwell explained in 1868 why a steam engine’s governor could make the engine hunt; the Bell System chose its nines and engineered to them; AT&T’s network showed in 1990 how a recovery path can carry a failure from switch to switch. The tools have changed almost beyond recognition, and the lessons beneath them have hardly changed at all. I have come to believe that the newest techniques are best understood, and best used, by someone who knows the older lessons that explain why they work, because a team that has the tool without the reason will misuse the tool the first time the situation departs from the tutorial. So each chapter roots today’s practice in its lineage. AI appears where it really changes the practice, and nowhere else.

This is the third book in a series. The first, Software Architecture in the AI & Cloud Era, which this book calls the Architecture book, asks how to shape a system; the second, Building Products & Platforms, is still in preparation. This one asks how to engineer that system so that it withstands production. Operating Production Systems in the AI & Cloud Era, the Operations book, follows it and asks what happens when production fails anyway: how complex systems fail, how people and automation respond in the moment, and how organizations learn from what went wrong. Each book stands on its own, but they share one company and one story, and they were written to be read together.

Influences & Attribution

Much of this book’s method comes from others, and I want to be plain about where. Each chapter’s references point to the original works, which are always the better read; this book’s job is to connect them and show them working together at one company. Service level objectives follow Google’s Site Reliability Engineering and The Site Reliability Workbook, Alex Hidalgo’s Implementing Service Level Objectives, and Ben Treynor and colleagues’ “The Calculus of Service Availability”. Performance and capacity draw on Brendan Gregg’s Systems Performance, Neil Gunther’s Guerrilla Capacity Planning, Arun Kejariwal and John Allspaw’s The Art of Capacity Planning, and J.R. Storment and Mike Fuller’s Cloud FinOps. The runtime chapter leans on Kief Morris’s Infrastructure as Code, Bilgin Ibryam and Roland Huß’s Kubernetes Patterns, and Camille Fournier and Ian Nowland’s Platform Engineering. Delivery draws on Jez Humble and David Farley’s Continuous Delivery and on the DORA research of Nicole Forsgren and colleagues. Stability patterns come from Michael Nygard’s Release It!, deepened by Marc Brooker’s writing on retries and backoff and by Nathan Bronson and colleagues’ work on metastable failures; chaos engineering follows the principles published by its Netflix originators and Kripa Krishnan’s account of Google’s disaster-recovery testing. Data and recovery draw on Martin Kleppmann and Chris Riccomini’s Designing Data-Intensive Applications, security on Building Secure and Reliable Systems by Heather Adkins and colleagues and on Kelly Shortridge and Aaron Rinehart’s Security Chaos Engineering, and the operation of AI systems on Shreya Shankar and Hamel Husain’s work on evals and Chip Huyen’s AI Engineering.

The book’s older foundations come from engineers and scientists who never ran a cloud service: Agner Krarup Erlang on telephone traffic, James Clerk Maxwell on governors, Walter Shewhart on control charts, Jim Gray on why computers stop, Eugene Spafford on the Morris worm, and Joseph Weizenbaum on the trust people lend to fluent machines. The public post-incident reports of Amazon Web Services, Google Cloud, Cloudflare, CrowdStrike, GitLab and GitHub, and the SEC’s account of Knight Capital, supply the book’s real-world cases, and they are described here as history from the public record.

Several terms used in this book as common vocabulary were coined by specific people or groups, and they deserve the credit: error budgets (Google SRE), the USE method (Brendan Gregg), the RED method (Tom Wilkie), the Universal Scalability Law (Neil Gunther), metastable failures (Nathan Bronson and colleagues), shuffle sharding (Amazon Web Services), chaos engineering (Netflix) and the confused deputy (Norm Hardy).

Authorship & Disclaimer

This book was written and directed by Oleksiy Pylypenko, with AI assistance (Anthropic’s Claude). The judgment in it comes from twenty-plus years of practice: platform engineering inside a large bank, API architecture in fintech, and a decade of maintaining a widely used open-source library. The author chose the thesis, the structure, the examples and the trade-offs, edited every chapter, and takes responsibility for every claim, errors included. Encore and every kata company are fictional, and historical events are described from the public record. The content is provided as-is, without warranty of any kind. It is educational material, not professional advice for any specific system, and the license’s full disclaimer of warranties applies.

How to Read This Book

One company runs through all nine chapters. Encore sells tickets to live concerts, and its traffic can rise 500× in minutes when a major tour is announced, which makes it a demanding place to learn how to engineer for production. Readers of the Architecture book watched Encore grow from nine engineers to 400; this book picks the company up the morning after that story ended. Chapter 1 introduces Encore for readers meeting it for the first time, with a map of its services and of where they run, each later chapter adds the parts of the company it needs, and an appendix gathers all of them on one page for reference. One Friday sits on the book’s horizon: the on-sale for a fictional artist named Vesper Hale. This book prepares for it, chapter by chapter, and points out the small, reasonable decisions that will matter on the day. The Operations book tells the morning itself and what Encore learned from it.

Each chapter closes with a kata, a practice brief for a different company with its own scale and constraints, together with a rubric to check your work against. The word comes from the martial arts, where a kata is a set form practiced until it becomes second nature, and the idea is much the same as a musician’s scales: the value lies in repetition in a place where mistakes are cheap, so that the moves are familiar when mistakes become expensive. Every section ends with a short recap and an exercise. The exercises work best done against a system you actually run, because the point is to see the idea in your own production rather than in Encore’s. Each chapter’s references list the field’s standard works, and those marked free online cost nothing to read.

The chapters build on one another, but each can be read on its own. A reader setting objectives for a new service may start with Chapter 1, a reader facing a peak may start with Chapter 2, and a reader whose system keeps falling over in interesting ways may start with Chapter 5. Chapter 1 is the one I would ask every reader to make time for, because the objectives it defines are the yardstick every later chapter measures against.

The Arc

The nine chapters fall into four parts. They move from deciding what to promise, through running and changing the system that keeps the promise, to withstanding the failures that will test it, and they end with AI workloads and a capstone that applies the whole book at once.

Promises and capacity (Chapters 1–2): reliability as a product decision, and performance, capacity and cost.

Running and changing (Chapters 3–4): the runtime and its control plane, and shipping change safely.

Withstanding failure (Chapters 5–7): designing for failure, data and disaster recovery, and security operations.

The destination (Chapters 8–9): operating AI workloads, and the capstone.

Contents

1. Reliability as a Product Decision — SLOs, Error Budgets, and Risk
2. Performance, Capacity, and Cost
3. The Runtime and Its Control Plane — Kubernetes, Serverless, IaC, and GitOps
4. Shipping Change Safely — Continuous and Progressive Delivery
5. Designing for Failure — Stability Patterns and Chaos Engineering
6. Data, State, and Disaster Recovery
7. Security Operations — Secrets, Patching, and the Supply Chain
8. Operating AI Workloads — Models, Evals, and Agents
9. Production Readiness — The Capstone

Appendix: Encore at a Glance

Chapter 1: Reliability as a Product Decision — SLOs, Error Budgets, and Risk

Every promise about reliability is also a promise about what we will stop doing.

How reliable should a system be? Ask a room of engineers and most will answer, “As reliable as we can make it.” The answer sounds responsible, and it is nearly always wrong. The trouble is that reliability is never free. We pay for it in money, in the attention of the people who build and run the system, and above all in the freedom to change things, because every change is a chance to break something, and so each extra degree of reliability makes change a little more expensive. A system made as reliable as possible has spent all three of those currencies on a property its users stopped noticing several nines ago. (A “nine” is one more 9 in the availability figure: 99.9% is three nines, 99.99% is four.) Once we see the cost, the question changes shape. It stops being an engineering question about how high we can climb and becomes a product question: how much unreliability can our users absorb before they would rather be somewhere else, and what will we buy with everything we save by not climbing further?

That question is older than our industry. The telephone network was the first to ask it at scale, and its answer, “five nines,” became a byword for reliability itself. We will follow the telephone engineers first, and then take up the chain that the Architecture book introduced. The chain runs from the service level indicator (SLI), a measurement of how well the service is treating its users, to the service level objective (SLO), a target for that measurement, and on to the error budget, the amount of failure the target allows. Section 1.2 and Section 1.3 define each link precisely. In this chapter we turn the chain into a working instrument: one that helps us decide what is worth measuring, settles arguments between teams on evidence rather than on who argues hardest, and wakes people at night only when it should.

Encore needs that instrument now. Today it has exactly one published objective, for the seat map, and it has 40 teams, each with opinions about what the other teams ought to promise. The Reliability team, the platform group’s ten site reliability engineers, has been asked to close that gap before the next big Friday on-sale. Their task is to give checkout, the on-sale window and every tenant an objective that users would recognize as describing something they have actually felt. (The section that follows introduces Encore for readers meeting it for the first time, with a map of where each of these pieces sits.) The rest of this chapter follows them through that work, and along the way we will see that each choice they face is also a choice about what Encore is willing to stop doing.

Encore, the System We Will Run

Encore is our companion again, as it was throughout the Architecture book, which followed the company from nine engineers to 400. Most of the week it sells concert seats across Europe, in euros, like any other online shop. Then a famous artist’s tour goes on sale, nearly always at 10:00 on a Friday, and within minutes the traffic reaches 500 times its weekday level. Three demands arrive together on that morning. The crowd is enormous, a million people for a few thousand seats. Every seat must be sold exactly once, never lost and never sold to two fans who clicked it in the same instant. And the bots arrive before the fans do. Nearly every structure in Encore answers one of those demands, and much of the engineering in this book lies where they pull against each other.

Figure 1.1 is the map. The fan’s purchase path, the chain of services a fan passes through between arriving and holding a ticket, begins at the Sale Gate, Encore’s virtual waiting room, which lets fans through at a metered rate of about 1,200 a second per event. The rate is chosen so that Seat Inventory, behind it, runs below 80% of what it can carry. That makes the Gate an admission controller: a component that protects a scarce resource by deciding how much work may reach it. A concert hall does the same with doors and ushers, and the comparison holds until we notice that an usher can see the hall filling, while the Gate cannot see the services behind it at all. Its rate is a judgment made in advance, from capacity arithmetic and load tests, and it is only as good as the conditions under which those were made. Beside it, Bot Screening judges, session by session, whether it faces a fan or a script.

Figure 1.1 — Encore’s system map. The purchase path runs down the middle; everything else either hangs off it or listens to it.

Seat Inventory is the single authority on whether a seat is free. It holds a fan’s chosen seats for five minutes while they pay, and it sells each seat once, whatever else is failing. A large event’s seats are split into four shards, each led by one elected leader that puts the shard’s writes into a single order, and a write commits only once a majority of five replicas, spread across three EU locations, has recorded it. Orders & Payments then turns the hold into a sale within a 3 s checkout budget, calling an external payment service provider (PSP). It has two providers, because a slow one once took the whole site down, as the Architecture book told. A circuit breaker sends new payments to the second provider when the first falters, and if both are degraded, verified fans’ seats are held as a pending payment for up to 20 minutes, to be paid once a provider recovers. No ticket is ever issued without an authorized payment. Figure 1.2 follows one fan along this path in the order a person meets it.

Figure 1.2 — One fan’s purchase in eight steps, each tagged with the service that owns it. The admission rate and the checkout budget bound the rush from opposite ends.

The admission rate bounds how many fans begin each second, and the checkout budget bounds how long any one of them may take. Between those two bounds lies everything this book must keep healthy, and many of the failures we will study come from something in the middle quietly breaking an assumption one of the bounds relied on.

Forty teams build these services and carry their own pagers. They stand on a platform whose golden paths, the supported routes from a service template to production, deliver everything through Git into Kubernetes cells and serverless functions across two EU regions, as Figure 1.3 shows. Venues that buy Encore as a service are housed in cells, self-contained copies of the stack that keep one tenant’s trouble away from the rest. Within the platform group sits the Reliability team, ten site reliability engineers who own the tooling for objectives, the incident process, the game days and the readiness review held on the Wednesday before each big on-sale. On the morning itself, a self-service workflow called curtain-up warms the purchase path, and a change freeze holds the path still from two hours before the sale until it ends. The Reliability team advises and rehearses but runs no one else’s services, so that understanding stays where the pagers are.

Figure 1.3 — Where Encore runs: two EU regions of Kubernetes cells and serverless edges, reconciled from one Git repository. The seat ledger commits each write to a majority of five replicas in three locations, two in each region and a witness in a third, which is why losing a region loses no sale.

One Friday sits on this book’s horizon: the on-sale for Vesper Hale, a fictional artist whose 12-date European tour will bring about 1.1 million fans to the Gate in the same minute. This book is about the engineering that prepares for such a morning, the objectives, capacity, runtime, delivery, failure design, data, security and AI practice that decide how much of it the system can withstand. Along the way it points out small, reasonable decisions, each defensible on its own day, that will matter when the morning comes. The morning itself, and what Encore learned from it, belong to the Operations book. Everything else is introduced where it first does something, and the appendix keeps the whole company on one page.

1.1 Why 100% Is the Wrong Target

The Bell System and its nines

For most of the twentieth century the Bell System ran the American telephone network as a regulated monopoly. The Bell System was a family of companies: AT&T, which operated the network; Western Electric, which built its equipment; and Bell Telephone Laboratories, which did its research and design. Its engineering culture rested on one expectation, that when a subscriber lifted the handset the dial tone would always be there. That expectation was written into its machines. The first No. 1 Electronic Switching System (ESS), in service at Succasunna, New Jersey, from 1965, carried a design objective of no more than two hours of total downtime in 40 years. Forty years is about 350,000 hours, so two hours of downtime works out to a little better than 99.999% availability, the “five nines” that later became a byword for reliability itself. The engineers met the objective by duplicating the switch’s central processor. One copy carried calls while the other stood ready, so that when a fault appeared in one processor it could be isolated and repaired while its twin kept the exchange running. It is worth noticing that the famous number was not achieved by making a single processor perfect. It was achieved by accepting that processors fail and building a structure in which one failure did not reach the subscriber.

Two details of this story teach more than the famous number does. The first is that the objective belonged to the switch, and the switch was only one link in a chain. To hear a dial tone, the subscriber also depended on the copper loop running from the exchange to the house, on the handset itself and on the weather, which could bring down a line in a storm whatever the switch was doing. A switch with five nines did not give the subscriber five nines; it gave the subscriber one very reliable link in a chain whose weaker links were elsewhere. The second detail is more surprising: the same network was deliberately engineered to fail some calls. Trunks, the circuits that connect one exchange to another, were expensive, and the number of them between two exchanges was sized to a grade of service, a target probability that a call is blocked in the busiest hour of the day. A common target was one call in a hundred. The engineers chose that figure on purpose, because building enough trunks that no call was ever blocked, even at the busiest moment of the busiest day, would have cost a fortune for capacity that sat idle the rest of the time. So the most reliability-minded engineering organization of its century accepted a small failure rate at its peak, chose it knowingly and put a price on it. That is an error budget in all but name. Chapter 2 takes up the mathematics of sizing capacity to a target like this; here the point is the attitude. The Bell engineers did not treat reliability as something to maximize. They treated it as something to choose.

What each nine costs

The nines are easy to say and hard to feel, because a percentage hides the quantity that operators actually live with, which is time. Figure 1.4 translates each level of availability into the downtime it allows, both per year and per the rolling 28-day window that Encore uses to measure its objectives.

Figure 1.4 — The nines as time. Each additional nine divides the allowance by ten; the cost of staying inside it rises by much more.

Read the right-hand column as an operator would, by asking what kind of bad day each row can absorb. At 99.9%, the allowance is 40 minutes in 28 days. That is enough for Encore to survive one bad deploy a month that takes half an hour to notice and roll back: someone is paged, looks at a dashboard, decides the new version is at fault and reverts it, and the month’s budget is bent but not broken. At 99.99%, the allowance is four minutes. Four minutes is less time than it takes a person to be paged, wake up, open a laptop and understand what they are looking at, so a human in the loop is no longer affordable. Detection and rollback must both be automatic, which means building and trusting the machinery that does them. At 99.999%, 24 seconds, even the automation must be very fast and the failure must usually be contained before most users meet it, which is roughly what the duplicated processor of the No. 1 ESS was for. This is why the caption says the cost rises by much more than tenfold. Each nine changes the kind of engineering a team must do, as well as its amount: a team moves from careful people, to automated recovery, to redundancy designed so that failure is absorbed before anyone sees it.

All of that cost would be worth paying if users could feel the difference, and mostly they cannot. Users cannot perceive reliability beyond that of their own path to us. A fan reaches Encore’s seat map through a phone, a mobile network or a home router, an internet service provider and the public internet, and each of those fails now and then for reasons that have nothing to do with Encore. This is the copper loop and the storm again, in a new form. A fan cannot tell a 99.99% seat map from a 99.999% one, as the Site Reliability Engineering book observes, because the unreliability of everything between the fan and Encore swamps the difference. The reliability we buy beyond what users can perceive is not free. It is paid for in change not made: in features delayed, in deploys held back, in engineering time spent defending a margin nobody notices. There are exceptions, and they matter. On paths where failure harms someone rather than annoys them, clinical or safety systems among them, what users can perceive is the wrong yardstick, and regulators set the floor. Encore sells concert tickets, and for its paths the yardstick holds.

Civil engineers face a version of the same choice and have long made it openly. A levee is designed against a flood of a stated size, not against every flood the river could ever produce, because a levee high enough for any flood would cost more than the town it protects. The design flood is a choice, made in public, about how much risk the town will carry and what it will spend instead. The analogy breaks in one instructive place. A river floods whatever the engineers do, but many of a software system’s failures come from the changes its own team makes. Setting the target therefore decides not only how much protection to build but also how freely the team may go on changing the thing it protects.

Seen this way, a target of 100% is not a very high standard. It is a refusal to choose. A team that promises 100% has promised something it cannot keep, so every incident becomes a breach, and there is no agreed line between an acceptable bad day and an unacceptable one. In the absence of such a line, the decisions still get made: whether to freeze releases, whether to cancel a launch, whether to add another layer of approval. They are made one incident at a time, under pressure, by whoever is most frightened in the room. I have come to believe that this is the real cost of the 100% target. It does not buy reliability; it hands the choice about risk to fear.

This is why reliability is a product decision. Choosing a target means trading something users value, a more reliable seat map, against something else users value, such as a feature fans have been asking for, and weighing one against the other is the product owner’s job. Engineers have an essential part in the decision, but it is a different part: they can say what each option costs, in engineering effort and in the freedom to change, and what each nine would demand of the system and the people who run it. Encore has made this decision once, for the seat map, whose objective allows about 40 minutes of full-outage equivalent in 28 days. The rest of this chapter is about making it well for everything else, beginning with what exactly we should be measuring.

Recap. The Bell System chose its nines, engineered to them and knowingly accepted blocked calls at the busiest hour. Each further nine changes the kind of engineering required, not merely its amount, and users cannot feel reliability beyond that of their own path to us. So the target is a product choice about where to spend the freedom to change, except on paths where failure causes harm.

Exercise 1.1. For a service you run, find the row of Figure 1.4 at which a single incident, handled by your current detection, paging and rollback, would consume the whole monthly allowance. Is your published target above or below that row?

1.2 SLIs and SLOs That Users Would Recognize

From journeys to indicators

Two definitions come first, because everything in this chapter rests on them. A service level indicator (SLI) is a measurement of service, expressed as the proportion of valid events that were good. A service level objective (SLO) is a target for that proportion over a stated window of time. Written down like that, both sound like bookkeeping, and the bookkeeping is in fact the easy part. The hard part is deciding which events to count and what to call good, and there the choices are many and most of them are subtly wrong.

The most useful test I know is a simple one. Imagine showing the SLI, in plain words, to a fan. Would the fan recognize it as something they have felt? “99.9% of requests to the Orders service returned without a server error” fails the test, because no fan has ever experienced a request to the Orders service. “Of the fans who tried to pay, 99.9% either got their tickets or were told clearly why not, within three seconds” passes it, because every fan who has bought a ticket has lived through exactly that moment. The test pushes us away from components, which are how we build the system, and toward user journeys, which are how people use it: joining the waiting room, holding seats, checking out. For each journey we then define three things: the event we count, what makes that event good, and the place where we measure it.

Checkout shows why this takes care. The naive SLI is the one engineers reach for first: count the HTTP 5xx responses from Orders and call everything else good. It gets several cases wrong, in both directions. When a fan’s bank declines the card, Orders returns a perfectly healthy response and the system has done its job, so the event is good, however unhappy the fan may be. When both of Encore’s payment service providers (PSPs), the outside companies that actually move the money, are degraded at once, and a verified fan’s seats are held pending payment, the fan has been told the truth about what is happening and has lost nothing, so that event is good as well. But a decline that Encore caused itself, by timing out while waiting on the PSP or by breaking the strong customer authentication step, is bad, even though the fan sees the same polite “payment declined” message as in the first case. The 5xx count would call it good, since no server errored. The fan’s experience and the server’s status code have quietly parted company, and only an indicator built around the journey can tell the two declines apart.

Putting those cases together gives Encore’s checkout SLO, which reads: of checkout attempts by admitted fans, 99.9% end in a confirmed order, a hold pending payment or an issuer’s decline, correctly explained, within 3 s, measured at the gateway over a rolling 28 days. Each clause does work. “Admitted fans” fixes the valid events, so that bots turned away at the door do not enter the count. The three good outcomes are the three honest endings a checkout can have. “Correctly explained” is what separates the bank’s decline from Encore’s own. And the 3 s is a budget the whole path shares; about 800 ms of it belongs to the PSP, which leaves Encore’s own services the remainder.

Ratios, thresholds and percentiles

Once we know what to count, we have to choose how to count it, and there are two broad forms. An event-based SLI, also called a ratio SLI, divides good events by valid events: of all the checkout attempts, how many were good. A time-based SLI divides time into slices and counts the good slices, such as the proportion of minutes in which the error rate stayed below 1%. The difference lies in how each one weights a busy moment. In the event-based form, a minute in which tens of thousands of fans tried to check out contributes tens of thousands of events to the ratio, while a quiet minute at 04:00 contributes a handful, so the busy minute dominates. That is usually what we want, because a failure when many people are using the service hurts many people. In the time-based form every minute counts once, however crowded it was. That suits services with little traffic, where a ratio of a few events would lurch with every failure, and it suits contracts, whose readers tend to think in minutes of downtime.

Latency is where well-meant indicators most often go astray. “p99 latency below 800 ms” and “99% of requests faster than 800 ms” sound like the same sentence said two ways, and for a single server over a single period they are. The trouble begins when we combine measurements, because percentiles cannot be averaged. Suppose one server handled 1,000 requests and another, unlucky, handled ten requests that were all slow. Their p99 values might be 300 ms and 2 s, and the mean of the two, 1.15 s, describes no request anybody made and says nothing reliable about how the 1,010 requests fared as a whole. The mean of ten servers’ p99 values is likewise not the fleet’s p99. Counts, by contrast, simply add. If 995 of the first server’s requests and none of the second’s came in under 800 ms, the fleet served 995 of 1,010 requests within the threshold, about 98.5%, and that is exactly the proportion fans experienced. The ratio form survives being summed across servers, regions and time; the percentile form does not.

Encore therefore states its latency objectives as ratios, and it records latency in histograms with a bucket boundary placed exactly at each threshold. The reason for that last detail is easy to miss. A histogram stores counts per bucket and nothing finer. If its buckets ended at 500 ms and 1 s, it could say how many requests took less than 500 ms and how many took less than 1 s, but any answer about 800 ms would be an interpolation, which is a polite word for a guess. A histogram can only answer questions about the boundaries it was given, so the boundaries have to be chosen with the objectives in hand.

Where we measure matters as much as what and how. Encore measures at the gateway, because a server timing itself cannot see the requests queued in front of it: a request that waited two seconds for a free worker and was then served in 50 ms looks fast to the server and slow to the fan. The gateway is closer to the fan and catches that queueing, but it is still not the fan. It cannot see a failed DNS lookup, a content delivery network fault or a phone that never managed to connect, since in each case the request never reached it. So the purchase journeys also carry client-side measurement from Encore’s app, which sees what the fan’s device saw. The ticket-delivery SLI goes one step further and is confirmed by the wallet’s receipt, because Notifications can only know that it sent something, never that it arrived. The general rule is to measure as close to the user as we can get, and to be honest about what each vantage point cannot see.

Windows, and the hour that matters

The third choice is the window, the stretch of time over which the proportion is computed. I find it helpful to think of the window as the SLO’s memory: it decides how long a bad day keeps counting against the service. A rolling window remembers continuously. A bad Friday weighs on the team for exactly the length of the window, and then it slides out of the back and is forgiven. Encore follows The Site Reliability Workbook in choosing four weeks, 28 days, and the choice is less arbitrary than it looks. Traffic at a ticketing company has a strong weekly rhythm, with on-sales clustered on particular mornings, and a 28-day window always holds exactly four of each weekday. A 30-day window would sometimes hold four Fridays and sometimes five, and the indicator would wobble with the calendar rather than with the service. A calendar window, such as the calendar month, suits reports and contracts, because people read and sign those by the month. But it creates strange incentives: a budget nearly spent on the 29th is restored in full on the 1st, so the same failure matters a great deal on one day and hardly at all two days later.

Fans, however, do not remember months. They remember 10:00 on the Friday their artist went on sale. A 28-day objective can stay green while that hour was a disaster, because the hour’s failures are diluted by 27 quiet days on either side. A touring musician understands this instinctively: a tour can average well across 30 nights, and the fans in the one city where the sound failed will still talk about nothing else. So for each on-sale classed as major at the readiness review the Wednesday before, Encore also evaluates an on-sale-window SLO: an objective computed over a fixed window around a scheduled event, in this case the first 60 minutes of the sale, with a budget of its own. The comparison with a tour holds only so far. A concert’s quality is judged by everyone in the room, whereas this SLO is judged by a ratio, and a ratio over an hour of very uneven traffic behaves differently from one over a month, as the next point shows.

The on-sale-window SLO is Encore’s own adaptation, and there is no published practice of that name behind it. It rests on the Workbook’s advice that a window should match how users experience the service, and its costs were accepted knowingly. There are three. First, a short window that is heavily front-loaded, with most fans arriving in the opening minutes, is statistically noisy: a few unlucky minutes can swing it, so a miss is a strong signal to investigate and a weaker one to conclude from. Second, burn-rate alerting, the usual way of turning an SLO into a page, cannot work inside a window this short, for reasons Section 1.4 sets out. Third, error budget policies are written for rolling windows, and per-event budgets have no policy precedent to lean on. So a missed on-sale-window SLO earns a postmortem and an item for the next readiness review, never a change freeze.

The on-sale SLI also counts fans rather than requests, and that choice changes the number considerably. Consider a fan who hits an error and retries ten times before the eleventh attempt goes through. Counted by requests, that fan generated ten bad requests, and would have done so whether or not the eleventh had succeeded, so a request-based indicator charges the service ten times for one frustrated person, and charges it most heavily for the fans who are most determined to keep trying. Counted by fans, that person had one bad on-sale, however many requests it took. That is what the fan will remember and what the SLO should measure. The price is a per-fan key shared by the Gate and the gateway, so that the admission the Gate granted and the requests the gateway served can be joined into one journey. It is harder to build than it sounds, because the two systems see the fan at different moments and for different purposes, and both must agree on who that fan is throughout the hour.

The table below gathers the objectives this section has argued for. The “shop once admitted” and major on-sale rows share one indicator, judged over two different windows.

Each objective in the table is a product decision, signed by the journey’s product owner, for the reason Section 1.1 gave: only the product owner can weigh a little more reliability against the features it would cost. Each began near what the journey had actually delivered in the previous quarter, so that the first target described a service the team already knew how to run, and each is reviewed quarterly. That review matters more than it seems. Users’ expectations move, traffic grows and the system changes beneath the number, and an objective nobody revisits slowly drifts into irrelevance, still green on the dashboard long after it has stopped describing anything a fan would recognize.

Recap. Good SLIs start from user journeys and define three things: the event, what makes it good, and a place of measurement as close to the user as we can get. Ratios add up correctly across servers and time, while percentiles do not, so latency belongs in ratios backed by histograms with boundaries at the thresholds. Rolling windows give an SLO a steady memory, and Encore gives its scheduled peaks a window shaped like the event, counted per fan, knowing the costs of doing so.

Exercise 1.2. Take one SLI you have. Find a case it counts as good that the user felt as bad, and a case it counts as bad that nobody noticed. Rewrite the SLI so it counts both correctly, and say where you would have to measure to make the rewrite possible.

1.3 Error Budgets as a Negotiation Instrument

A contract written in calm weather

Section 1.2 gave each of Encore’s journeys an objective. Every objective short of 100% carries a second number with it, the failure it leaves room for, and that number is where reliability stops being a matter of measurement and becomes a matter of negotiation. Two definitions will carry us through the section. An error budget is the amount of unreliability an SLO permits over its window: for a 99.9% objective, the 0.1% of events that may go wrong before the objective is missed. An error budget policy is the written agreement about what the teams involved will do as that budget is spent. The budget is arithmetic, and any dashboard can compute it. The policy is something else entirely: an agreement among people with different interests about how to share something scarce, and like every such agreement it is only as good as the willingness of the people who made it.

The policy’s value lies in its timing. Every engineering organization eventually has the argument about whether to keep shipping features or stop and stabilize, and left to itself that argument happens at the worst possible moment, during or just after an outage, when everyone is tired, someone is angry and the loudest voice tends to win. A policy moves the argument to a quiet Tuesday, when nobody’s work is on fire and the people at the table can still think about the general case rather than the particular wound. Touring productions learned the same lesson long ago: what happens if the headliner falls ill, or the venue loses power, is settled in the contract before the tour leaves, because nobody negotiates well at the stage door an hour before the show. The analogy holds as far as the timing and breaks at enforcement, since a tour contract has lawyers behind it and an error budget policy has only the goodwill and authority of the people who signed it, a point we return to in a moment.

Encore’s purchase-path policy is taken nearly unchanged from the example in The Site Reliability Workbook. It has three clauses from that example. While budget remains, teams ship at will. Once the budget is exhausted, only reliability work, security fixes and risk-reducing changes ship until it recovers. And any incident that consumes more than 20% of a budget requires a postmortem. Between the first two clauses Encore added a tier of its own: when less than 25% of the budget remains, every change to the journey is also reviewed, in the same pull request, by the site reliability engineer embedded with the team. The form of that review is deliberate. DORA’s research found that approval by a body outside the team, such as a change advisory board, slows delivery without making it safer. So the embedded engineer does not act as an outside approver. They review as a peer who knows how the journey tends to fail, they can point out the risky line in a change, and they hold no gate that could stop it from shipping.

A policy like this holds only because people with authority over both kinds of work have signed it. At Encore those people are the heads of engineering and product: one decides how much engineering effort goes into reliability, the other decides which features fans will get and when, and a freeze on feature work touches both. Without their signatures the policy is a suggestion, and suggestions give way the first time a launch date is at stake. Many organizations publish SLOs and never enforce any policy about them, and in those organizations the targets become decoration, numbers on a dashboard that nobody’s behavior depends on. Encore also follows the advice of Alex Hidalgo, in Implementing Service Level Objectives, that a policy should guide decisions rather than freeze teams mechanically. The thresholds tell people when to stop and talk; they are not meant to replace the talking.

There is a second half to the idea, and it is the half engineers find harder to accept: budgets are meant to be spent. The usual way to spend one is on faster change, shipping more often and taking more risks with each release, because the budget exists precisely so that the organization can buy speed with a known, bounded amount of failure. The Site Reliability Engineering book tells of Google’s Chubby lock service, which was so reliable in practice that its users came to depend on more availability than it promised. The service’s owners responded by taking it down with planned outages until its real availability matched its objective, so that the services depending on it would discover their hidden assumption on a planned day rather than an unplanned one. The lesson generalizes. Unspent budget is reliability nobody promised, and sooner or later someone will build on it as though it were a promise.

The idea underneath the budget is older than site reliability engineering. In the 1920s Walter Shewhart, working for the Bell System, devised control charts, which told managers when the variation in a manufacturing process was simply the process behaving as designed and when it had a cause worth hunting. An error budget makes the same distinction for a service. Failures that stay within the budget are the ordinary variation we agreed to accept, and chasing each one would cost more than it returns. Failures that burn the budget quickly are a signal that something has changed and deserves attention. What the budget does not settle on its own, as Encore was about to discover, is whose attention it deserves when the change that burned it belonged to someone else.

The dispute between On-Sale and Bot Screening

Encore’s policy met its first real test in the spring. Two teams were involved. The On-Sale team owns the waiting room and the journey of a fan who has been admitted to the sale. Bot Screening writes the rules that keep bots out of the waiting room and that continue to watch sessions after they are admitted. That season Bot Screening switched on a new device-fingerprint rule during a mid-size arena on-sale, one the readiness review had classed as major. The rule ran at the hold step, where a fan places a hold on seats before paying, which is after admission. In 15 minutes it blocked about 6,000 admitted fans. Most of them, as Support’s inbox later showed, were real people using an older phone browser whose fingerprint the rule mistook for a bot’s.

The cost showed up in On-Sale’s numbers rather than Bot Screening’s, and the reason is the way the indicator was written. The “shop once admitted” indicator, as it stood then, counted every journey that ended on a block page as bad, with no way to tell a blocked bot from a blocked fan. About 200,000 fans were admitted in the first hour of that sale. The 6,000 blocks alone were 3% of them, and together with the hour’s ordinary errors they brought the on-sale-window SLO in at 96.8% against its objective of 99%. The damage did not stay inside the hour. The 28-day objective for the same journey is 99.9%, and a typical four weeks admits about 8.6 million fans, so that objective allows some 8,600 bad journeys in its window. The 6,000 blocked journeys took 70% of that allowance in a quarter of an hour, which dropped the remaining budget below 25% and put the On-Sale team into the review tier. Figure 1.5 shows the burn.

Figure 1.5 — The On-Sale team’s 28-day error budget through the spring on-sale. Ordinary spending keeps the budget well above the steady burn line; one rule from another team takes 70% in 15 minutes and drops the budget below the policy’s 25% line.

Both teams had a fair case, which is what made the dispute instructive rather than merely awkward. On-Sale argued that a real fan blocked in the middle of a purchase is the worst error the system can make, worse than a slow page or even a failed one, and that it had now paid for someone else’s change with its own freedom to ship. Bot Screening argued that a block is the system doing its job, and that an indicator which charged it for every bot it caught would, over time, teach it to let bots through. Both were right, and the reason they could both be right lies in the shape of Bot Screening’s work. Its mistakes come in two kinds that are never equally visible. A bot let through shows up quickly and loudly, because the resale market reports it within the hour as tickets appear on secondary sites. A fan wrongly turned away is almost silent, because only the fan can report it, and many fans simply give up instead. A team that sees one kind of error clearly and the other dimly will drift, quite reasonably, toward the error it cannot see.

The Reliability team brokered the resolution, and it began with the question of what the indicator should count. The Workbook’s example policy already allows out-of-scope traffic, such as load tests and abusive clients, to be excluded from the budget. That clause is easy to write and hard to apply, because it does not say how to tell an abusive client from a customer. Encore gave the clause an operational test. Every block now offers a challenge and a way back into the sale. A blocked session that fails the challenge or abandons it is treated as bot traffic, and it leaves the journey’s valid events altogether, neither good nor bad. A blocked session that passes the challenge is a false block: a person the rules turned away. A false block counts as bad even though the fan got back in, since the fan had already lost the seats they were holding, whatever happened next. Applied to the spring, the test did not let anyone off. Most of the 6,000 blocked fans were real people and therefore false blocks, so the burn stood.

Once the indicator counted the right thing, the rest of the settlement followed. First, the policy now attributes consumption: the review tier applies to the team whose change caused the burn, so it was Bot Screening, not On-Sale, that went under review. That goes a step beyond the Workbook, whose example only spares a team whose budget was spent by another team’s outage and says nothing about putting the cause under review. The step carries a risk worth naming. Attribution can drift into blame, and blame assumes a single culprit, when, as the Operations book shows in its chapter on learning from incidents, a burn often has more than one cause. Encore uses it with care for that reason.

Second, Bot Screening gained an indicator of its own, the false-block rate: false blocks divided by admitted sessions screened, with an objective of no more than one in a thousand. This gives the team a number that rises when it errs toward caution, to set against the resale market’s loud report of the bots that got through, so that both kinds of mistake are finally visible to it. The rate is a proxy, and it errs in both directions. Solver farms, services that pay people or software to pass challenges, let some bots pass and be counted as false blocks, which inflates the rate; and some real fans give up rather than attempt the challenge, which hides them among the bots and deflates it. So the team checks the measure every month against a sample of Support complaints. Cloudflare offers the same measure as a “challenge solve rate” for tuning rules; giving it an objective is Encore’s own step.

Third, new rules now run in shadow mode through one on-sale before they enforce anything. In shadow mode a rule logs what it would have blocked without blocking it, so the team can count its would-be false blocks before any fan pays for them. This is an established practice, which AWS WAF calls Count and Cloudflare calls Log.

Nobody thought to ask the same of the rules already in force. Among them was a velocity rule, written years earlier, that watches how fast and how often a session acts. It had never caused a dispute, so it was never reviewed, and it was never brought under the new arrangement of a challenge and a way back. That omission will matter on the morning of the Vesper Hale on-sale, which the Operations book follows minute by minute in its Chapter 5.

Recap. An error budget policy is an agreement, written in calm weather and signed by people with authority over both features and reliability, about how teams share a scarce allowance of failure, and the allowance is meant to be spent. When a dependency spends another team’s budget, the answer has three parts: a clear definition of what counts, attribution of the burn to its cause, used with care, and an indicator that makes the dependency’s own errors visible to it.

Exercise 1.3. Draft a one-page error budget policy for a journey you own. State the thresholds, what changes at each one, who grants exceptions, which traffic is out of scope and how you would tell it apart, and how a burn caused by another team is attributed. Then ask that team to read it.

1.4 Alerting on Burn Rate

Fuel flow and the fuel gauge

The oldest question in operations is when to wake a human. For most of the field’s history the answer was a threshold on some cause: page when CPU passes 90%, when a disk fills, when a queue grows long. Such alarms are easy to write and hard to live with, because a busy CPU may harm no one while a quiet one may be serving errors to every fan in the queue. An SLO offers a more principled answer, because it lets us page on the thing users actually feel: wake someone when the error budget is burning fast enough to matter. To make “fast enough” precise we need one more term. The burn rate is the speed at which a service consumes its error budget, measured relative to the speed that would use the budget up exactly at the end of the window. A burn rate of 1 spends the whole budget in exactly one window; a burn rate of 10 spends it in a tenth of the window; a burn rate below 1 means the service is living within its means.

Pilots learn early to tell level from flow. The fuel gauge shows how much remains, while the fuel-flow indicator shows how fast it is going, and a leak shows up in the flow long before the gauge looks alarming. The two readings answer different questions, and an error budget has the same pair. Budget remaining is the gauge, the number the error budget policy of Section 1.3 acts on over days and weeks. Burn rate is the flow, the number an alert should act on within minutes. Where the comparison fails is just as telling: an engine burns fuel steadily, while errors arrive in bursts, so any single reading of burn rate is noisy. A five-minute spike can look like a catastrophe and be nothing at all; a slow leak can look like nothing and quietly drain a month. The Site Reliability Workbook works through simpler rules first, and finds that each one either pages too often or pages too late. It arrives at multi-window, multi-burn-rate alerting: page only when the budget is burning fast over a long window and is still burning over a short one. Two windows, read together, smooth the bursts without slowing the response, and the next subsection shows how.

The arithmetic

The Workbook expresses its rules in terms of budget consumed, which is the quantity an on-call engineer cares about, and converts each one to a burn rate, which is the quantity a monitoring system can measure. For a 30-day window, which holds 720 hours, it suggests three rules:

• page if 2% of the budget is consumed in one hour. That is a burn rate of 0.02 × 720 h ÷ 1 h, or 14.4; left alone, the service would exhaust its whole budget in 50 hours;
• page again if 5% is consumed in six hours, a burn rate of 6;
• open a ticket rather than page if 10% is consumed in three days, a burn rate of 1.

The pattern behind the three rules is a trade between urgency and confidence. A fast burn is dangerous and obvious, so a short look suffices; a slow burn is less urgent but harder to distinguish from noise, so it earns a longer look and a gentler response.

Each rule then pairs its long window with a short one, by convention a twelfth of its length, and both windows must exceed the threshold before the alert fires. The two windows do different jobs. The long window supplies significance: an hour of evidence is hard to fake with one unlucky minute. The short window supplies timeliness, and its real value shows at the end of an incident rather than the beginning. Once the bleeding stops, the long window still remembers the bad hour and stays above the line for a long time, but the short window forgets within minutes, drops below the line, and so resolves the page. Without it, the on-call engineer would fix the problem and then watch the alert keep firing for most of an hour, which teaches people to distrust their alerts.

Figure 1.6 — Multi-window, multi-burn-rate alerting, with the Workbook’s parameters for a 30-day budget.

Encore’s windows are 28 days rather than 30, but it kept the Workbook’s burn-rate multipliers, as most SLO tools do. The multipliers are what the monitoring system checks, so keeping them means keeping well-tested rules; the price is only that the budget fractions shift slightly, since 28 days hold 672 hours rather than 720. An hour at 14.4 now spends about 2.1% of the budget, and six hours at 6 about 5.4%. The next step is to turn a burn rate into a number a query can compare against. Take the seat map’s objective, 99.9% of requests within 800 ms, which leaves an error budget of 0.1% of requests. A burn rate of 1 means a bad-request ratio of exactly 0.1%; a burn rate of 14.4 means a ratio 14.4 times that, above 1.44%.

Encore’s monitoring runs on Prometheus, where a recording rule is a query evaluated on a schedule and stored as a new time series, so that expensive expressions are computed once and read cheaply many times. Encore records the seat map’s bad ratio once per window: 5 minutes, 30 minutes, 1 hour, 6 hours and 3 days. Each recording counts the requests that were good, meaning they did not return a server error and completed within the 0.8 s histogram bucket, divides by all requests, and subtracts from 1. The alert rules then compare those ratios against the thresholds:

groups:
- name: seat-map-slo   # 99.9% of requests <= 800 ms over 28 days: budget 0.001
  rules:
  - record: seatmap:bad_ratio:rate5m   # likewise for 30m, 1h, 6h and 3d
    expr: 1 - sum(rate(gateway_request_duration_seconds_bucket{route="seat-map",code!~"5..",le="0.8"}[5m]))
            / sum(rate(gateway_request_duration_seconds_count{route="seat-map"}[5m]))
  - alert: SeatMapBudgetFastBurn
    expr: (seatmap:bad_ratio:rate1h > 14.4 * 0.001 and seatmap:bad_ratio:rate5m > 14.4 * 0.001)
       or (seatmap:bad_ratio:rate6h > 6 * 0.001 and seatmap:bad_ratio:rate30m > 6 * 0.001)
    labels: {severity: page}
  - alert: SeatMapBudgetSlowBurn
    expr: seatmap:bad_ratio:rate3d > 0.001 and seatmap:bad_ratio:rate6h > 0.001
    labels: {severity: ticket}

Read the fast-burn alert as two sentences joined by “or”: the one-hour and five-minute ratios both exceed 14.4 times the budget, or the six-hour and thirty-minute ratios both exceed six times it. Either condition wakes someone. The slow-burn alert asks only whether the service has been spending faster than it can afford for three days and is still doing so now, and its answer goes to the team’s queue instead of anyone’s phone.

Nobody at Encore writes these rules by hand, and the reason is drift. A rule copied from a wiki carries the objective that was true on the day it was copied, and when the objective later changes, say from 99.9% to 99.95%, the copied rules go on enforcing the old one without complaint. So the golden path generates the rules from each service’s SLO entry in the service registry, and changing the objective changes the alerts in the same commit. Open-source generators such as Sloth and Pyrra work the same way, expanding a few declared lines into the full set of recording and alert rules. It is a small instance of a large principle: when two artifacts must agree, derive one from the other, because two copies kept in step by good intentions eventually stop agreeing.

Every alert rule is, in effect, a promise about how long a failure can go unnoticed, so the useful way to review one is to compute its detection times. Those times are the alert’s real specification. Suppose the seat map fails completely, so every request is bad. The one-hour ratio climbs by one percentage point for every 36 seconds of failure, and it crosses 1.44% after about 52 seconds; the five-minute ratio crosses sooner, so the page fires in under a minute. Now suppose a subtler fault, where one request in 50 is slow, a bad ratio of 2%. The five-minute ratio passes 1.44% within a few minutes, but the one-hour ratio rises slowly, because most of the hour it averages over came before the fault, and it crosses the line only after about 43 minutes. By then the incident has spent about 2% of the 28-day budget, which is exactly what the rule was designed to tolerate before waking anyone. Whether 43 minutes is acceptable is a product question, not a monitoring one, and that is the point of writing the number down. Figure 1.7 follows a third case, a partial outage with one request in ten failing for 40 minutes, through both windows: the page fires about nine minutes in, when the one-hour ratio crosses the line, and resolves about four minutes after the fix, when the five-minute ratio falls back below it.

Figure 1.7 — A fast-burn page through one incident. The one-hour ratio decides when the page fires and the five-minute ratio decides when it stops: without the short window, the page would outlast the fix by more than three-quarters of an hour.

Where the rules run out

Burn-rate alerting is a statistical method, and like every statistical method it needs a sample large enough to mean something. It needs traffic. At a few requests a minute, a single failure moves the ratio by whole percentage points, so one unlucky request can look like a fast burn and page someone for nothing. Section 1.5 meets the same problem again with small tenants, whose traffic is too thin for ratios to settle.

The rules also cannot protect an on-sale-window SLO, the 60-minute objective Section 1.2 introduced for major on-sales. Its target is 99% of fans served well over 60 minutes, which leaves a budget of 1% of the hour. A total failure spends that budget in 36 seconds, and sooner in practice, because admissions crowd into the first minutes of a sale and so do the bad requests. The alerting windows make things worse. The fast-burn rule’s long window is one hour, as long as the SLO window itself, so by the time an hour’s worth of evidence had accumulated the sale would be over, and the page could fire only after the budget was long gone. Shrinking the windows does not rescue the method, because over a few seconds the ratios become too noisy to trust. Here the arithmetic of burn rates simply runs out of room.

For major on-sales, therefore, Encore has people watch the signal. From the moment the Gate opens, the owning teams and an incident commander watch one-minute per-fan error ratios, the share of fans who hit an error, counted once per fan as Section 1.2 described. A page stays armed all the same, because attention fades over a long sale and trouble can begin in a component nobody happens to be watching. That page uses a rule rescaled to the hour: it fires at the On-Sale on-call when both the one-minute and the five-minute per-fan error ratios exceed 10%. Ten percent is ten times the hour’s 1% allowance, a burn rate of 10, at which the whole on-sale budget would be gone in six minutes. Google’s practice for launches, as the SRE book describes it, is the same arrangement: people watching closely, with the alerts still switched on. The two are complements. People notice what no rule anticipated, and rules keep noticing after people have stopped.

Recap. Burn rate measures how fast the error budget is being spent, and it tells us when to act long before the budget remaining looks alarming. Pairing a long window with a short one gives each alert both significance and timeliness: the long window keeps noise from paging, and the short one lets the page end when the incident does. Computing detection times turns an alert rule into a specification that a team can argue about, and generating the rules from the SLO keeps them faithful to it as it changes. Where the window is too short or the traffic too thin for the statistics to work, as in an on-sale hour, people watch, with a rescaled page as the backstop.

Exercise 1.4. For one SLO you own, compute the thresholds for the three rows of Figure 1.6, expressed as bad-request ratios for your objective. Then work out how long a total outage and a 5% partial outage would each take to page. Would your users accept those times?

1.5 SLOs Across Dependencies and Teams

Composition only subtracts

So far we have treated each SLO as if it belonged to one service standing alone, but no service meets its SLO alone. When a fan checks out, the request passes through the Gate, Seat Inventory, Orders & Payments and the payment service provider, and if any one of them fails, the fan’s checkout fails with it. We met the same truth at the start of the chapter: the Bell System’s famous objective belonged to the switch, while the subscriber also depended on the copper loop, the handset and the weather. The paper that gives this problem its clearest modern treatment, “The Calculus of Service Availability” (Ben Treynor and co-authors, 2017), states the hard limit plainly: a service can be no more available than its least available critical dependency.

The arithmetic behind that limit depends on how the dependencies fail. If they fail independently, so that one being down tells us nothing about whether another is down, then the chance that all of them are up at a given moment is the product of their separate availabilities. Four dependencies at 99.95% each give 0.9995 × 0.9995 × 0.9995 × 0.9995, which is about 99.80%. That is already below checkout’s 99.9% objective, before Orders has written a line of its own code or made a single mistake of its own. To feel what the gap means, translate it into time: over Encore’s 28-day window, 99.9% allows about 40 minutes of failure, while 99.80% means about 80 minutes. The dependencies alone would spend twice the budget.

If, on the other hand, the dependencies’ failures coincide, because they share a database, a network or a region, then their bad minutes overlap. When the shared database goes down, all four fail during the same minutes, so the journey loses those minutes once rather than four times. The composite is then better than the product, but it can never be better than the weakest link, here 99.95%, because the journey still fails whenever that one dependency fails. These two cases bracket the truth. For services that are run separately, on separate infrastructure, by separate teams, the independent case is the closer one, and so the product is the honest guide. Figure 1.8 draws the arithmetic.

Figure 1.8 — The availability of a chain of hard dependencies whose failures are independent. At 99.95% each, the chain reaches checkout’s objective at the second dependency and falls below it at the third; at 99.99% each, three still leave room for checkout’s own failures.

Concert sound has exactly this shape. A sound engineer knows that the finest mixing desk cannot fix a crackling cable: the audience hears the microphone, the cable, the desk, the amplifier and the speakers in series, and the chain sounds only as clean as its worst link. That is exactly the multiplication in Figure 1.8. But here the analogy breaks, and it breaks in our favor. A sound system has no way to carry on when a cable fails short of someone walking out on stage with a spare. A service, unlike a sound system, can often keep its promise when a dependency fails, provided it was designed to. That design choice is what separates two kinds of dependency. A hard dependency is one whose failure fails the journey. A soft dependency is one whose failure the journey survives, perhaps in a degraded form that users can still accept.

With that distinction in hand, we can look at the two remedies the paper offers, both of which Encore uses. The first is a rule of thumb: critical dependencies should be roughly one nine more reliable than the service that depends on them. Encore holds its internal purchase-path services to 99.99%, and three of them multiplied together come to about 99.97%. That leaves room inside a 99.9% promise for checkout’s own failures, about 30 of the 40 minutes in a 28-day window. But the paper also warns against applying the rule mechanically, demanding an extra nine at every layer down the stack, because each nine costs far more than the one before it, as Section 1.1 showed, and the layers at the bottom would be asked for reliability nobody can afford. Where a dependency cannot supply the extra nine, the second remedy applies: make the dependency soft.

The payment service provider is Encore’s clearest case. The PSP’s contract promises 99.9% per calendar month, and Encore cannot buy another nine from it. If checkout depended on a single PSP as a hard dependency, the arithmetic would be fatal: the three internal services at about 99.97%, multiplied by the PSP’s 99.9%, give about 99.87%, below checkout’s objective before checkout has failed on its own account. So Orders & Payments routes each new payment to whichever of two PSPs is healthy, as large merchants do. This is real protection, though not perfect independence, because both PSPs lean on the same card networks, and a failure there would reach both at once. For the moments when both PSPs are degraded, Encore has a further step. Verified fans’ seats are held for up to 20 minutes until they can pay, the fan completes payment with strong customer authentication once a PSP recovers, and holds that remain unpaid lapse and return their seats to sale. No ticket is ever issued without an authorized payment. The design turns what would have been an error page into an honest delay, and that is what makes the PSP soft. Figure 1.9 shows the result for every dependency of checkout.

Figure 1.9 — Composing checkout. Hard dependencies multiply and need an extra nine; soft ones are designed out of the product, and the PSP row is why the checkout SLO is achievable at all.

Read the table from the consumer’s side and something important appears. Orders & Payments could make the PSP soft only because it knew what the PSP promised, and the Gate’s consumers can plan around it only because they know it is held to 99.99%. A published SLO thereby becomes part of a service’s interface. Encore’s service registry shows each service’s SLOs beside its API contracts, so that a team about to take on a new dependency learns how much to trust it in the same place it learns how to call it, much as a type signature tells a caller what to pass. This is information hiding, in the sense the Architecture book took from Parnas, extended from behavior to reliability. A consumer should depend on the promise and nothing more. A service that has never failed in the year since we started calling it has made us no promise at all; its good record is only a coincidence that has lasted so far, and a team that builds on the record instead of the promise will learn the difference on the day the coincidence ends.

Tenants, contracts and small numbers

Dependencies are one way an SLO is shared across teams. Customers are the other, because Encore does not make one promise to everyone. It serves venues in three tiers. About 500 self-service venues are pooled in serverless cells of about 50. Professional tenants run in Kubernetes cells with a database each. And one Enterprise tenant, venue-chain-a, runs in a silo of its own. Self-service venues are promised 99.9%, and Professional tenants 99.95%. For venue-chain-a the promise is of a different kind: a service level agreement (SLA), a contractual promise with consequences, in this case 99.95% per calendar month, with service credits owed when Encore falls below it.

An SLA and an SLO serve different purposes, and the relation between them matters. The SLA is what Encore owes the customer, and missing it costs money. The SLO is what Encore asks of itself, and missing it costs only a change in how the teams spend their time. So the SLO is set tighter than the SLA, and the gap between them is where engineers notice trouble and fix it before anyone writes an invoice. For venue-chain-a, the internal SLO is 99.97% over a rolling 28 days. The gap is thin. In time, 99.97% over 28 days allows about 12 minutes of failure, while 99.95% over a 30-day month allows about 22, so the warning arrives with perhaps ten minutes of margin to spare. The windows also differ, and that difference can hide a breach: a rolling 28-day window and a calendar month cover different stretches of days, so a bad incident early in one month can drop out of the rolling window while still counting fully against the month the contract measures. For that reason Encore tracks the SLA on its own calendar month as well, beside the SLO, and does not assume one tells it about the other.

Small tenants raise a different problem, one of statistics. Consider a venue selling 300 tickets a month under a per-venue 99.9% objective. Its error budget is 0.1% of 300, or 0.3 failed purchases, so a single unlucky fan whose checkout fails would breach the objective for the whole month. Such an SLO measures chance. Hidalgo’s book gives a whole chapter to the probability involved, and its lesson is that an SLO needs enough events to tell a real signal from noise. Encore’s answer is to measure self-service SLOs per cell, because the cell is the unit that fails together: its 50 or so venues share the cell’s pooled serverless infrastructure, and a fault there reaches all of them at once. Measured that way, the cell has enough traffic for its ratio to mean something.

Aggregation, though, has its own blind spot. If one small venue in a cell is entirely down while its busier neighbors are serving fans normally, the cell’s ratio may barely move, because the dead venue contributes only a sliver of the cell’s traffic. The cell’s SLO would report health while every fan of that venue was failing. So Encore adds a synthetic probe: a scripted visitor that loads each venue’s storefront and seat map every minute from outside Encore’s network, as a fan would. It needs no volume of real traffic to be meaningful, because it asks a direct question of each venue, and it pages when a venue fails several probes in a row. The cell’s SLO answers how well the cell is serving its fans as a whole, and the probe answers whether any one venue has gone dark. Encore needs both questions answered.

The tiers, finally, share more than they seem to. Every tenant’s checkout, whatever tier it belongs to, passes through the same Seat Inventory. That core is therefore a hard dependency of every promise Encore makes, and by the arithmetic of the first half of this section it must be at least as reliable as the strongest promise it supports. Cells, Chapter 5’s subject, limit the blast radius of a failure, so that a fault in one cell harms only the tenants inside it. What they cannot do is lift any tenant’s reliability above the reliability of the shared core beneath them all.

Recap. A journey can be no more available than its weakest critical dependency, and when dependencies fail independently their availabilities multiply, so each critical dependency needs roughly an extra nine or must be designed to be soft. A published SLO belongs in a service’s interface, because consumers should build on the promise and not on a record of good luck. An SLA should sit outside its SLO with room to spare and be tracked on its own window, low-volume objectives need enough events to mean anything, and synthetic probes catch what aggregation hides.

Exercise 1.5. Draw the dependency chain for your most important journey, mark each dependency hard or soft with its published objective (or “none”), and multiply the hard ones. Does the product leave room for your SLO? Which of the dependencies share infrastructure, so that their failures would coincide? And which one would you make soft first?

Kata: The Eight O’Clock Slots

Your brief: “Kindred Health.” Kindred is a telehealth platform that runs video consultations between patients and about 2,400 doctors in three EU countries, some 6 million consultations a year. Bookings arrive by two routes. Patients book in Kindred’s app, and 85 insurer and clinic partners book on their behalf through an API, under a contractual service level agreement (SLA) that promises them 99.9% availability each month. A single consultation leans on several systems at once: Kindred’s own booking service; an external video relay that carries the call; a national electronic identity (eID) login, which patients in two of the three countries must pass before they see a doctor; and an insurer eligibility check that confirms the visit is covered. Payments go through an external payment service provider (PSP). The busiest moment of every weekday is fixed and known in advance. At 08:00 the day’s same-day slots open, and in the flu season traffic rises 30× for about 40 minutes as patients who woke up ill compete for the morning’s appointments. Kindred has 35 engineers in six teams, and their alerting is not their own: they inherited it from a vendor template that puts a 99.99% alert on every service, whatever that service does for patients. On-call took 58 pages last month. Meanwhile the clinical lead and product disagree about whether to freeze changes during the flu season, and nothing written down helps them settle it.

Deliverables:

1. Journey SLOs — Write service level objectives for four journeys: a patient booking a consultation, joining it, the video staying usable once the call has begun, and a partner booking through the API. For each, give the service level indicator (SLI) as good events divided by valid events, the point where you measure it (as close to the patient as you can get), the target and the window. Then say where clinical safety justifies a stricter objective than patients would notice on their own. A dropped call with a doctor midway through a diagnosis is a different kind of failure from a slow seat map, and the objective should show that you know the difference.
2. The eight o’clock SLO — Write an event-window objective for the morning slot opening, in the manner of Encore’s on-sale-window SLO in Section 1.2. Count it per patient, so that a patient who retries a failing booking ten times counts as one bad experience and not ten. State its statistical limits honestly: a window of about 40 minutes, crowded at the front, is noisy, and a burn-rate rule cannot protect it in time (Section 1.4). Then say how people will watch it live during the flu season, and what page stays armed behind them for the moment their attention drifts.
3. Error budget policy — On one page, write the agreement that governs the budget. Give the thresholds of remaining budget and what happens at each; the traffic that sits outside the SLO and why; how a burn caused by one team’s change, or by an external dependency such as the video relay, is attributed; and who signs the policy. That last point matters most here, because the signatories are the people who must stop arguing about the flu-season freeze, and the policy is the place where they settle it in advance.
4. Burn-rate alert set — Write multi-window, multi-burn-rate rules for the joining-a-consultation SLO, using the arithmetic of Figure 1.6. Then work out the detection time for two failures: a total outage, and a partial one in which 3% of attempts to join fail. Those two numbers are the real specification of your alerts, so ask whether a patient waiting in a virtual waiting room would accept them.
5. Dependency composition — Draw the chain a consultation depends on. Mark each dependency hard (the consultation fails when it fails) or soft (the consultation can carry on without it), multiply the availabilities of the hard ones as in Section 1.5, and note which dependencies share infrastructure, since their failures will not be independent. Finally, choose one hard dependency and redesign the journey so it becomes soft, saying what the patient experiences when it fails.
6. Partner SLA and internal SLO — Set the internal objective that sits behind the 99.9% partner SLA, and explain the margin between the two. Remember that the SLA counts calendar months and carries consequences, while the SLO is where engineers notice trouble and fix it before any partner has a claim.

Rubric:

Where you now stand

You can now choose indicators from what users actually experience, and decide which traffic belongs in them. You can give scheduled peaks, an on-sale or a morning rush, a window of their own instead of letting them dissolve into a quiet month. You can write an error budget policy that teams with different interests are willing to live with, alert on how fast that budget is burning, and reason about reliability across the dependencies and contracts no service escapes. These are real powers, but they have two limits. An SLO can tell us that users are suffering, and how much, but not why; finding out is the craft of observability, which the Operations book treats in its Chapter 4. And an objective is only a promise, which has to be paid for in machines. The 99% that Encore now owes its fans in the first hour of a major on-sale says nothing about how many fans the Gate may admit each second, how close to its limit each Seat Inventory shard may run, or what the spare capacity costs through the quiet weeks between sales. Those are questions of queues, load and money, and telephone engineers were the first to ask them. Chapter 2 takes them up.

References

• Betsy Beyer, Niall Richard Murphy, David K. Rensin, Kent Kawahara, Stephen Thorne (eds.) — The Site Reliability Workbook. O’Reilly/Google, 2018 — free online.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering. O’Reilly/Google, 2016 — free online.
• Alex Hidalgo — Implementing Service Level Objectives. O’Reilly, 2020.
• Ben Treynor, Mike Dahlin, Vivek Rau, Betsy Beyer — The Calculus of Service Availability. ACM Queue, 2017 — free online.
• DORA — Streamlining change approval. dora.dev — free online.
• Prometheus — Histograms and summaries. Prometheus documentation — free online.
• Sloth — Sloth: Prometheus SLO generator. sloth.dev — free online.
• Cloudflare — Challenge solve rate. Cloudflare documentation — free online.
• Amazon Web Services — Testing and tuning your AWS WAF protections. AWS WAF Developer Guide — free online.

Chapter 2: Performance, Capacity, and Cost

Every queue is a wager that the future will be less busy than the present.

In 1908 a young Danish mathematician named Agner Krarup Erlang joined the Copenhagen Telephone Company. The company put to him a question that sounds like bookkeeping and turned out to be the beginning of a science: how many lines, and how many operators, does a city need? Build too few, and callers hear a busy tone at the moments they most want to talk. Build too many, and copper and wages sit idle through every quiet afternoon, paid for and unused. Nobody could answer the question by counting subscribers, because subscribers do not call on a schedule. Erlang’s first insight, published in 1909, was that this randomness has a shape. Calls from many independent subscribers, each deciding on their own when to pick up the receiver, arrive as a Poisson process: random in the timing of any single call, yet describable in aggregate by one number, the average rate. Once arrivals could be described, they could be reasoned about, and by 1917 Erlang had derived two formulas from them. The first gave the chance that a call would be refused because every line was busy. The second gave how long a call would wait if the exchange held it in line instead of refusing it.

The profession honored him by turning his name into a unit. One erlang is the amount of traffic that keeps one server continuously busy. A system’s offered load in erlangs is its arrival rate multiplied by the mean time each arrival holds a server. A small worked figure makes the idea concrete: a switchboard that receives 600 calls an hour, each lasting three minutes on average, receives ten calls a minute, each holding a line for three minutes, so at any moment about 30 calls are in progress, an offered load of 30 erlangs. It is tempting to conclude that 30 lines will do. They will not. Thirty lines would be enough only if the calls arrived in perfect succession, and random arrivals bunch together, so some moments bring 35 calls and others 25. A switchboard offered 30 erlangs therefore needs more than 30 lines, and how many more depends on how long callers will tolerate waiting and how often we are willing to turn one away. Notice what has happened in that sentence. How long a caller waits is a question of performance. How many lines we provide is a question of capacity. What those extra lines cost through the quiet afternoons is a question of cost. Erlang’s work shows that these are one problem seen from three sides, and this chapter treats them that way.

The question Erlang was handed is the one Encore faces before every big on-sale, with fans in place of callers. When tickets for a popular tour go on sale, the fans gather first in the Sale Gate, Encore’s virtual waiting room, which admits them to the purchase path at about 1,200 fans per second per event, however many are waiting. Behind the Gate sits Seat Inventory, the ledger whose one essential job is to sell each seat exactly once. For a large event that ledger is split into four shards, and each shard is a small consensus group whose elected leader puts every seat write for that shard into a single agreed order. The Architecture book chose those numbers, 1,200 fans a second and four shards, but did not fully show where they come from. Why not admit 2,000 fans a second and sell out sooner? Why not run the shards closer to their limit and pay for less hardware? This chapter answers those questions with Erlang’s reasoning and its modern descendants, building on the service level objectives (SLOs) of Chapter 1, which say what fans must experience, and on the telemetry that lets us see what they actually do experience, which the Operations book treats at length in its chapter on observability. (The map of Encore in Chapter 1 shows where each piece sits.)

2.1 Performance Methodology — USE, RED, and Latency Distributions

Two checklists, two points of view

Most performance investigations fail because they begin with a hunch. Someone suspects the database, so they look at the database. There they find something mildly alarming, because a busy production system always has something mildly alarming in it, and they tune it. The graphs wiggle, the team feels productive, and the real cause sits untouched somewhere nobody thought to look. Brendan Gregg called this the streetlight method, after the old joke about the drunk who searches for his lost keys under the streetlight. He did not drop them there; it is simply where the light is. The pattern is so common because looking where we already have dashboards is easy, and looking everywhere else is work.

Gregg’s USE method, published in 2012, is his answer to the streetlight. Rather than starting from a suspicion, it starts from an inventory. We list every resource in the system and ask the same three questions of each one:

• its utilization: the fraction of time the resource was busy doing work;
• its saturation: how much work is queued up waiting for the resource because it was already busy;
• its errors: how often the resource failed at what it was asked to do.

A resource, in this sense, is anything that can run out. The obvious ones are physical: CPUs, memory, disks and network links. The ones that cause the most surprising outages are logical: connection pools, thread pools, file descriptors, rate-limit quotas, and, at Encore, the ordered write log of a Seat Inventory shard, which can accept only so many seat writes per second no matter how many CPUs stand idle beside it. The operating system will report CPU utilization for free. Nobody reports the saturation of a connection pool unless we add that instrumentation ourselves, and so the logical resources are usually the ones we cannot see when we need to.

The method is deliberately dull and deliberately exhaustive, and that is its virtue. Because it walks every resource whether or not we suspect it, it finds the problem we did not suspect. A theatre’s stage crew works the same way before a show: they check every fly line, every dimmer and every radio pack, and not only the one that failed last night, because last night’s failure is the one they are least likely to repeat. The analogy breaks in one useful place. A stage crew’s list is fixed for the run of the show, whereas a software system’s list of resources changes with every deploy. A new dependency brings a new pool, and the checklist is only as complete as our latest inventory of what can run out.

USE looks at the system from the resource’s point of view. Tom Wilkie’s RED method looks from the request’s side, and asks three questions of every service:

• its rate: how many requests per second it is serving;
• its errors: how many of those requests fail;
• its duration: the distribution of time each request takes, from arrival to response.

RED descends from the four golden signals of Google’s site reliability engineering (SRE) book, and it maps naturally onto the service level indicators (SLIs) of Chapter 1: a request rate, an error ratio and a latency threshold are exactly the raw material an SLI is built from. RED fits request-driven services best. Batch jobs and stream processors don’t serve requests in that sense, and they need other signals, such as lag: how far behind the newest input the processing has fallen.

The two methods are complementary because they answer different questions. RED tells us where users feel the problem: which service is slow or failing, and for whom. USE tells us which resource is causing it: the pool, the shard or the disk that has run out. Used together they give an investigation a direction, from the symptom inward to the cause. Picture an Encore engineer paged at 3 a.m. by a seat-map burn-rate alert. They begin with RED for the seat-map service, to confirm that duration has indeed grown and to see whether every request is slow or only some. Then they walk USE down the resources behind it: the service’s own pods, its cache, its connection pools, and the Seat Inventory shards it reads from. At no point do they have to guess. The method tells them where to look next, which is a great kindness at that hour.

Latency is a distribution

Of RED’s three signals, duration is the one most often mismeasured, and the mistake usually begins with the habit of summarizing it as a single number. A service’s latency is not a number. It is a distribution: the whole spread of times that individual requests took. That spread is usually skewed, with most requests bunched near the fast end and a long tail stretching toward the slow end. It often has two humps. A seat-map request that finds its geometry in the cache returns quickly, a request that misses the cache must fetch and assemble it, and the two kinds of request form two separate clusters with a valley between them.

To describe such a shape we use percentiles. The percentile pN is the latency value below which N% of requests completed. If the p99 of the seat-map service is 2 s, then 99% of requests finished in under 2 s and the slowest 1% took longer. The p50, or median, describes the typical request; the p99 and p99.9 describe the requests at the tail.

The mean, the number most dashboards show first, is the worst summary of a two-humped distribution. A seat-map service whose mean is 120 ms can have a p99 of 2 s. The mean is pulled up by the slow cache misses and down by the fast hits, and it settles in the valley between them, at a latency that almost no actual request experienced. The fan who waited 2 s for the seat map does not experience the mean. They experience 2 s, and at Encore’s scale the slowest 1% of seat-map requests is still a great many fans (Figure 2.1).

Figure 2.1 — A seat-map service’s latency, on a log scale. The mean falls in the valley between the cache hits and the misses, where almost no request lands; the p99 lives in the misses’ long tail.

Two working habits follow from treating latency as a distribution.

The first is that percentiles do not aggregate. The p99 of a service is not the average of its pods’ p99s. A percentile is a position in a sorted list of latencies, and averaging two positions taken from two different lists does not give the corresponding position in the combined list. A quiet pod and a busy pod get equal weight in the average even though the busy one served far more requests, and a single pod with a terrible tail is diluted by its healthy neighbors. The correct approach is to record each pod’s latencies as a histogram, a set of buckets that counts how many requests fell into each latency range. Histograms can be merged honestly, because counts simply add: the service’s histogram is the sum of its pods’ histograms, and we read the service’s percentiles from that merged whole. The price is precision. We know only which bucket a request fell into, not its exact latency, so the bucket layout limits how accurately any percentile can be read. Exponential histograms, whose buckets widen in proportion to the values they cover, reduce that error across the wide range from fast cache hits to slow misses.

The second habit is that we read the tail in proportion to the fan-out. The more backends a single request depends on, the more the backends’ tails matter to it. Suppose a purchase calls 20 backends in parallel and must wait for all of them before it can answer. For any one backend, the chance of answering faster than its own p99 is 0.99. For all 20 to answer faster than their p99s, the chance is 0.99 multiplied by itself 20 times, which is about 0.82, or 82%. So in roughly 18% of purchases, nearly one in five, at least one backend lands in its slowest 1%, and the purchase waits for it. A latency that each backend team thinks of as rare, one request in a hundred, has become an ordinary experience for the user. Keep adding backends and the arithmetic keeps going: at about 70 backends, 0.99 to the seventieth power falls to about one half, which means that half of all requests now touch some backend’s p99, and a backend’s p99 has become the purchase’s median. “The Tail at Scale” gives the figure for a fan-out of 100: 63% of requests wait on at least one backend’s slowest 1%. The practical rule is that a service with wide fan-out must care about its dependencies’ p99 and p99.9 as much as a simple service cares about its own median.

There is a subtler trap, and it lies not in how we summarize latency but in how we produce the measurements in the first place. Gil Tene named it coordinated omission: a measurement error in which the load generator waits for each response before sending the next request, so that the requests that would have arrived during a stall are never sent and never measured. The load generator, in effect, coordinates with the system under test. When the system stops, the generator politely stops too, and the stall leaves almost no trace in the data.

A small worked example shows how large the error can be. Suppose a benchmark means to send 100 requests a second for 60 s, 6,000 requests in all, and the system normally answers in about 40 ms. Halfway through, the system freezes for 2 s. A closed-loop generator, one that sends the next request only after the previous one returns, sends a single request into the freeze, records one slow response of 2 s, and then carries on. Of the roughly 5,800 requests it records, only one is slow, so its p99 is still about 40 ms. The report says the system is fine. Now consider what real users would have seen. The 200 requests that were due during those 2 s would have arrived anyway, and they would have waited anywhere from 2 s down to almost nothing, depending on when in the freeze each one arrived. Those 200 slow requests are more than 3% of the 6,000, so the true p99 is no longer 40 ms; it lands at about 1.4 s. The closed-loop benchmark did not round the stall down. It erased it.

The difference is between two models of load. In a closed model, a fixed population of clients each waits for an answer before asking again, so a slow system automatically receives less work. In an open model, arrivals keep their own schedule regardless of how the system is doing. Bianca Schroeder and her colleagues showed in 2006 how far the two kinds of model diverge, and how misleading it is to study a system with the wrong one. Encore’s fans are an open population, as Erlang’s telephone callers were: when the on-sale opens at 10:00, fans click because the clock says 10:00, not because the previous fan’s request has finished. A closed-loop test of an open system measures a gentler world than the one the system will meet. The remedy is to give each request in a test an intended send time from a fixed schedule, and to time its latency from that intended moment rather than from when the generator finally got around to sending it. That way the time a request spends waiting for a stalled system counts as latency, which is exactly what it is.

Recap. Methods beat hunches because they cover the whole system whether or not we suspected the right part of it. RED describes what requests experience, and USE describes what resources endure; together they lead from the symptom to the cause. Latency is a distribution rather than a number: it is measured with merged histograms rather than averaged percentiles, read at the percentile that matches the fan-out, and taken from load that arrives on its own schedule, as real users do.

Exercise 2.1. For one service you own, list every resource the USE method should cover, including pools and logical limits such as quotas and write logs. For how many of them can you produce utilization, saturation and errors today? Then count how many backends your busiest request type calls and must wait for, and use the arithmetic above to work out what fraction of those requests touches at least one backend’s p99.

2.2 Queues, Load, and the Universal Scalability Law

The hockey stick

Erlang’s formulas gave the telephone company its answers, but the deepest lesson they hold lies in the shape they draw. When requests arrive at random and each takes a varying time to serve, a server can be idle part of the time and still make some requests wait, because arrivals do not politely space themselves out: two often turn up together, and the second must stand in line while the first is served. That waiting grows faster than the load does. To see how much faster, take the simplest model the theory offers: random arrivals, random service times and a single server. Write the server’s utilization, the fraction of time it is busy, as ρ (rho). The mean response time, which is the time a request spends waiting in the queue plus the time it spends being served, is then the service time divided by one minus the utilization. The arithmetic is worth doing slowly, because it surprises almost everyone the first time. At 50% utilization a request takes twice its service time, half of it spent waiting. At 80% it takes five times its service time, and four of those five parts are spent in the queue. At 90% it takes ten times; at 99%, a hundred. Plotted against utilization, the curve stays nearly flat for a long while and then turns sharply upward, like the blade of a hockey stick, and the bend where it turns is what engineers call the knee. Real systems differ from this model in nearly every detail, since their arrivals are not perfectly random and they have many servers, caches and pools rather than one, but they share its shape, and the shape is what we use.

In 1961 J. F. C. Kingman sharpened the point. For a single server under heavy load, he showed, the expected wait depends on two things multiplied together. One is the utilization term we have already met, ρ/(1−ρ), which explodes as ρ approaches one. The other is the variability of the arrivals and of the service times: the more irregular they are, the longer the wait at the same utilization. Two systems running equally busy can therefore have very different queues, and the one fed by lumpier traffic suffers more. This matters to Encore because an on-sale’s arrivals are about as bursty as traffic ever gets: a million fans watching the same clock tick toward 10:00 do not arrive at random, they arrive together. It helps to hold two quantities apart. Utilization is what the machine feels, a measure of how busy it is. Queueing delay is what the fan feels, a measure of how long she waits. On a quiet day the two move gently together, but near the knee they diverge sharply, and they diverge most at exactly the moment the business is busiest, which is when a fan’s wait matters most. Figure 2.2 draws both curves: the gentle model and the steeper one that burstier arrivals produce.

Figure 2.2 — Response time against utilization for one server with random arrivals, and the steeper curve that burstier arrivals draw. Encore’s write shards are planned at 80%; the same curve explains why a hot shard at 96% hurts so much more than 16 points should.

The caption’s last claim deserves its arithmetic. At 80% utilization the simple model gives five times the service time; at 96%, one minus the utilization is 0.04, so the response time is 25 times the service time. Sixteen more points of busyness have made each write wait five times longer. Section 2.3 shows how a shard comes to run that hot, and why Encore plans for it.

Every motorway at rush hour draws the same curve. Add a few more cars to a road that is already near its capacity and the traffic does not slow a little; it collapses into a jam that takes far longer to clear than it took to form. Many cities answer with ramp metering, a traffic light on the on-ramp that admits cars one at a time at the rate the motorway can absorb. Metering keeps the main road below its knee, where traffic flows, and the price is a wait on the ramp that every driver can see. The trade is a good one, because a short, visible, orderly wait for some is far better than a jam for everyone. The Sale Gate is Encore’s ramp meter, and its limit of about 1,200 fans per second per event is the timing of the green light. Metering does a second, quieter job as well, the one Kingman’s formula predicts: by admitting fans at a steady rate, the Gate strips much of the burstiness out of the arrivals, so the shards behind it see a smoother stream than the crowd in front of it. That is also why the Gate itself must not become the bottleneck. Its edge runs as serverless functions, pre-warmed before each scheduled on-sale and able to grow to 400 instances, because a waiting room that collapses under its own crowd is worse than having no waiting room at all.

Like every analogy, this one breaks somewhere, and here it breaks in a useful place. A driver held at the ramp light waits in the car; she cannot vanish and reappear a minute later as a second car. A fan who is refused can do exactly that, and does: she presses the button again, and her retry arrives as new load on top of the load that caused the refusal. Chapter 5 takes up that problem. Erlang’s waiting formula, which describes many servers sharing one queue, adds one more lesson that bears on how we carve up capacity. At the same utilization a large pool of servers keeps requests waiting far less than a small one, because when one server is tied up with a long request the others are likely to be free, whereas in a small pool one slow job blocks a larger share of the whole. So splitting capacity into isolated cells, for all the containment it buys, has a price: each small cell needs its own headroom to keep its own queue short, and the cells together need more spare capacity than one large pool would.

How far will it scale?

Queueing theory describes a single resource under load. Scaling asks a different question: what happens when we add more workers, threads or connections to share the work? Intuition says that doubling the workers should double the throughput, and experience says it never quite does. Most teams answer the question empirically, running a load test at increasing concurrency until the system stops improving and reading the knee off the graph. That is honest work, but a graph of points tells us where the system stopped scaling, not why, and without the why we cannot tell whether a change of design or of configuration would help. Neil Gunther’s Universal Scalability Law (USL) is a small model for interpreting those points. It is used mostly by capacity planners in his tradition and in database performance practice, where Baron Schwartz and the MySQL community applied it widely. The law describes the relative capacity C(N) of a system at concurrency N, where C(1) is the throughput of a single worker, and it charges two separate penalties against the ideal of perfectly linear growth:

C(N) = N / (1 + σ(N−1) + κN(N−1))

Each penalty corresponds to a different kind of cost, and keeping them apart is the whole value of the model. The first is the contention coefficient σ (sigma), the fraction of the work that must be serialized: done one worker at a time, while the others wait for a lock, a single log or a shared queue. If we set the second coefficient to zero, the law reduces to the one Gene Amdahl stated in 1967, and throughput approaches a ceiling of 1/σ times a single worker, no matter how many workers we add. Contention, in other words, bends the curve toward a flat line but never turns it down. The second penalty is the coherency coefficient κ (kappa), the cost of keeping parallel workers consistent with one another, as when they exchange messages, invalidate each other’s caches or retry after colliding. That cost grows with the number of pairs of workers, which is why the term contains N(N−1): every new worker must, in some measure, coordinate with every worker already there. Because pairs multiply faster than workers, coherency eventually outruns the benefit of adding workers, and throughput peaks and then actually falls once N passes √((1−σ)/κ). Gunther calls this retrograde scaling, and it is the most counterintuitive and most practically important thing the law says: past a certain point, adding concurrency makes the system slower in total, not merely slower to improve.

An orchestra offers a fair picture of the two penalties. Contention is the single conductor: every player watches the same baton, and no number of extra violinists lets the piece go faster than one person can beat time. Coherency is the listening: each player adjusts to the others, and in a large ensemble the effort of staying together grows faster than the number of players. The analogy breaks, as they do, because an orchestra adds players for richer sound, not for throughput, but it captures why the two costs differ in kind.

Seat Inventory has both. Each shard’s leader appends every seat write to a single ordered log, so writes take their turn at one point, and that is contention. When several fans try to hold the same popular seats at the same moment, their holds collide and the losers retry, and the work of resolving those collisions grows as more writers compete, which is coherency. To learn how large each cost is, the Inventory team ran a load test against a single shard at six concurrency levels and fitted the USL to the measured throughput:

Figure 2.3 — The USL fitted to one Seat Inventory shard. Peak throughput falls near 44 concurrent writers; beyond that, more concurrency buys less.

Read the fitted coefficients first. A σ of 0.03 means about 3% of each write’s work is serialized; on its own, that would cap the shard at 1/0.03, about 33 times a single writer, or roughly 4,000 writes/s. The κ of 0.0005 looks tiny, but it is multiplied by the number of pairs, and at 64 writers there are thousands of them: the coherency term adds about 2.0 to the denominator at that concurrency, more than the contention term’s 1.9. That is why the measured throughput at 64 writers is lower than at 32. The fitted curve, drawn in Figure 2.4, then tells the team three things.

First, the shard peaks at roughly 1,630 writes/s near 44 concurrent writers, the point where √((1−0.03)/0.0005) comes to about 44. The published figure of 1.5k writes/s per shard is therefore a measured ceiling with a little margin below the peak, not a guess. Second, raising the concurrency cap above its current 32 would gain almost nothing, and pushing it far past 44 would lose throughput, so the cap is a setting whose change belongs in configuration review rather than in an engineer’s afternoon of tuning. Third, if Encore ever needs more write capacity per event, it must come from more shards per event, a change of design that no tuning knob can stand in for. This last conclusion is the one I value most, because it turns a vague hope that the system can be tuned a little further into a clear statement of where the limit lies and what it would take to move it.

The model earns that trust only under stated conditions, and they are worth naming because each one is easy to violate. The fit needs a measured point at a concurrency of one, since C(N) is defined relative to a single worker and every coefficient is scaled from it. It needs a steady workload with a fixed mix of operations, since a test that blends holds and reads differently at each level is measuring a different system at each point. And it must not be extrapolated far beyond the largest N actually measured, because the coefficients describe the range we observed and nothing guarantees that a new kind of cost does not appear beyond it. The USL predicts throughput, not latency, so it says nothing directly about the 3 s checkout budget. And it must be refitted whenever the code changes, because a new lock or a new retry path changes σ and κ, and yesterday’s coefficients then describe a system that no longer exists.

Figure 2.4 — The USL fit of Figure 2.3 against linear scaling and Amdahl’s law. Contention alone would bend the curve; coherency turns it over, and the concurrency cap of 32 sits just below the peak.

Recap. Response time rises with utilization along a hockey stick, and variability in the arrivals steepens it, so the fan’s wait grows far faster than the machine’s busyness. Admission control, like a ramp meter, keeps a system below its knee and smooths its arrivals, at the cost of a visible queue. Load tests find where scaling stops, and the USL, where its conditions hold, explains why by separating the cost of waiting in turn from the cost of staying in step, which tells us whether the remedy is a setting or a redesign.

Exercise 2.2. Load test one component at four to six concurrency levels, including a concurrency of one, and plot throughput against concurrency. Fit the USL to the points (a spreadsheet’s solver will do). Which penalty dominates, contention or coherency? Is your current concurrency setting on the rising or the falling side of the peak?

2.3 Capacity Planning and Load Testing

On-sale arithmetic

Erlang’s question was how many lines a city needs, and ours is how many fans Encore can let through the Gate each second. Capacity planning begins with the same first step he took: choosing what to count. The unit of capacity is the measurable quantity that actually limits the business workload, the thing that runs out first when demand rises. It is rarely processor time. Arun Kejariwal and John Allspaw insist that we find that unit’s ceiling by measurement, because a ceiling we have only estimated is one we have not yet met. For Encore during an on-sale, the unit is ordered seat writes per shard. Reads can be spread across replicas and caches, but every write to a shard passes through one elected leader that puts it in order, and Section 2.2 showed that such serialized work stops scaling. The unit also matters to the business directly: a delayed seat write is a fan who cannot buy.

To plan in this unit, we need to know how much of it each fan consumes. A fan admitted by the Gate spends about three minutes choosing seats and checking out, comfortably inside the 5-minute hold. In that time the fan generates about four ordered writes: a first hold, often a second hold after losing a seat to someone faster, the sale itself, and, for the holds that fans abandon, a release or an expiry. The same fan generates about 26 availability reads, as the seat map refreshes and the fan looks around. If the Gate admits fans at a steady rate, the shards see that rate multiplied by each fan’s demand, and the arithmetic is short enough to fit in a table:

Figure 2.5 — Encore’s on-sale capacity arithmetic. Writes bind first, and the admission rate is set so that they sit at 80% of the shards’ rated capacity.

Read the table one row at a time. Each of the four shards is rated at 1,500 ordered writes a second, so together they can absorb 6,000; 1,200 fans a second, each making four writes, ask for 4,800, which is 80% of that. Reads come in at 78% of their capacity, close behind but still lower, which is why we say the writes bind: if we raised the admission rate, the write ceiling is the one we would hit first.

The last row applies Erlang’s unit to today’s fans. Recall that offered load in erlangs is the arrival rate multiplied by the time each arrival holds a server. Here 1,200 fans arrive each second and each holds a session for about 180 s, so the offered load is 216,000 erlangs of fans. Little’s law says that the average number of items inside any stable system equals their arrival rate multiplied by the time each one spends there, so 216,000 is also the number of fans we should expect to find in seat selection or checkout at any moment. This number does not limit Encore directly, but it tells every service that keeps per-session state how many sessions it must hold at once.

Now return to the 80% and ask what it costs a fan. Section 2.2’s hockey stick says that at 80% utilization the mean response time of a seat write is about five times its service time, one part service and four parts waiting. Five times a few milliseconds is still a few milliseconds, and the 3 s checkout budget absorbs that easily. No standard prescribes 80%. It is a rule of thumb, and it is defensible here for a specific reason: the wait it produces fits the budget. The remaining 20% of write capacity looks idle on a cost report, and anyone reading only that report could fairly ask why Encore pays for it. The answer is that the 20% has two jobs.

The first job is absorbing skew. Our arithmetic assumed that each shard receives a quarter of the writes, but fans do not choose seats uniformly. The front-floor sections sell first and hottest, and the seats in them may live mostly on one shard. If that busiest shard takes 30% of the writes instead of 25%, it receives 1,440 writes a second against its 1,500 ceiling and runs at 96%. This is the hot shard that Figure 2.2 anticipated: as Section 2.2 worked out, 96% on the hockey stick means a response time of about 25 times the service time, five times worse than its neighbors, although its utilization is only 16 points higher. The average across the four shards still reads 80%, which is why a dashboard that shows only the average will hide exactly the shard that fans are waiting on.

The second job is absorbing failure. When the node holding a shard’s leader is drained for maintenance or dies, the shard’s consensus group elects a new leader in the same region. The fencing that keeps a seat from being sold twice works by rejecting writes from any leader whose term has ended, and during the handover it rejects all writes on that shard, for about 40 s. The fans whose holds failed do not go away; they try again. So a shard carrying its even share of 1,200 writes a second comes back from those 40 s owing about 48,000 writes (1,200 multiplied by 40), on top of the new writes that keep arriving. The new leader can work off that backlog only with its spare capacity. With 300 writes a second to spare, it drains 48,000 in 160 s, under three minutes, and it runs at its ceiling the whole time, so every fan on that shard waits at the far end of the hockey stick until the queue is gone. The hot shard is in a much worse position. It owes more, about 57,600 writes from its 1,440 a second, and it has only 60 a second to spare, so it would need about 16 minutes, more than a quarter of an hour, which on an on-sale morning is most of the sale.

This is old wisdom with several names. Google’s SRE book plans for N+2: enough units that the service survives losing one to maintenance and another to failure at the same time. Amazon’s Well-Architected framework calls the idea static stability, meaning a system that already holds the capacity it will need when part of it fails, rather than one that must go out and acquire capacity in the middle of the failure. Encore’s version is specific to its unit: every shard’s headroom must be able to absorb the backlog of its own failover, quickly enough that the fans on that shard are not waiting through the sale. Seen this way, headroom is the price of a short queue, paid in advance.

Load tests that resemble Friday

The arithmetic is only as good as its per-fan numbers, and those come from load tests. Encore’s harness, which the On-Sale team calls Soundcheck, replays recorded on-sales, scrubbed of personal data, against a production-sized copy of the purchase path. Replaying a real on-sale matters because it carries the real mix of browsing, holding, losing seats and abandoning carts, which no hand-written script reproduces faithfully. Four further choices make Soundcheck’s results trustworthy, and each answers a lesson from earlier in this chapter.

First, Soundcheck generates load from an open model, in which each simulated fan arrives on a schedule of its own, whether or not earlier requests have been answered; the harness uses k6’s arrival-rate executors to do this. Section 2.1 explained why: a closed-loop generator waits for each response before sending the next request, so during a stall it simply stops sending, and the stall never appears in its measurements. An open-model generator can still fall into the same trap if it runs short of resources and quietly skips arrivals it was scheduled to send. Soundcheck therefore fails any run in which the generator dropped scheduled arrivals, since dropped arrivals mean coordinated omission has reappeared inside the rig.

Second, its arrivals are bursty, because Kingman’s variability term is real: at the same average rate, bursty arrivals produce longer queues than smooth ones, and a test with smooth arrivals would flatter the system. Third, its seat choices are skewed toward the best sections, because uniform random seats spread writes evenly across the shards and would understate the hot shard we met above. Fourth, it stubs the payment service provider (PSP), the outside company that processes card payments, with a latency distribution taken from production rather than a fixed fast response. A stub that always answers in a few milliseconds would hide what happens when the PSP slows down, and the first outage in the Architecture book began with exactly that, a PSP that slowed down.

Even a faithful copy is not production. The copy may have the same number of machines, but it does not share production’s network paths, its noisy neighbors, its cloud account limits or its real dependencies, and some problems appear only there. Other companies with sharp, predictable peaks have reached the same conclusion: Shopify has written about running its Black Friday scale tests against production infrastructure, and Hotstar’s engineers about load-testing at the full scale of a cricket audience. Encore does the same twice a year. It runs Soundcheck at off-peak hours against the hidden test event that Encore’s synthetic buyers already use, so that the test exercises the real purchase path without selling any real seat. One cost-saving detail follows from the purpose of the test: the load generators may run on cheap spot capacity, which the cloud provider can reclaim at short notice, because losing a generator only ends a run early. The system under test does not run on spot, because its behavior is the thing being measured.

One lesson concerns state rather than traffic. When a team runs a load test twice, the second run meets a system that the first run has already warmed: caches are full, connection pools are open, and code paths have been compiled and loaded. On-sale morning is often the first run. At Encore the clearest case is the seat-map geometry cache, which holds each venue’s layout (rows, sections, the shape of the hall) but never which seats are free. When that cache is cold, every request misses it and goes to Catalog, the service that owns venue data, and it does so at the worst possible moment, in the opening seconds of the sale when the most fans arrive at once.

A touring band understands this problem from the stage. The crew soundchecks in an empty arena in the afternoon, knowing the room will sound different at nine o’clock, when 15,000 bodies absorb the reflections that bounced off empty seats. Experienced engineers adjust for the difference, because they know which way it runs. The analogy breaks in a direction that should worry us: the hall grows kinder when it is full, while a cold cache makes the system harsher just when it is busiest. A load test against warm caches therefore errs in the dangerous direction, telling us the system is stronger than it will be at curtain-up. So every test plan at Encore must say which cache state it assumed, and on the day itself the cache is warmed before the doors open.

The Wednesday review

All of this arithmetic and testing would be wasted if it never met a decision, and at Encore it meets one each week before a big on-sale. The Reliability team holds an on-sale readiness review on the Wednesday before each big on-sale, which usually opens at 10:00 on a Friday. It is a launch review of the kind Google’s SRE book describes, covering capacity, dependencies and who will be on call. The review’s most important rule concerns the admission rate, since that single number drives every row of Figure 2.5. A proposal to raise it must arrive with the arithmetic redone at the new rate and a load test run at the new point, and the review asks what cache state that test assumed, because a test against a warm cache answers a different question from the one the on-sale will ask.

The review also checks the limits that belong to someone else, which are easy to forget because no Encore dashboard shows them. The cloud account caps how many serverless functions may run at once, and the Gate runs on those functions; Chapter 3 tells how Encore learned that one. The load balancers have their own limits, and each PSP has agreed to a request rate that Encore must stay under. The review requires each of these quotas to stand at least 1.3 times the forecast peak, so that a busier night than forecast still fits. Forecasts are sent in advance to any provider the night depends on, so that the provider is not surprised by Encore’s traffic, a practice Amazon formalizes as Infrastructure Event Management. The core of the checklist fits on a card:

On-sale readiness — Wednesday, T-2 days
[ ] Admission rate unchanged, or new arithmetic + load test attached
[ ] Load test: open model, bursty, skewed seats, cache state stated
[ ] Writes per shard ≤ 80% at planned rate; failover drain < 5 min
[ ] Pre-scale plan: Gate concurrency, Inventory floors, cache warm
[ ] Quotas: function concurrency, LB, PSP rates ≥ 1.3 × forecast peak

Each line on the card points back to something this section has argued. The first two lines guard the per-fan numbers and the honesty of the test that produced them. The third holds the 80% and asks that every shard, the hot one included, could drain a failover’s backlog in under five minutes. The fourth names the capacity that must be in place before the doors open, which Section 2.4 explains, and the fifth covers the limits other people set. I have come to believe that a review like this earns its keep less by catching errors on the day than by making each assumption visible enough that someone will ask about it when it changes.

Recap. We plan capacity in the unit that limits the business, which for Encore is ordered writes per shard, and we keep headroom for skew, for variability and for the failover that will eventually come, sizing it so that every shard can drain its own failover backlog quickly. Load tests use open, bursty arrivals, realistic seat skew and realistic dependency latency, state the cache state they assumed, and are confirmed periodically at production scale. Before a scheduled peak, the limits other people set are checked as carefully as our own.

Exercise 2.3. Write your system’s equivalent of Figure 2.5 for its busiest hour: name the unit of capacity, the demand each user generates, and the utilization that results. Then compute how long your binding resource would take to drain the backlog of its own failover at that utilization, and check whether your last load-test report says whether caches were warm or cold.

2.4 Autoscaling and Elasticity

The lag in the loop

Section 2.3 did its arithmetic in advance, on a Wednesday, for a Friday that everyone could see coming. A natural question follows: why plan at all, when the cloud will add machines as the load arrives? Autoscaling does make that promise, and for much of the year it keeps it. What it removes, in practice, is the need to plan for demand that changes slowly. Demand that changes quickly is another matter, and the reason lies in what an autoscaler actually is.

An autoscaler is a feedback loop, and like every feedback loop it acts on the past. It cannot see the load that is coming; it sees only a measurement of the load that has already arrived, and it reacts after a chain of steps, each of which takes time. First the metric must rise, and the monitoring system must scrape it. Then a stabilization window must pass, so that the autoscaler does not chase every momentary blip. Then a new pod must be scheduled, perhaps on a node that must first be bought from the cloud provider and booted. Then the container image must be pulled, the process must start, and the new instance must warm its caches and connections before it serves at full speed. Each step takes seconds. Together they often take minutes. Suppose, for illustration, that each of those six steps took a modest fifteen seconds: the loop would need a minute and a half to complete one cycle, and it would add capacity in proportion to the load it saw a minute and a half earlier. An on-sale that rises 500× in minutes is past its first surge before the loop has completed even that one cycle, and the fans who arrived during that minute and a half have already met a system sized for the quiet morning.

Pilots learn the same lesson from their engines. A turbofan at idle takes several seconds to spool up to full thrust, so a pilot who waits until the aircraft is low and slow on approach before asking for power will get it too late. That is why the stabilized approach is flown with the engines already spooled, ready to answer at once. Encore’s equivalent is pre-scaling: provisioning capacity ahead of a load we can predict, instead of waiting for the load to ask for it. Encore’s curtain-up workflow, the self-service routine that warms the purchase path on the morning of a sale, does exactly this before each scheduled on-sale. It raises the Gate’s provisioned concurrency toward its 400 instances, sets Seat Inventory’s minimum replica counts (its floors), and warms the seat-map cache, all before the doors open at 10:00. Hotstar’s engineers reached the same conclusion for cricket, writing that capacity must be in place ahead of the match. The analogy is worth following to the point where it breaks, which is the end of the flight. A pilot lands and shuts the engines down, and the flight is over. A service, after its peak, must also scale back down, and scaling down has hazards of its own: instances being removed must first drain their open connections without dropping requests in flight, and the crowd does not always leave on schedule. A late second wave of fans, for instance those returning as abandoned holds lapse and seats come back on sale, can arrive just as the capacity that would have served them is being handed back.

Reactive scaling still has its place, because not every surge is announced in advance. When it is used for unscheduled demand, it fails in three characteristic ways, and it helps to know them by name.

It can lag, as we have just seen: the loop reacts to the load of a minute or two ago.

It can thrash, scaling up on a burst, down in the lull that follows, and up again on the next burst, paying the full startup cost each time and never quite having the right amount of capacity. The usual remedies damp the loop. A stabilization window makes the autoscaler wait before acting on a change. An asymmetric policy scales up quickly and down slowly, on the reasoning that too much capacity for a few minutes costs a little money while too little costs failed requests. And a floor keeps a minimum number of instances running however quiet the moment looks. Kubernetes follows the asymmetric reasoning by default, scaling up at once and waiting five minutes before scaling down. Chapter 3 treats thrashing more fully, as a problem of control loops in the lineage of the governor.

And it can amplify, which is the most dangerous of the three, because it turns a small problem into a large one. In Encore’s first outage, told in the Architecture book, the PSP slowed from 200 ms to 8 s. Requests waiting on the PSP piled up, connection pools filled, and health checks began to fail. The autoscaler read this distress as a need for more capacity and added instances. But each new instance opened still more connections to the PSP, which was the very bottleneck causing the trouble, and so the autoscaler’s help made the bottleneck worse. The lesson generalizes beyond payments: adding instances to a service raises its capacity only if whatever it depends on can serve the extra work. The autoscaler can only buy what the dependency can sell.

Scaling on the right signal

If an autoscaler can be fooled into amplifying a failure, the defense begins with what it measures. We should choose the scaling signal as carefully as we chose an SLI in Chapter 1, by asking what the number really says about the work the service is doing.

The most common signal is CPU utilization, and it suits work that really is computation, such as rendering, encryption or compression, where a busier service burns more processor time. It misleads badly for services that spend their time waiting. A thread blocked on a slow dependency is not using the processor at all, so a service whose every thread is stuck waiting for the PSP looks idle on CPU while being saturated on everything else: its connection pool is full, its request queue is growing, and its callers are timing out. An autoscaler watching CPU would conclude that all is well, or even scale the service down.

Signals that count waiting work track the truth more directly. Concurrency in flight is the number of requests a service has accepted and not yet finished; queue depth is the number of requests or messages waiting to be started. Kubernetes’ Horizontal Pod Autoscaler (HPA) can scale on such custom metrics, KEDA scales on the depth of event sources such as queues and streams, and Knative scales on concurrency in flight. These signals carry their own hazard, though, and it follows directly from Little’s law, which we met in Section 2.3: the number of requests in flight equals the arrival rate multiplied by the time each request takes. When a dependency slows, each request takes longer, so concurrency rises even if not a single extra fan has arrived. A concurrency-driven autoscaler therefore sees a sick PSP as a surge in demand, and it will chase that PSP exactly as the old one did unless something stops it. A better signal makes the autoscaler more accurate in normal times, but it does not by itself make the autoscaler safe when a dependency fails.

That something is a ceiling. Kubernetes requires every HPA to declare a maximum number of replicas, and most teams pick that maximum by feel, a round number that seemed generous at the time. Encore derives its ceilings from the arithmetic of Figure 2.5 instead. The four shards of a large event can absorb a rated 6,000 ordered writes a second, so no service that sends work toward those shards is allowed to scale past the point where its replicas could generate more than the shards can absorb. However distressed the metrics look, the autoscaler cannot add the instance that would overload the shards. The Wednesday review rechecks each ceiling whenever either side of the calculation changes, whether that is a new admission rate, a different number of shards or a change in how many writes each fan generates. Connection pools and client-side concurrency limits do the same job one layer down, capping how much any single instance can ask of a dependency at once, so that the limit holds even inside an instance the autoscaler has already added.

There is a broader point here about the word “elastic”. A system is elastic only as far as its least elastic part. At Encore, the Gate’s serverless edge can grow to 400 instances and the stateless services can add pods within minutes, but the shard count of Seat Inventory does not grow during an on-sale at all; as Section 2.2 showed, more write capacity means more shards per event, which is a design change made in advance. Everything upstream of the shards can stretch only as far as the shards allow, and the ceilings simply make that fact explicit in configuration. I have come to believe that this is the right test of anyone’s claim about elasticity: anyone who calls a system elastic should be able to name its slowest-scaling component and say how long it takes to grow.

Recap. An autoscaler is a feedback loop that acts on the past, and it can lag behind a fast surge, thrash between too much and too little, or amplify a failing dependency by adding load to it. We pre-scale for the peaks we can foresee, scale reactively on signals that measure waiting work, and cap every service at what its dependencies can absorb, because any signal that rises when a dependency slows will otherwise drive the autoscaler to feed that dependency more work.

Exercise 2.4. For one autoscaled service, measure the time from a sudden step in load to a new instance serving at full speed, and compare it with how fast your steepest real peak rises. Then find the service’s maximum replica count: what downstream limit justifies that number, and what would happen downstream if the service reached it?

2.5 FinOps — Cost as an Operational Signal

The night’s settlement

After every concert on a tour, the tour manager and the local promoter sit down and settle the night. They set the ticket revenue against the cost of the hall, the crew and the local expenses, and they do it show by show, while the numbers are fresh and the people who spent the money are still in the building. The point of the ritual is that nobody discovers in December that the Tuesday shows lost money all year. FinOps, the practice of making cloud spend a shared, continuous responsibility of engineering and finance, is that settlement brought to software. The analogy is worth taking a step further, because it also shows where software differs. A promoter’s costs are mostly agreed before the doors open: the hall’s fee and the crew’s wages are in a contract. A cloud bill is written by the system’s own behavior, minute by minute, by every retry, log line and idle replica, and it arrives hours after the spending happened. So the settlement cannot wait for the end of the night; it has to run continuously, like any other operational signal. The Architecture book introduced FinOps’ central move, measuring cost per ticket sold. In this section we go further and treat cost as a telemetry stream with its own anomalies and its own choice of denominators, read alongside the capacity arithmetic of the sections before it. Erlang’s question at the opening of this chapter was, after all, partly a question about money: lines and operators sitting idle through a quiet afternoon were a cost, and so were callers hearing a busy tone.

J.R. Storment and Mike Fuller describe FinOps as a lifecycle of three phases: Inform, Optimize and Operate. Inform means making spend visible to the people who cause it; Optimize means reducing it; Operate means building those habits into the organization’s routine. The first phase is the hardest, and everything else depends on it, because nobody can optimize a cost they cannot see or do not own. Informing requires allocation, attributing each part of the bill to an owner, a service or a tenant. Most of that work is done through cloud accounts, tags and labels, so that a database instance carries the name of the team that runs it. The difficult case is shared infrastructure, above all the Kubernetes nodes on which many teams’ pods run side by side, where the provider bills for the node and says nothing about who used it. Tools such as OpenCost solve this by apportioning each node’s cost according to what each workload requested: a pod that requested a quarter of a node’s capacity is charged a quarter of its price. Encore allocates a little over 90% of its bill in this way, per service and per tenant, which is about as far as mature practice gets. The remainder, the genuinely shared costs that no single owner caused, is not spread thinly across everyone; it is shown as its own line, so that its size is visible and someone is accountable for it.

Once the bill has owners, we can divide it by the work it bought, and here the choice of denominator matters more than it first appears. The figure the Architecture book followed, which drifted from €0.11 to €0.19 per ticket, was the blended cost per ticket: the month’s whole bill divided by the month’s tickets. A blended figure is easy to compute and easy to explain, but it mixed two quite different causes into one number. Part of the rise was real inefficiency: a metrics forwarder, billed per invocation, that cost about €9,000 a month, and logging that was more verbose than anyone needed. Part of it was simply a larger Encore, which had added a second serving region and new tenants, with that larger fixed base spread over a quiet quarter in which fewer tickets were sold. The first cause is a defect to fix; the second is a consequence of growth and of the calendar, and the blended figure cannot tell them apart. So Encore now reports two figures instead of one. The first is the fixed monthly cost of being open for business at all, whatever the month’s sales. The second is the variable cost per ticket, the spend that rises and falls with the number of tickets sold, divided by that number. The split is approximate, and it is worth being honest about why. Much cloud spend is step-fixed: flat until demand crosses a threshold, and then a whole new node or a whole new shard at once. If Encore ever added a fifth shard for a season of large events, that shard would be not quite fixed, since it would exist because of ticket volume, and not quite variable, since it would not shrink when one week is quiet. Someone has to decide which side of the line each step belongs on, and write the decision down. After the forwarder fix, Encore’s variable cost per ticket stands at €0.12, broken down in Figure 2.6. In a typical month the fixed line adds about four cents per ticket, so the Architecture book’s blended figure would now read about €0.16.

Figure 2.6 — Encore’s variable cost per ticket after the forwarder fix. Tokens are one line among six; they get the same treatment as the others.

The table repays a slow reading. The largest line, purchase-path compute, carries a note that ties it directly to Section 2.4: the capacity that pre-scaling buys before an on-sale is billed from the moment it is warm, whether or not fans have arrived yet, and that is a cost Encore pays knowingly. Telemetry, at nearly a fifth of the total, is a reminder that the instruments that let us see a running system are not free, which is why their sampling policy has a review date. And the token line, the newest kind of spending, is small and sits in the same table as everything else, which is exactly where it belongs.

Splitting the two figures pays off because each one, when it moves, points to a different kind of cause. A rising variable cost per ticket is an engineering signal: some part of the design has become less efficient, and each ticket now costs more to sell than it did. A rising fixed cost while ticket volume stays flat is a planning signal: capacity was bought for demand that has not come, or has not come yet. The first kind of problem lives in code, and so it can be found the way performance problems are found. Continuous profiling, which Google described in 2010 as sampling the call stacks of every production machine all the time, makes this practical: because the profile is always running, a team can compare the profile from before a release with the profile after it and see which change made a request more expensive, in CPU seconds and therefore in euros. The second kind of problem is not found in any profile. It lives in forecasts and purchase decisions, and it is settled in planning meetings rather than in pull requests.

The token line is the youngest in the table, and it deserves a closer look precisely because it is new. It can be attributed at all only because every model call Encore makes passes through its AI gateway, which tags each call with the feature that made it, the tenant it served and the model it used; without that single chokepoint, token spend would be scattered across many teams’ invoices and nobody could say what the support assistant costs per conversation. Three levers move the token line, and they are not equally mature. The first is batch processing. The large model providers price their batch interfaces at about half the interactive rate, in exchange for returning results later rather than at once, and that suits work that can wait, such as the nightly summaries of the day’s support tickets. The second is prompt caching, also a priced feature offered by providers. Every turn of the support assistant begins with the same long block of instructions, and caching lets that unchanging prefix be resent cheaply instead of being paid for in full each time. The third lever is routing: sending most turns to a smaller model at about a tenth of the price, and reserving the larger model for the turns that need it. This lever is the least proven of the three. The best published account, RouteLLM (2024), reports large savings on benchmarks, but production accounts are few. Encore routes about 80% of turns to the cheaper model, and it does so only because each routing rule must hold its score on the 200-conversation golden set of Chapter 8 before it ships. The order matters: we pull the levers whose behavior we understand first, and we pull the young one only with a test that would tell us if it had started to cost us quality instead of money.

Commitments and anomalies

Storment and Fuller draw a distinction that keeps two conversations from blurring into one. Using less is engineering: fixing the forwarder, sampling telemetry, routing tokens. Paying less for what we do use is purchasing, and its main instrument is the pricing model. Cloud providers offer three broad kinds. Commitments, such as reserved instances and savings plans, trade a promise of steady spend for a lower rate. On-demand capacity promises nothing in either direction and costs the most per hour. Spot capacity is the cheapest of all, but the provider can reclaim it at short notice when it needs the machines back. A good portfolio follows the shape of demand. We commit to the floor that runs all year, because that capacity will be used whether or not the promise is made; we pay on demand for the scheduled peaks, because they are tall and brief; and we put interruptible work on spot, meaning work that can be stopped and restarted without harm, such as batch analytics and Soundcheck’s load generators. (The system under test never runs on spot, for the reason Section 2.3 gave.) Encore commits about 60% of its steady compute, which is less than the 70–90% many practitioners aim for, and it does so by choice. Its floor is still moving, as workloads shift between containers and serverless functions, and a commitment to capacity it will soon stop using would lock a forecast error into the bill. It revisits the figure each quarter, once the next on-sale calendar is known and the floor can be forecast with more confidence. The trade-off is a familiar one: a lower price is bought with less freedom to change the architecture, and a team in the middle of changing it should buy less of that kind of discount.

Commitments are about the price of the spending we expect; anomaly detection is about the spending we do not. Cost anomaly alerts are built like the burn-rate alerts of Chapter 1. A forecast says what the month’s spend should be, and the alert fires when actual spend runs ahead of that forecast fast enough to matter. The difference lies in the data. Billing data arrives between 8 and 24 hours late, so detection works on a scale of hours rather than minutes, and no cost alert can catch a problem in its first few minutes the way an SLO burn alert can. That delay also shapes how urgent the response should be. Most organizations route cost anomalies to tickets or chat, on the reasoning that money lost overnight is recoverable in a way that a failed on-sale is not. Encore’s choice is a little sharper. A spend rate that would exhaust the month’s budget within a week, which means spending at roughly four times the planned rate, sends an urgent notification to the owning team during business hours, though it does not page anyone at night. A sudden step in a team’s line opens a ticket. The reason for having any alert at all is that a cost regression, unlike a latency regression, makes no noise. No fan sees an error page and no dashboard turns red; the money simply accumulates until someone looks, as the forwarder’s €9,000 a month did for as long as nobody was looking.

There is also a judgment that no dashboard can make for us, and it brings this section back to the arithmetic of Section 2.3. The 20% of write headroom in Figure 2.5 appears on the bill as capacity that, on most days, does nothing. A well-meaning optimization pass, reading the cost report on its own, could propose to remove it and would see only savings. It would not know that this idle-looking capacity is what absorbs the backlog of writes after a shard failover, and that without enough of it a shard would take far longer to recover, as the hot shard’s quarter of an hour already showed. The cheapest configuration and the safest configuration are seldom the same, and the difference between them is not a matter of efficiency; it is a decision about how much risk to carry. I have come to believe that the most useful thing FinOps does is to make that decision visible, so that it is made deliberately by people who understand both sides of it, rather than by accident in a cost-cutting spreadsheet that has never heard of a failover.

Recap. FinOps begins by allocating the bill to owners, then splits fixed from variable cost, using judgment for the costs that come in steps, and watches the variable cost per ticket as an engineering signal. Commitments follow the shape of demand and are held back while the architecture is still moving. Token cost is reduced with the proven levers first and the young one only under evaluation. And cost is always read beside capacity, because headroom shows up on both, and only the capacity model explains why it is there.

Exercise 2.5. Split last month’s cloud bill into fixed and variable costs, and compute the variable cost per business unit, whatever your system’s equivalent of a ticket is. Where did you put the step-fixed costs, and would your finance partner agree with that choice? What fraction of your steady compute is covered by commitments, and who decided that fraction, from what forecast?

Kata: Filing Season

Your brief: “Ledgerline.” Ledgerline sells tax-preparation software as a service in one EU country. It serves 2.4 million filers a year, and they do not arrive evenly: 35% of them file in the final six days before the deadline, and on the last evening the load reaches 14× its normal level. Every filing runs through a calculation engine that takes about 400 ms of compute, and the engine runs again each time the filer edits a figure, so a hesitant filer costs several times what a decisive one does. Every one of those runs works against a shared PostgreSQL cluster with a single primary, which is the one place in the system where all the work converges.

Thirty engineers build and run all of this, and last season their preparation did not survive contact with the deadline. Their load tests were closed-loop, waiting for each response before sending the next request, and ran on a staging cluster a quarter of production’s size; they reported a p99 of 180 ms. On deadline night, support logged thousands of complaints from filers who waited 10 s. The autoscaler, which scaled on processor utilization, oscillated every few minutes and twice exhausted the database’s connection limit. Meanwhile the cloud bill rose 40% in a year in which filings rose only 12%, and a new AI feature that explains each calculation to the filer has added its own token line to the invoice, one that nobody forecast.

Each deliverable below applies one part of this chapter to Ledgerline, in roughly the order the chapter developed them, and together they should tell a single, consistent story about deadline night.

Deliverables:

1. USE and RED inventory — List every resource and service on the filing path, from the calculation engine’s instances to the database’s connection pool and its primary. For each resource give its utilization, saturation and error signals, and for each service its rate, errors and duration, then mark the signals Ledgerline cannot produce today. The missing ones are where last season’s surprises were hiding.
2. Capacity model — Choose the unit of capacity that actually limits deadline night, which may not be processor time. Estimate the demand one filing places on it, counting the re-runs that edits cause, and compute the utilization of the binding resource at the last evening’s peak. Then add headroom for skew and for a failover of the database primary, and show the arithmetic so that a colleague could check every step.
3. Scaling curve — Load-test the calculation engine against the database at five or more concurrency levels and plot the throughput. Identify the knee, the point beyond which adding concurrency stops helping, and, if the Universal Scalability Law’s conditions hold, fit it to the points to explain what is bending the curve.
4. Test and readiness plan — Design a load test from an open model, in which arrivals keep their own schedule, with bursts, skewed demand and a stated cache state, and say how its results will be confirmed at production scale. Add a review of the quotas and limits that other people set. Finally, explain why last year’s tests reported 180 ms while filers waited 10 s.
5. Autoscaling policy — Name the signal the autoscaler will scale on and say why it is better than processor utilization. Set floors and a pre-scaling schedule for the final six days, stabilization settings that stop the oscillation, and a maximum instance count justified by what the database can absorb.
6. Cost plan — Express cost per filing, split into its fixed and variable parts, so that the 40% and the 12% can be compared on one footing. Propose commitments, anomaly alerts, and the token levers you would pull first.

Rubric:

Where you now stand

You can now investigate performance by method rather than by hunch, and read latency as a distribution whose tail matters in proportion to how many backends each request fans out to. You can explain why waits explode as a resource nears saturation, plan capacity in the unit that truly limits the business, with room for skew and for the failover that will eventually come, and rehearse a peak with load that arrives on its own schedule, so that the rehearsal does not flatter you. You can keep an autoscaler from amplifying the very trouble it was meant to absorb, and you can read the invoice as one more stream of telemetry, always beside the capacity it pays for. The autoscaler was our first glimpse of a larger family: much of what we measured here is run, day to day, by loops that observe the system and correct it. Chapter 3 turns to those loops in full, the reconciliation engines of Kubernetes, serverless platforms and GitOps.

References

• Brendan Gregg — Systems Performance, 2nd ed.. Addison-Wesley, 2020.
• Arun Kejariwal, John Allspaw — The Art of Capacity Planning, 2nd ed.. O’Reilly, 2017.
• J.R. Storment, Mike Fuller — Cloud FinOps, 2nd ed.. O’Reilly, 2023.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering. O’Reilly/Google, 2016 — free online.
• Jeffrey Dean, Luiz André Barroso — The Tail at Scale. Communications of the ACM, 2013.
• Bianca Schroeder, Adam Wierman, Mor Harchol-Balter — Open Versus Closed: A Cautionary Tale. USENIX NSDI, 2006.
• Neil J. Gunther — Guerrilla Capacity Planning. Springer, 2007.
• Brendan Gregg — The USE Method. brendangregg.com — free online.
• Shopify Engineering — Performance Testing at Scale. Shopify — free online.
• Hotstar Engineering — Scaling Is Not An Accident. Hotstar — free online.
• Amazon Web Services — AWS Infrastructure Event Management. AWS — free online.
• FinOps Foundation — FinOps Framework: Unit Economics. FinOps Foundation — free online.
• Gang Ren, Eric Tune, Tipp Moseley, Yixin Shi, Silvius Rus, Robert Hundt — Google-Wide Profiling. IEEE Micro, 2010.
• Isaac Ong et al. — RouteLLM: Learning to Route LLMs with Preference Data. arXiv, 2024.
• Anthropic — Prompt caching. Anthropic documentation — free online.

Chapter 3: The Runtime and Its Control Plane — Kubernetes, Serverless, IaC, and GitOps

A controller is an opinion about the world, held continuously and acted upon without being asked.

We still speak of a runtime as the place where our software sits, as if it were a shelf. A modern runtime is better pictured as a crowd of small machines, each of which keeps noticing that something in the world differs from what was asked for and then sets about correcting it. A scheduler notices a pod with nowhere to run and places it on a node. An autoscaler notices that CPU is running hot and adds replicas. A GitOps agent notices that someone has hand-edited a setting in the cluster and puts it back the way the repository says it should be. An upgrade reconciler notices that a node is on an old version and, through the night, drains it and replaces it, one node after another. None of these machines waits to be told; each holds its opinion of how things ought to be and acts on it continuously.

That changes what operating a system means. A generation ago we operated servers: we logged in, we restarted processes, we resized disks. Today we mostly operate the things that operate servers, and our work is to understand, configure and sometimes restrain them. The question of how to supervise a machine that is itself correcting another machine sounds thoroughly modern, but it is older than the computer, and it is worth going back to where it was first asked carefully.

In 1868 James Clerk Maxwell published a short paper in the Proceedings of the Royal Society called “On Governors.” Its subject was the centrifugal governor that James Watt had fitted to his steam engines some 80 years earlier. The device is simple enough to picture: two iron balls hang on hinged arms from a spindle, and the engine itself spins that spindle. As the engine speeds up, the balls fly outward and rise; through a sleeve and a lever, that rising motion closes the steam valve, and the engine slows. As the engine slows, the balls fall back, the valve opens, and the engine speeds up again. The engine, in effect, regulates itself. Engineers who lived with these machines had learned two uncomfortable things about them. A governor sometimes hunted, swinging the engine above and below the intended speed instead of settling on it. And making a governor more sensitive, so that it reacted more strongly to small errors, often made the hunting worse rather than better. The intuitive cure was aggravating the disease. Maxwell’s contribution was to stop treating engine and governor as two separate things. He wrote down the equations of the engine and the governor together, as one system, and showed that whether the swings die away or grow depends on the roots of a single characteristic equation for that combined system. Stability, in other words, was not a property of the governor alone, nor of the engine alone, but of the way the two were coupled.

The paper is usually cited as the beginning of control theory, and when Norbert Wiener named the new science of cybernetics in 1948, he pointed back to it. I find it a useful lens for everything in this chapter, because the analogy holds closely. Every reconciliation loop in a cloud runtime is a governor: it senses the state of something, compares it with what was intended, and pushes to close the gap. Like Watt’s governor, every such loop can hunt, overshooting and correcting and overshooting again. And two governors attached to the same shaft, each with its own idea of the right speed, can fight each other indefinitely while the engine lurches between them. Where the analogy breaks is in number: Watt’s engine had one governor, and a cloud runtime has dozens, written by different people for different purposes, most of whom never imagined the others.

Encore runs dozens of these loops. They live in its Kubernetes cells, the self-contained copies of the stack that keep SaaS tenants apart from one another; in its serverless Sale Gate, the waiting room fans queue in before an on-sale; and in its GitOps repositories, which describe what every environment should look like. Most of these loops were written by someone else, by the Kubernetes project, a cloud provider or a tool vendor, and Encore’s engineers meet them as configuration rather than as code they can read at a glance. This chapter is about operating them well. For each loop we will ask three questions: what it owns, how fast it acts, and what it cannot see. Those three questions are enough to explain most of the surprises a runtime delivers. (The map of Encore in Chapter 1 shows where each piece sits.)

3.1 Declarative Systems and Reconciliation Loops

The anatomy of a loop

Before we can operate a crowd of governors, we need a clear picture of a single one. A reconciliation loop is a process that repeatedly observes the actual state of some part of the world, compares it with a declared desired state, and takes the smallest action it knows that moves the actual state toward the desired one. Then it does the same thing again, and keeps doing it for as long as it runs. The Architecture book introduced Kubernetes as such a loop and was content to let it work. Here we take the loop itself apart, because most of the surprises in running a modern platform come from parts of the loop that our configuration files never show.

Every loop has the same four parts as Watt’s governor, and it helps to set them side by side. The sensor reports what is true now: on the steam engine it is the pair of spinning flyballs, and in a Kubernetes controller it is a watch on the Kubernetes API, a stream of updates about the objects that controller cares about. The set point is what ought to be true: the intended engine speed, or the spec of a resource, the part of its manifest where we write down what we want. The comparator works out the difference between the two: the sleeve and lever that turn the height of the balls into a position of the valve, or a function in the controller that computes a diff between the spec and what the watch reports. The actuator changes the world: the steam valve, or an API call that creates or deletes a pod. When a deployment asks for five replicas and the watch shows three, the comparator finds a shortfall of two and the actuator creates two pods. That is the whole mechanism, and it is simple enough that it is tempting to stop looking at it too soon.

The loop also has two properties that the YAML never mentions. The first is its gain: how hard it pushes for a given error. A governor with long lever arms throws the valve wide at the slightest change of speed; an autoscaler that adds many replicas for a small rise in CPU is high-gain in just the same way. The second is its delay: how long the loop takes to sense the effect of its own last push. On the engine, steam takes time to reach the cylinders and the flywheel has inertia. In a cluster, a new pod must be scheduled, started and warmed before its share of the work shows up in the metrics the autoscaler reads. Maxwell’s result, translated out of his equations, is that a loop with high gain and long delay overshoots, notices the overshoot only after it has gone too far, and then pushes back too hard the other way. Picture an autoscaler that sees CPU high and adds pods; because the new pods are still starting, CPU still looks high on the next check, so it adds more; when they all arrive at once, CPU falls sharply and it removes too many. That is hunting, and the engineers of Watt’s day would have recognized it at a glance.

We should be honest about what this way of thinking buys us. Nobody tunes a Kubernetes controller by solving Maxwell’s characteristic equation, and the analogy is a way of seeing rather than a tuning method. What it gives us is a pair of questions to ask before trusting any loop: how hard does it push, and how long before it sees what its push has done? The answers are rarely in the manifests we write. They live in defaults, in documentation, and in the physics of whatever the loop is acting on, which is exactly why they are so often overlooked.

The idea did not arrive with Kubernetes. Its software lineage runs through Mark Burgess’s CFEngine (1993), which described the state that Unix machines should converge to and, on every run, repaired whatever had drifted from it. The shift was a quiet but deep one. A traditional script is a list of steps, run once, that assumes it knows the starting point; a convergent description says where the machine should end up, and can be run again and again because a machine that is already correct needs nothing done to it. Kubernetes and GitOps agents sit squarely in that tradition, which the OpenGitOps principles now write down in four words: state is declared (we describe what we want rather than how to get there), versioned (kept in Git, with its history), pulled (an agent inside the environment fetches it, rather than a pipeline outside pushing it in) and continuously reconciled (compared and repaired over and over, never applied once and forgotten). What has changed since CFEngine is the tempo. A configuration pass every few minutes has become hundreds of loops reacting within seconds, and interactions between loops that were once too slow to notice can now play out several times inside a single on-sale.

Level-triggered, not edge-triggered

How a loop decides when to act matters as much as what it does, and the vocabulary for that comes from electronics. A digital circuit can be built to respond to the moment a signal changes from low to high, or to whether the signal is high right now. Software borrowed both terms. An edge-triggered system acts on changes: it hears “a pod died” and starts a replacement. A level-triggered system acts on state: it looks, finds three pods where five were asked for, and creates two, whatever history led there. Kubernetes controllers are level-triggered. They do subscribe to watch events, but they treat each event only as a hint to look again sooner; the decision always comes from comparing the current state with the desired one. So if a controller misses an event, or restarts in the middle of its work, it loses nothing that matters. On its next pass it simply looks again, and the only price is the time before that pass comes round.

A theatre crew offers a useful picture. Before every performance the crew walks the stage with a preset sheet, checking each prop and piece of scenery against where it should be, and they do not need to remember what the actors moved last night, because the sheet and the stage tell them everything. That is level-triggered work. A crew that relied only on hearing each change called out over the headset would be edge-triggered, and one missed call would leave a chair in the wrong place for the rest of the run. The analogy breaks at one revealing point. A human crew member who finds something out of place may stop and ask whether the director moved it on purpose. A controller never asks.

Encore learned the difference the hard way. Its first tenant-provisioning automation was edge-triggered: a TenantCreated message started a script that created a database, a namespace and a DNS record. When the script failed halfway, nothing noticed, because the message had already been consumed and nobody was looking at the state it was meant to produce. Three Professional tenants lived for a month with a namespace and no database. The replacement is a small operator that reconciles a TenantDatabase custom resource on every pass. Its steps are idempotent, meaning that running a step twice has the same effect as running it once (“create the database if it does not exist”), so the operator can safely repeat its whole sequence each time it looks. Deletion goes through a finalizer, a marker on the resource that Kubernetes requires to be removed before the object can disappear, which gives the operator its chance to clean up the database and DNS record first. Now a half-finished tenant is simply a resource whose actual state differs from its desired state, and the next pass finishes the job without anyone being paged. A level-triggered loop forgives the past, because it never consults it.

That forgiveness has a price, and the crew analogy has already shown us where it lies. Because a level-triggered controller judges only the present state against its desired state, it cannot tell accidental drift from a deliberate human change. A hand-edited setting that is not in the desired state looks to the controller exactly like damage, and it will undo that change on every pass, as often as the human makes it. When that human is an engineer trying to help during a busy evening, and the controller is one of several on the same system, the result is the subject of the next part of this section.

When governors fight

Two definitions will keep the discussion precise. A controller in Kubernetes is a reconciliation loop for one kind of resource, and an operator is a controller that encodes one application’s operational knowledge behind a custom resource, as the TenantDatabase operator does for Encore’s tenant databases. A production cluster runs many of them at once. Besides the built-in controllers, Encore’s clusters run a Horizontal Pod Autoscaler (HPA) for each scaled deployment, the cluster autoscaler, the GitOps agent (Argo CD), the TenantDatabase operator and the platform’s node-upgrade reconciler, which drains and replaces nodes one at a time to keep each pool at its target Kubernetes version. Each of these is reasonable on its own terms. Trouble begins where two of them claim the same thing, which is to say where two actuators write to the same field, each steering toward a set point of its own.

The largest public example is recent: the race in Amazon DynamoDB’s DNS automation in October 2025, which the Operations book tells in full in its chapter on automation, in which two instances of one controller interleaved their work and left a regional endpoint’s record empty. Neither instance was wrong on its own; the fault lay in how their actions were interleaved. That is the uncomfortable lesson of control systems in general. We can verify each loop carefully in isolation and still know very little about how the loops behave together.

Encore’s version was small, and all the more instructive for being ordinary. On a Thursday before an on-sale, a Catalog engineer expecting a surge of browsing ran kubectl scale to take Catalog from 20 replicas to 60. Argo CD runs with self-heal on, which means it continuously compares the live cluster with Git and reverts any difference; it found replicas: 20 in the manifest and restored that count within seconds. Meanwhile, as browsing ebbed, the HPA, set to aim for 60% CPU with a floor of 12 replicas, began recommending fewer pods. An HPA does not scale down the moment its recommendation drops. It keeps a scale-down stabilization window, five minutes here, and scales down only to the highest recommendation it has made within that window, so that a brief lull does not shed capacity that will be needed a moment later. For five minutes, then, it held. Once the window was full of low recommendations, it trimmed toward 12. Argo CD saw a count that differed from Git and restored 20 within seconds, and the HPA, whose window still held nothing but low recommendations, trimmed again at its next evaluation. For the rest of the half hour the two traded the count back and forth several times a minute, and each trade started pods with cold caches. Nothing failed, which is why the episode is easy to dismiss. But the engineer believed the service was running at 60 replicas when it never was for more than a few seconds, and on a busier day real decisions would have rested on that belief. Figures 3.1 and 3.2 show the half hour and the governors involved.

Figure 3.1 — Catalog’s replica count during the fight, sketched. The spike to 60 is reverted in seconds; the autoscaler waits out its five-minute window and then trades the count with the GitOps agent several times a minute. Notice that the count never returns to the 60 the engineer expected.

Figure 3.2 — The governors on Catalog’s shaft. Two rows claim spec.replicas, and the manifest claimed it too. The last row has no record and no delay, which is why it loses.

Read Figure 3.2 as Maxwell might have read a drawing of an engine. Each row is a governor, and the columns give its lever, its delay and the place its set point is kept. The fight happened because three set points (the number in Git, the HPA’s target and the number in the engineer’s head) were all attached to the same field, and the engineer’s was the only one written down nowhere, so the two machines kept erasing it.

The repairs follow directly from that reading. First, one field, one owner. Encore removed replicas from every manifest whose deployment has an HPA, as both the Kubernetes and Argo CD documentation advise. The team followed the documented migration steps while doing so, because simply deleting the field can let the next sync reset the deployment to the default of one replica. Policy now rejects any manifest that sets replicas on a deployment an HPA also manages. Second, express intent to the owner. A planned surge is no longer a hand-typed replica count; it is a change to the HPA’s floor, made in Git through the curtain-up workflow that pre-warms each on-sale, which also warms node capacity ahead of the sale and lowers the floor again when the sale ends. The engineer’s intention is the same as it was on that Thursday, but it now reaches the one loop entitled to act on it, and it leaves a record. Third, respect the damping. The scale-down window plays the part of a dashpot on the flyballs (a piston moving through oil, which resists sudden motion), slowing the governor’s response so that it does not chase every flicker, and teams that shorten it without measuring tend to rediscover hunting.

Underneath the three repairs lies one structural idea. Two loops pulling on one lever oscillate. Two loops on two levers, with the slower one setting the faster one’s set point, form a hierarchy instead: Git, changed slowly and with review, sets the HPA’s floor; the HPA, acting in seconds, sets the replica count; the cluster autoscaler, acting in minutes, adds nodes when pods cannot be placed. Each loop has its own job and its own timescale, and none writes a field that another owns. Hierarchy is how control engineers have long mastered complexity, and it serves us just as well when the governors are made of software (Figure 3.3).

Figure 3.3 — Catalog’s governors as a hierarchy. Each slower loop writes the set point of a faster one, and only the dashed accent arrow claims a field another loop owns. Notice that it is also the only arrow that leaves no record.

Recap. A reconciliation loop observes, compares and acts, and its gain and delay decide whether it settles or hunts, just as Maxwell showed for the governor. Level-triggered loops forgive missed events and half-finished work, but for the same reason they undo any human intent that has not been declared. When two loops own one field they fight, so we give each field a single owner, express our intent by changing that owner’s set point, and arrange the loops in a hierarchy in which slower loops steer faster ones.

Exercise 3.1. Pick one production deployment and list every controller, script and human that can change its replica count or its nodes. For each, note its set point, where that set point is recorded and how quickly it acts. Is any field claimed twice? If so, decide which loop should own it and how the others should express their intent instead.

3.2 Kubernetes in Production

Requests, limits and the honest workload

Section 3.1 took the reconciliation loop apart in general. Kubernetes is where most of us meet such loops every day, and the first of them, the scheduler, reasons from information we hand it ourselves. Kubernetes schedules on promises. A pod’s resource request is the amount of CPU and memory it promises to need, and the scheduler places the pod only on a node where that much is still unclaimed. A pod’s resource limit is the most it will ever be allowed to take, however much is free. Notice what the scheduler does not look at when it places a pod: what the pod actually uses. It adds up the requests already on each node and compares them with the node’s capacity, much as a hotel takes bookings against its number of rooms without checking how many guests actually sleep in them.

Requests and limits together also set the pod’s quality-of-service class, and that class decides the order of eviction when a node runs short of memory. A pod whose limits equal its requests is guaranteed its share and is evicted last. A pod that requests less than its limit is allowed to burst, and is evicted earlier when the node is under pressure. A pod that declares nothing at all is evicted first. The numbers in a manifest are therefore not paperwork. They are the pod’s statement of how it should be treated on a crowded day.

Bilgin Ibryam and Roland Huß call the discipline of declaring these numbers the Predictable Demands pattern, and the reason it matters follows directly from the fact that the scheduler plans with promises. If a team requests far less than its service uses, the scheduler, believing the requests, keeps packing pods onto a node that looks half empty on paper but is in fact full, and the pods on it then fight over resources nobody reserved. If a team requests far more than it uses, the opposite happens: nodes look full while their processors idle, and the company pays for capacity it never touches. Either way the scheduler makes sound decisions from false information. This is why Encore right-sizes its requests every quarter from observed usage, so that the promises the scheduler plans with stay close to how the services really behave.

The two resources fail in very different ways when a container reaches its limit, and the difference shapes how we set them. A container that exceeds its memory limit is killed. That failure is loud and visible: the pod restarts, a counter goes up, and someone notices. A container that reaches its CPU limit is not killed but throttled: the kernel hands out CPU time in short accounting periods, and a container that has spent its quota for the current period simply waits until the next one begins. Nothing restarts, no counter that anyone usually watches goes up, and the only symptom is that some requests take longer than they should. Because the wait falls on whichever requests happen to be running while the quota is spent, the damage shows up in the tail of the latency distribution, at the p99, while the average looks healthy.

Encore learned this the slow way. The Orders service had a spiking p99 for a month before someone put a graph of throttled periods beside the latency graph and saw the two line up. The house rule that came out of that month is simple to state. Memory limits are set equal to memory requests, so a service is never surprised by a memory ceiling it did not plan for. CPU limits are left off latency-sensitive services, so that a short burst of work can borrow processor time sitting idle on the node, instead of waiting for the next accounting period.

The rule is contested, and I think rightly so, because it is safe only where CPU requests are honest. When a busy node has more demand than processor time, the kernel divides the time in proportion to the requests, so the request is what guarantees each pod its share. A service that drops its CPU limit while understating its request has given itself freedom to take from its neighbors and only a weak claim to protect itself from them. Noisy batch workloads at Encore therefore keep their CPU limits. A batch job that would happily use every core it is offered gains little from finishing sooner, and on shared nodes one team’s burst is another team’s tail latency. The rule is a trade between latency for the services fans wait on and fairness among neighbors, and it holds only while every team keeps its promises.

Disruption is a normal day

Requests and limits decide where a pod lives. The next question is what happens when the cluster needs to take that home away, which in a well-run cluster happens all the time. Kubernetes distinguishes two kinds of disruption. A failed node is an involuntary disruption: it cannot be negotiated with, and our only defense is having enough replicas elsewhere. Voluntary disruptions are the ones the platform causes on purpose, such as a drain so a node can be upgraded, or the cluster autoscaler removing a node it considers underused. These can be negotiated with, and the instrument of negotiation is the PodDisruptionBudget (PDB): a declaration that at most so many pods of a given set may be voluntarily unavailable at once.

The mechanism is straightforward. A drain does not simply delete pods; it asks the eviction API to evict them, and the eviction API checks each request against the relevant budget. If evicting a pod would leave more of its set unavailable than the budget allows, the request is refused and the drain waits, retrying until a replacement elsewhere has become ready. A stage crew works the same way during a set change: it may strike only so many pieces at once, because the scene still in front of the audience needs the rest to stand.

But managed upgrades wait only so long, and here the budget’s protection runs out. Google Kubernetes Engine (GKE) respects a PDB for up to an hour during a node upgrade and then evicts by force; Amazon’s managed node groups fail the update after 15 minutes unless told to force it. The providers have good reason to behave this way, since a single carelessly written budget, one that allows no pod to be unavailable, would otherwise block an upgrade indefinitely. The consequence for us is that a PDB is a request for patience, honored for a bounded time, and a service that relies on it must be able to become safe to evict well within that time.

A budget also protects only the count of pods, and nothing inside them. When a pod is evicted, Kubernetes does two things at about the same moment: it sends the process a SIGTERM, telling it to stop, and it begins removing the pod from the endpoints that services and load balancers route to. Those two messages travel by different paths and arrive at different times. The process may receive its SIGTERM while requests are still in flight, and while load balancers that have not yet heard the news are still sending it new ones. A pod that exits the moment it is told to drops that work on the floor, and a budget that counted perfectly will not have helped.

The proven remedy is graceful termination, which has three parts that work together. First, a readiness probe that fails as soon as shutdown begins, so the pod announces that it should receive no new traffic. Second, a short preStop pause, which delays the stop signal for a few seconds so the endpoint removal reaches every load balancer before the process stops accepting work. Third, a grace period long enough for the process to finish the requests it already holds. With all three, a drained pod leaves the way a good session musician leaves the stage: it stops taking new requests, finishes the phrase it is playing, and only then puts down the instrument.

The budget makes one more assumption that is easy to miss: that its pods are spread out. If two members of one group share a node, one drain takes both, and a budget that imagined them as independent is quietly wrong. Topology spread constraints close that gap by telling the scheduler to place a set’s pods across nodes and across zones, so that no single node or zone holds more than its share. Encore’s golden-path template sets up both graceful termination and topology spread for every new service, so a team has them from its first deploy without having to learn either lesson from an incident.

Even with all of this in place, a PDB is a promise about how many, never about which. Seat Inventory shows why the distinction matters. It is the service that sells each seat exactly once, and it runs each shard as a small replicated group with a fenced leader, its members spread across zones. Its PDB allows one pod unavailable per shard, so no drain will take a shard below quorum, the majority it needs to keep accepting writes. In counting terms the shard is safe. But the budget treats every member of the group as interchangeable, and they are not, because one of them is the leader. A drain may remove the leader, and a leader’s departure, though safe, triggers a failover, with a short window of rejected writes while a new leader is elected and fenced in. Database operators such as CloudNativePG know this and move leadership to another member before a drain, so the departing pod is an ordinary follower by the time it goes. Seat Inventory has no such operator yet.

On a quiet Tuesday the window is invisible: few fans are buying, and a retry a moment later succeeds. Whether it stays invisible every day of the year depends on which other loops run at the same time and what they know about the calendar, since nothing in a drain knows whether a major on-sale is under way. The Operations book, in its account of the Vesper Hale on-sale, shows the cost when nobody asks.

Upgrades as a standing condition

Drains are frequent for a simple reason: the platform underneath never stops moving. Kubernetes ships three minor releases a year, each patched for about a year, and managed providers eventually upgrade clusters that fall behind, ready or not. An organization that runs Kubernetes is therefore never finished upgrading; it is always somewhere in the middle of the next one. The useful image is a ship’s hull, which must go to dry dock on a schedule because fouling and corrosion accumulate whether anyone is watching or not. The analogy breaks in an instructive place. A ship must leave service to be docked, but a runtime can be docked one compartment at a time while it sails, and the craft of Kubernetes upgrades lies in choosing those compartments and the order in which they go.

Encore’s Runtime team chooses them by upgrading in waves. Each region runs a core cluster for Encore’s own services and six cells for Professional tenants. A new version reaches a canary cell of internal test tenants first, where nothing a customer depends on can break, then the other cells, then the core clusters, with the node-upgrade reconciler from Section 3.1 draining node pools through the PDBs described above. The ordering is deliberate: each wave tests the next, and the clusters whose failure would hurt most take the version last, after it has run on clusters whose failure would hurt least.

Before each wave a check scans manifests, add-ons and custom resource definitions for APIs the new version removes. Kubernetes retires old API versions as it evolves, and the commonest upgrade failure is not a fault in Kubernetes itself but a chart still asking for something that no longer exists. Finding those requests before the wave turns a failed upgrade into an ordinary pull request. Figure 3.4 shows the waves.

Figure 3.4 — A Kubernetes upgrade moving through Encore’s 14 clusters in three waves. The canary cell takes a new version first, and the core clusters, which carry Encore’s own services, take it last.

Many clusters, deliberately

The waves are possible only because Encore runs many clusters, and that was a choice with a cost. One large cluster would pack better, since spare capacity in one corner could absorb demand from another. But its control plane would be a single blast radius, so one bad configuration could reach every service and tenant at once, and each of its upgrades would be a single event with nowhere to rehearse it. Cells, which Chapter 5 treats in depth, trade some efficiency for containment: a failure, or an upgrade gone wrong, stays inside one cell.

The price of containment is paid in tooling, because 14 clusters tended by hand would become 14 slightly different clusters within a year. Encore pays it with one GitOps layout, one set of policies and one set of dashboards, which make the 14 clusters one fleet whose members differ only where a difference was intended. This is decomposition used well: the fleet is split so that failures stay small, and held together by shared structure so that the parts do not drift apart.

Recap. Requests and limits are the promises the scheduler plans with, so they must be honest, and a CPU limit fails quietly, as tail latency, where a memory limit fails loudly. PDBs govern how many pods a voluntary disruption may take, for as long as the upgrader will wait, and say nothing about which; graceful termination and topology spread protect what the count cannot. Upgrades never end, and they are best absorbed in waves through cells run as one fleet.

Exercise 3.2. For one latency-sensitive service, graph CPU throttling against p99 latency for a week, and list its PDB, its spread constraints and its shutdown sequence. Which pod’s loss would hurt most, and does anything in your cluster know that?

3.3 Serverless and Managed Services — Operating What You Don’t Run

The shared responsibility line

So far in this chapter we have operated machinery we could open: clusters whose nodes we drain, controllers whose settings we tune, manifests we can read line by line. A growing share of every modern system is machinery we cannot open at all. A serverless function platform runs our code on servers we never see, and a managed queue, database or object store is a service whose internals belong entirely to someone else. Encore’s Sale Gate, the waiting room we met at the start of the chapter, is built this way, and so the question of this section is a strange one for an operations book: what does it mean to operate something we do not run?

Cloud providers answer part of that question with what they call the shared responsibility model: the provider is responsible for the security and operation of the cloud, and the customer is responsible for what they put in it. The provider keeps the data centers powered, the hypervisors patched and the function platform scheduling; the customer chooses the configuration, writes the code, guards the credentials and decides who may call what. Amazon has since extended the same division to resilience, so that the provider answers for the resilience of its infrastructure and the customer for the resilience of the workloads built on top of it. The model is a useful map of who fixes what. What it does not mention, though, is the user, and responsibility to the user is never shared. A fan who refreshes a frozen waiting-room page does not know or care which side of the line the fault lies on. When the Sale Gate is slow, fans blame Encore, and they are right to, because Encore chose the provider, chose the configuration and promised them a fair place in the queue.

Operating a managed service therefore means operating everything around a box we cannot open: how we configure it, the quotas that bound it, the ways it fails as seen from outside, and how we respond when it does. Touring crews know this work well. Before a show arrives in a new city, a production manager advances the venue, which is to say she learns ahead of time its power limits, its load-in doors, its rigging points and its curfew, so that nothing about the building surprises the crew when the trucks pull up. She cannot rewire the hall, but she can know its limits precisely and plan the show to fit inside them. We do the same for a provider, reading its quota pages and documented limits and, because documents are never the whole story, taking our own measurements of how it actually behaves. The analogy breaks in one instructive place. A venue’s limits are written on a spec sheet and stay put for the night, and the production manager can walk the building to check them. A provider’s internals cannot be walked, and its behavior can change beneath us without notice, so our measurements are never finished; they are a standing practice, renewed the way a crew re-advances a venue it has played before.

Quotas are part of the architecture

Every managed service has limits, and the most consequential ones are often invisible until we hit them. Function platforms, for example, cap concurrency, the number of function instances that may be running at the same moment, per account and per region. That cap is a single pool shared by every function in the account, unless some of it is set aside for a particular function. Two functions that have nothing to do with each other in our architecture diagram are therefore neighbors on the provider’s ledger, drawing from the same well.

Encore learned this during an on-sale rehearsal. A Sales Analytics backfill, deployed as functions in the same cloud account as the Gate, consumed most of the regional concurrency just as the Gate’s synthetic fans began arriving. Nothing was broken in either system; each did exactly what it was built to do. The failure lay in a dependency that no diagram showed, the shared concurrency pool, and it is a small lesson in one of the oldest themes of our discipline: the accidental complexity of a system often hides in the resources its parts share rather than in the parts themselves.

Three changes followed, each aimed at a different part of the problem. First, the Gate now runs in its own cloud account, as does each serverless cell for self-service venues, as Amazon’s multi-account guidance recommends; an account boundary turns one shared pool into separate pools, so a busy neighbor can no longer drain the Gate’s well. Second, the Gate has reserved concurrency, a portion of the pool set aside for it alone. A reservation cuts both ways, because it is a ceiling as well as a floor: the function is guaranteed that many instances, but it can never use more. So the reservation is sized to the forecast peak and revisited whenever that forecast changes, since a reservation sized for last year’s biggest show would quietly cap this year’s. Third, a quota inventory is generated nightly and kept beside the capacity models of Chapter 2. It records the limits on the path a fan travels, compares each one with the forecast peak, and alerts when the forecast comes within 30% of any limit.

That threshold is deliberately stricter than the warning a provider typically offers, which fires at around 80% of current use, and the difference is one of timing. A warning based on current use tells us we are close to a limit only once we are already close to it, which during an on-sale is far too late. A warning based on the forecast tells us well ahead that the next big show will come close. Encore needs that lead time for two reasons. Raising a limit means asking the provider, and an increase can take days to be granted. And some limits do not appear in the provider’s quotas API at all, so no automated warning would ever fire for them; they live in the inventory only because someone read the documentation and wrote them down.

Cold starts deserve the same treatment. A cold start is the delay a function suffers when the platform must create a fresh instance to serve a request, loading the runtime, the code and its dependencies before any work can begin; a warm instance, already running, skips that delay. For a waiting room, where a crowd of fans arrives within seconds, a wave of cold starts comes at exactly the wrong moment. The Gate therefore pre-warms with provisioned concurrency, instances the platform creates and keeps ready in advance, before every scheduled on-sale. The pre-warming begins early enough for the allocation to complete, because allocating provisioned concurrency itself takes minutes, and a warm-up that finishes after the doors open has warmed nothing. An unscheduled surge, such as a surprise announcement that sends fans to the site without warning, still meets cold functions. Encore cannot prevent that, so it plans for it: its latency service level objective (SLO) includes Encore’s own allowance for cold starts, and the team measures real cold-start time monthly by deliberately forcing one, rather than trusting a figure that may have drifted since the last check.

When the provider is down

Quotas and cold starts are limits we can learn and plan within. The harder case is the day the provider itself fails, and one such day is well documented. On 28 February 2017, a mistyped command during routine maintenance removed more capacity than intended from Amazon S3 in the US-East-1 region, and much of the internet that depended on S3 failed for about four hours. One detail of that day has stayed with operators ever since. The administration console of Amazon’s own Service Health Dashboard depended on S3, so for a while the provider could not post the outage on the very status page its customers were refreshing to learn whether there was one.

That detail carries the general lesson. A provider incident is one that the team owning the symptoms can mitigate but cannot repair. Encore’s engineers cannot restart a provider’s storage fleet or roll back its deployment; the fix belongs to someone else, on someone else’s schedule. What remains in their hands is everything on their own side of the line: noticing early, limiting the harm to fans, and keeping people informed. Those are exactly the decisions that go badly when improvised in the moment, under pressure and with incomplete information, so Encore makes them in advance.

Each managed dependency in Encore’s runbooks therefore comes with three pre-decided answers. The first is how we detect its degradation ourselves: through our own synthetic checks, which exercise the dependency the way fans do, whatever the provider’s status page says, because 2017 showed that the status page can be the last to know. The second is what we degrade to, chosen so that the experience worsens gracefully instead of collapsing: the Gate holds fans in the queue with admissions paused, preserving their places instead of turning them away, and the Catalog serves cached pages, which may be slightly stale but are far better than an error. The third is how we tell fans and organizers what is happening, since a silent waiting room breeds more refreshes, more support calls and more distrust than an honest message would.

I have come to believe that this is the real shape of the bargain behind every managed service. A managed service removes the work of running it and leaves us the work of surviving it. The first kind of work is real, and handing it to a provider is often the wisest decision a small team can make. The second kind stays with us whatever we buy, and a team that forgets it will rediscover it on the day of its biggest on-sale.

Recap. The provider shares responsibility for the machinery, but responsibility to users stays with us, because the product they experience is ours. Quotas, reservations and cold starts are properties of our own architecture, to be kept in an inventory, budgeted and tested, and measured against the forecast rather than against today’s use. We can mitigate a provider incident but never repair it, so we decide in advance how we will detect it ourselves, what we will degrade to, and what we will tell the people waiting.

Exercise 3.3. List every managed service on your most important user path, together with its tightest quota, your forecast peak measured against that quota, and what your system does when the service is down for an hour. Mark any entry where the honest answer is “we would find out.”

3.4 Infrastructure as Code and GitOps

State, drift and the reality behind the file

The Architecture book made the case for infrastructure as code (IaC) and GitOps: describe the environment in files, keep the files in version control, and let tools make the world match them. That case was about design. Operating these tools is about something less tidy, namely the gap between what the code says and what actually exists, and that gap is where most of this section lives.

The gap shows itself first in a file most engineers would rather not think about. A Terraform configuration, or one written for its open-source fork OpenTofu, says something like “there is a database called ledger_db with these settings.” The cloud provider knows nothing of that name; it knows an instance with an identifier it generated itself. So the tool keeps a state file, a record of which real resource corresponds to which declaration in the code, and consults it on every run to decide whether a declaration is already satisfied, needs changing or needs creating. The state file is the tool’s memory of the world, and it is the tool’s only way to connect a line of code to a thing that costs money and holds data.

That makes the state file production data in every sense that matters. Two engineers running the tool at once can corrupt it, so it needs locking. Losing it leaves the tool unable to recognize resources it created, so it needs backups. And because the tool records the attributes the provider returns, such as a database password generated at creation time, the state often holds secrets that the code itself never contains. It therefore needs encryption, access control and an audit log, exactly as a customer database does. Teams that keep state on a laptop or in an unprotected bucket have, without meaning to, created the most sensitive and least guarded file in their estate.

Once the tool has a memory of the world, the world can disagree with it. Drift is any difference between the declared state of infrastructure and its actual state. It matters how the drift arose, because the right response depends on who or what made it. Drift made by people, like the console fix applied at 2 a.m. during an incident, should be either reverted or written back into code the next morning; left alone, it is a change nobody reviewed and the next run of the tool will silently undo. Drift made by other loops is a different matter. An autoscaler that resizes a node group is doing its job, and the IaC tool should be told not to care, which Terraform expresses with an ignore_changes setting on that attribute. If we forget, the IaC tool becomes another governor on the autoscaler’s shaft, in the sense of Section 3.1: two loops claiming one value, each undoing the other. The rule of one owner per field applies to infrastructure code as much as to Kubernetes manifests.

Drift is easy to find if we look for it, and the way to look is to ask the tool what it would change. Encore runs a plan against every stack nightly, without applying it. A plan that finds nothing to do means code and world agree; a non-empty diff means something has drifted, and Encore files it as a ticket for the team that owns the stack. Most of those tickets are dull, a tag edited by hand or a setting nudged during an incident, and that dullness is the point: drift found the next morning is a chore, while drift found months later, in the middle of an unrelated change, is a puzzle.

The blast radius of a plan

A plan is the tool’s statement of intent: the list of resources it will create, change and destroy if we approve it. Its most important line is the summary at the end that counts what it will destroy, because creation is usually reversible and destruction of a stateful resource often is not. The blast radius of a plan is the set of resources a single apply could affect, and it is bounded by the stack, the unit of code the tool plans and applies at once. A stack that holds one service’s queues can, at worst, break that service; a stack that holds the whole company can break the whole company.

That is the operational reason behind Kief Morris’s case for small stacks. Morris also warns against cutting too fine, since every boundary between stacks becomes a dependency, with outputs from one stack feeding inputs to another, and those dependencies become the new risk. The question is where to cut. Encore cuts along ownership and rate of change: resources that the same team owns and changes at the same pace live together, and resources that change rarely, like networks, sit apart from those that change weekly. In practice that means stacks follow cells and services, which is also how the rest of Encore’s runtime is decomposed.

Encore learned why the summary line matters before it had any of this. Its earliest infrastructure lived in a single stack, and a harmless-looking module rename once produced a plan that would destroy and recreate the Finance ledger’s database. The reason lies in how the state file identifies resources. It keys each one by its address in the code, so when the module was renamed the tool saw a resource vanish under the old address and a new one appear under the new address, and it proposed what any literal-minded bookkeeper would: delete the first and create the second. Nothing in the plan said “rename.” A reviewer caught it by reading to the bottom of a long plan, where the destroy count sat. Encore was lucky in its reviewer, and it decided not to depend on that luck again. Three safeguards now do her job.

The first removes the misunderstanding at its source. A rename is declared with a moved block, which tells the tool that the old address and the new one name the same real resource, so the plan shows a move and nothing destroyed:

moved {
  from = module.finance_db.aws_db_instance.this
  to   = module.ledger_db.aws_db_instance.this
}

The second guards against destruction even when the code asks for it. Stateful resources carry prevent_destroy in code and deletion protection on the resource itself. Either alone is weaker than it looks. prevent_destroy makes the tool refuse any plan that would destroy the resource, but it lives in the very code a careless edit removes, and a refactor that deletes the resource block deletes the guard with it. Deletion protection lives with the provider, outside our code, so the provider refuses the deletion whatever our code says. Together they mean two independent mistakes are needed to lose the ledger’s database.

The third changes what the reviewer sees first: plans are reviewed with the destructive summary at the top, so the most consequential fact is the first thing read rather than the last.

Surgery offers an older version of this discipline. Surgical teams adopted a practice for the moment before the first incision, the time-out, in which the team stops and confirms aloud the patient, the procedure and the site. The World Health Organization’s checklist study of 2009 found that complications and deaths fell where the checklist was adopted. The lesson I take from it is that the worst errors are made confidently, at the moment of commitment, by people who have no doubt they are about to do the right thing; a ritual pause at that moment catches what confidence hides. A plan review is our time-out. The analogy has a limit worth noticing, though. In an operating theater the whole team stands in the room and speaks; a plan review is often one tired engineer skimming a diff alone, and a pause that depends on one person’s attention is a weak pause.

So we automate the part of the time-out that can be automated. Policy as code means rules written as code and evaluated by a machine against every plan and manifest before it is applied, so the check happens every time whether or not anyone is alert. Open Policy Agent (OPA) is a common engine for such rules. Encore’s rules deny three things: any plan that destroys a resource labeled stateful without a separately approved override; any Kubernetes manifest that sets replicas beside an HPA, which is the configuration that set Argo CD and the autoscaler fighting over Catalog in Section 3.1; and any public storage bucket. Each rule records a lesson that someone, at Encore or elsewhere, learned the hard way, which is perhaps the best way to read any organization’s policy set.

GitOps repositories and promotion

Infrastructure code shapes the cloud resources a system runs on; GitOps applies the same thinking to what runs on the clusters themselves. The term GitOps was coined at Weaveworks in 2017, for an older idea made precise. The older idea is that the desired state of a system should be written down and versioned. The precision is in how changes reach the cluster. In a push model, a pipeline outside the cluster holds credentials and applies changes to it. In GitOps, a pull-based agent inside each cluster watches a Git repository and continuously reconciles the cluster toward it, so that people change the repository and nothing else changes the cluster. The agent is one more governor of the kind Section 3.1 described, with Git as its set point.

Encore splits its repositories by purpose. Application repositories build container images and publish them by digest, the content hash that names one exact image, rather than by a tag like latest that can be moved to point at something else. A separate environment repository, encore-deploy, holds every cluster’s desired state. It is organized as a shared base, describing what every environment has in common, with one overlay per environment and cell that records only how that place differs. Because each overlay states its differences explicitly, environments cannot quietly diverge: any difference between staging and production is a line someone wrote and someone can read. An Argo CD instance in each cluster reconciles from encore-deploy.

With that layout, releasing a new version means changing a digest in one directory, and promoting it means changing the same digest in the next directory. Figure 3.5 shows the stages.

Figure 3.5 — Promotion as a series of pull requests. Each stage is a commit someone can read and revert, and the gates are the same SLO signals Chapter 1 defined, applied to the deployment itself.

Two of these gates need more explanation than a table can give. The canary cell serves only internal test tenants, so its natural traffic is a trickle, and an SLO that holds for an hour on a trickle proves little; almost anything survives being barely used. Encore therefore drives synthetic buyers through the canary cell to give the new version real work, and it treats the first Professional cells, where paying tenants arrive, as the real test. That is why a person on the owning team approves each step from there on, watching each cell’s burn rate before moving to the next.

The second subtlety concerns what the repository can tell us. The repository records what was intended to run, which is not always what did run. A sync can fail halfway; a resource can be applied and then crash; someone can change the cluster directly. Argo CD reports two statuses for each application that close this gap. Its sync status says whether the live cluster matches Git, and its health status says whether the resources it applied are actually working. Encore alerts on any application that has been out of sync for more than 15 minutes and on any whose health Argo CD reports as degraded. Only those two signals, read together, tell us what is really running.

Emergencies are where GitOps is most often misunderstood. The model says nothing changes the cluster except Git, but during an incident a person may need to act faster than a pull request can be written, reviewed and merged. The danger is that the agent does not know there is an incident. Encore’s break-glass procedure grants a named engineer an hour of direct cluster access, logged and reviewed afterwards, and its first step is to suspend automated sync for the affected application. Without that step, Argo CD’s self-heal treats the engineer’s fix as drift and reverts it within seconds, as it reverted the Catalog scale-up in Section 3.1, long before anyone could write the change back to Git. The procedure ends where the model requires it to: with a commit that records the fix in encore-deploy, and with automated sync restored. The order matters. A person acting during an incident must first tell the loop to stand aside, and the loop must be brought back only once Git agrees with what the person did.

Break-glass is for the unplanned. For planned quiet periods, Argo CD offers a gentler tool: its sync windows declare, per project, when automated syncs are allowed or denied. Together with GKE’s maintenance exclusions and their equivalents at other providers, they are the established form of calendar-aware automation: automation that knows what day it is and holds still when the business needs stillness. Tied to Encore’s on-sale calendar, a deny window takes only a few lines:

kind: AppProject
metadata: {name: purchase-path}
spec:
  syncWindows:
  - kind: deny
    schedule: "0 8 * * 5"      # T−2h before a Friday 10:00 on-sale
    duration: 4h
    timeZone: Europe/Paris
    applications: ["*"]
    manualSync: true           # break-glass can still sync by hand

Read in plain terms, the window opens at 08:00 Paris time every Friday, two hours before the usual 10:00 on-sale, and lasts four hours, until noon. During it, Argo CD will not automatically sync any application in the purchase-path project, while manualSync: true leaves the door open for a person following the break-glass procedure.

Encore has the tool and has not wired it to the calendar. Its freeze lives instead in the delivery pipeline and in the on-sale channel, as Chapter 4 describes, which means it holds back the changes people make and leaves the agents to their own schedule. And even a sync window, had Encore configured one, would hold only what Argo CD applies. The node-upgrade reconciler is not an Argo CD application but a platform controller, and it reads disruption budgets and nothing else; it does not know an on-sale is under way, and no sync window would hold it. A freeze is only as wide as the set of loops that obey it, and it is worth listing, for every loop that can change production, which calendar, if any, it reads.

One more piece of state never belongs in the repository at all. Secrets stay in a secrets manager, and an operator in each cluster materializes the references declared in Git into the secrets the workloads use, so the repository names a secret and never holds it. Anyone who can read encore-deploy learns which secrets exist and nothing about their values. Chapter 7 takes up rotation.

Recap. IaC state is production data, the tool’s only memory of which real resource each line of code describes, and it deserves the protection we give any database. Drift is reverted, written back or deliberately ignored, depending on whether a person or another loop made it. Small stacks, moved blocks, deletion protection held by the provider and policy as code together make it hard for a destructive plan to pass review unnoticed. GitOps promotion is a chain of readable pull requests; only sync and health status say what is actually running; and an emergency change must first tell the loop to stand aside, then end by writing the change back to Git.

Exercise 3.4. Count the resources in your largest stack, and write one policy rule that would have blocked the worst infrastructure change your team has applied. Then walk through your break-glass procedure step by step: at which step would your GitOps agent undo the fix, and what in the procedure stops it?

3.5 The Platform as a Paved Road

A product with a road crew

Look back over this chapter and count the machinery: reconciliation loops and the fights between them, requests and limits, disruption budgets, cluster upgrades in waves, provider quotas, Terraform state, GitOps promotion and sync windows. Every piece of it matters, and almost none of it is what Encore’s 40 stream-aligned teams are paid to think about. A stream-aligned team is one organized around a single flow of business change, such as checkout, the seat map or notifications, and its attention belongs to that flow. If each of those 40 teams had to master the runtime as deeply as this chapter has, Encore would be paying 40 times over for the same expertise, and paying it in the currency it can least spare, which is the attention of people who should be thinking about fans buying tickets. That is why Encore has a platform at all. The Architecture book described how Encore built its platform; the question for us now is how that platform is run, day after day, by the people responsible for it.

Matthew Skelton and Manuel Pais gave the platform team’s purpose its clearest modern statement in Team Topologies. A platform team exists, they argued, to reduce the cognitive load of stream-aligned teams, by which they mean the amount of knowledge and mental effort a team must carry in order to do its work. A team can hold only so much; whatever room the runtime takes up is room the team’s own domain cannot use. From that purpose follows their second idea, the thinnest viable platform: the smallest set of services that lets stream-aligned teams deliver without deep knowledge of the layers beneath them. It is tempting to read “thin” as a matter of taste or budget, but I have come to believe it is first of all an operating virtue. Every capability a platform offers is a promise, and every promise must be run, upgraded, patched, supported at awkward hours and, one day, retired. A platform that offers 20 capabilities has made 20 such promises, and the platform team will keep paying for each of them long after the excitement of building it has faded. Thinness is how a platform team keeps its promises few enough to keep well.

The phrase paved road is Netflix’s term for this kind of platform; Spotify’s “golden path” is its close cousin, and both describe a supported, well-lit route from an idea to a running service. The metaphor deserves to be taken seriously, because it teaches more than it first appears to. A sensible city paves the routes people actually walk, and it often learns where those routes are by watching where the grass wears away. Nobody is forbidden to cut across the park; the paved path simply becomes the easiest way to go. Encore’s platform teams work the same way. They do not forbid teams from leaving the road, but they measure where teams leave it and ask why, because each worn track across the grass is a sign of a need the road does not yet serve.

The analogy breaks at one point, and the break is the most useful part of it. A city can count on its citizens needing to get from home to work; the traffic arrives whether or not the road is any good. A platform enjoys no such certainty. A stream-aligned team can always choose to build its own deployment pipeline, run its own database or reach past the platform to the cloud provider directly, and it will do so the moment the road becomes slower or more frustrating than the detour. So a platform must keep earning its traffic, release after release, much as any product must keep earning its customers. The evidence that this is hard is sobering. DORA’s 2024 report found that people using an internal platform reported higher productivity, but that platforms also came with lower delivery throughput and lower change stability, and that they helped most where developers kept their independence. The road, in other words, is good for its travelers only when they remain free to leave it, and the platform teams that understand this tend to build better roads.

Operating the platform itself

If a platform is a product, then someone must run it in production, and the obvious candidate is the team that built it. Camille Fournier and Ian Nowland, in Platform Engineering, insist on exactly this: a platform team must own the operations of what it builds, on-call included. The reasoning is the same one that makes stream-aligned teams run their own services. The people who carry the pager learn what breaks, and that knowledge flows back into what they build next. Encore holds to this rule. The Reliability team carries the pager for the platform planes, and each plane has its own SLOs, just as a checkout service does. Among them are GitOps sync latency (how long a merged change takes to reach the cluster), cluster API availability (whether teams and controllers can talk to Kubernetes at all) and the time to provision a new service (how long a team waits between asking for a service and having one). Each of these measures something a stream-aligned team feels directly, which is the point: a platform’s SLOs should describe the road as its travelers experience it.

I spent years doing platform engineering inside a large bank, and the costliest lesson I took from that work was that users judge a platform by its worst week more than by its average one. A platform that is excellent 49 weeks a year and fails badly during the other three will not be remembered for the 49. Teams who were burned during a release crunch or a quarter-end close start building their own detours, quietly, and the platform loses traffic it may never win back. This is one more reason for the platform team to hold its own pager: the worst week is something you feel most sharply when it is your phone that rings.

The same reasoning applies, with even more force, to the platform’s own automation. The node-upgrade reconciler, the GitOps agents and the policy engine all act on other teams’ services, and they often do so when nobody is watching. They are, in effect, members of the road crew who work through the night. The published practice for such automation is plain. It should roll out under the same safety rules as any other change, and windows tied to the business calendar should keep it out of business-critical hours. Encore classifies its platform automation as “low-risk,” and on most days it is. A node drain at 03:00 on a Tuesday, one node at a time and within the disruption budgets, is exactly the kind of quiet maintenance that keeps a fleet healthy. But “low-risk” describes the action on an ordinary day, not on every day. As Section 3.4 showed, the reconciler is not an Argo CD application, so no sync window would hold it, and it is exempt from the on-sale freeze. Whether it should be held to the calendar on the days that matter most, the Friday mornings when a big on-sale opens, is a question the platform group has not yet asked.

Migrations are the platform’s other permanent work, and they are where a platform’s promises are tested most directly. Every time the platform changes a base image, a cluster version, a database engine or a deployment template, some number of stream-aligned teams must move from the old road to the new one, and every such move costs them attention they would rather spend elsewhere. Fournier and Nowland treat migration and sunsetting as staffed product work, planned and resourced like any feature, much as Google runs its large-scale changes centrally rather than leaving each team to find its own way. Encore adds a rule of its own: the platform team performs the first three migrations itself before it writes the self-service guide. The reasoning is practical. A guide written in advance describes the road as its designers imagine it; a guide written after three real migrations describes the road as it really is, with the surprising dependency, the undocumented setting and the step that took an afternoon instead of five minutes. The teams that follow then travel a road someone has already walked.

Recap. A platform is a product whose purpose is to lower the cognitive load of other teams. It should stay as thin as that purpose allows, because every capability is a promise that must be run and eventually retired, and it must earn its traffic without mandates, since teams are always free to take a detour. The platform’s teams own its operations and its SLOs, and its automation is owed the same rollout rules and calendar windows as any other change. Migrations are staffed product work, and the platform team does the first ones itself so that the guide describes the real road.

Exercise 3.5. Pick one capability that your platform, or your cloud provider, offers your team. Who is on call for it, and what is its SLO? Does it respect any calendar of yours, such as a release freeze or a peak trading period, or does it act on its own schedule? When was the last migration it forced on you, and how much did that migration cost your team? If you cannot answer these questions, write them down and send them to the capability’s owners.

Kata: The Lunch Rush

Your brief: “Forkline.” Forkline is a food-delivery marketplace that connects 18,000 restaurants and 25,000 couriers across six countries. It reaches them through three pieces of software: a customer app, a courier app and a tablet that sits on each restaurant’s counter. Its demand has the shape every food business knows, quiet most of the day and then a sharp rise at lunch and again at dinner, and those meal peaks are the hours that matter most. Sixty-five engineers in nine teams build the product, and a platform team of four looks after everything underneath it.

The machinery underneath has aged. Forkline runs three Kubernetes clusters, each built by hand, and all three are two minor versions behind; in five months the provider will upgrade them by force, whether Forkline is ready or not. Its infrastructure lives in a single Terraform stack of 1,700 resources whose plan takes 25 minutes to run, and because nobody wants to wait that long for a small fix, engineers make their fixes in the cloud console instead, so the code drifts further from the world each week. The order webhooks, which carry each new order to the restaurant’s point-of-sale system, run as serverless functions, and last month they were throttled when a tablet update made every restaurant reconnect at once. And every weekday at lunch there is a quiet tug-of-war over the dispatch service, which assigns couriers to orders: its HPA scales it on load, an engineer habitually scales it up by hand at 11:30, and nobody is sure which of them wins.

Each deliverable below applies one part of this chapter to Forkline, roughly in the order the chapter developed them. Read together, they should describe a single runtime in which every lever has one owner, every change has a bounded blast radius, and the four people who run the platform can still sleep.

Deliverables:

1. Controller census — List every loop and every human that changes the dispatch service’s replica count, its nodes or the infrastructure beneath it. For each, record the field it acts on, the set point it steers toward and where that set point is written down, and how quickly it acts. This is Exercise 3.1 at the scale of a company, and it should make plain whether the 11:30 scale-up and the HPA are claiming the same field, as Catalog’s engineer and Argo CD once did at Encore.
2. Upgrade plan — Show how Forkline reaches a supported Kubernetes version before the provider’s deadline arrives. The plan should cover the PodDisruptionBudgets that keep drains from taking too many pods at once, the graceful shutdown that protects the requests inside each pod, the order in which the three clusters move, and the maintenance windows that keep every upgrade clear of the lunch and dinner peaks. Aim for a plan that does more than survive this one forced upgrade: it should make the next one routine.
3. Stack decomposition — Explain how to split the 1,700-resource stack into smaller stacks without destroying anything along the way, using moved blocks or their equivalent so that the plan shows moves rather than deletions. Name the resources that get deletion protection on the provider’s side as well as in code, and write three policy-as-code rules that would stop the most dangerous plans and manifests before anyone has to catch them by eye.
4. Managed-service survival sheet — Inventory the quotas and limits on the order path, especially those the provider sets, since one of them throttled the webhooks last month. Then decide, in advance, what the webhooks degrade to during a reconnection storm or a provider outage, how Forkline will detect that degradation with its own checks, and how it will tell restaurants and customers.
5. Thinnest viable platform — Describe what a four-person platform team should offer the nine product teams, what it should decline to offer, and the SLOs it will be held to. A platform team that small cannot run many things well, so the list of refusals matters as much as the list of services.

Rubric:

Where you now stand

You can now read a runtime as a collection of governors, each a loop that watches one part of the system and pushes it toward a set point. For any loop you can name the field it owns, how fast it acts and what it cannot see. You can keep two loops from fighting over one lever by giving each field a single owner and asking anyone else to express their intent through that owner’s set point, and you can tell a loop to stand aside when a person must act in an emergency. You can run clusters, functions, infrastructure stacks and GitOps repositories as one fleet rather than a collection of special cases, and you know where the provider’s responsibility ends and yours begins. Everything in this chapter has been about the machinery that holds a system steady. Chapter 4 turns to the changes we deliberately push through that machinery, and to why change, more than anything else, is what breaks production.

References

• James Clerk Maxwell — On Governors. Proceedings of the Royal Society of London, 1868.
• Kubernetes — Horizontal Pod Autoscaling and Disruptions. Documentation — free online.
• Argo CD — Automated Sync Policy and Sync Windows. Documentation — free online.
• Amazon Web Services — Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region. 2025 — free online.
• Amazon Web Services — Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region. 2017 — free online.
• Google Cloud — Node pool upgrade strategies. Documentation — free online.
• HashiCorp — Refactoring modules. Terraform documentation — free online.
• Bilgin Ibryam, Roland Huß — Kubernetes Patterns, 2nd ed.. O’Reilly, 2023 — free online.
• Kief Morris — Infrastructure as Code, 3rd ed.. O’Reilly, 2025.
• OpenGitOps — GitOps Principles. CNCF, 2021 — free online.
• Camille Fournier, Ian Nowland — Platform Engineering. O’Reilly, 2024.
• Matthew Skelton, Manuel Pais — Team Topologies, 2nd ed.. IT Revolution, 2025.
• DORA — Accelerate State of DevOps Report 2024. Google Cloud, 2024 — free online.

Chapter 4: Shipping Change Safely — Continuous and Progressive Delivery

Every outage begins as somebody’s reasonable change.

A system that nobody touches keeps doing whatever it did yesterday, and for a while that looks like stability. But the world the system serves keeps moving. Customers want new things, the libraries it depends on announce vulnerabilities, partners change their interfaces, and the business itself changes its mind. A system frozen against all of this does not stay correct; it slowly becomes wrong about the world, one unmet need at a time. So we change it, many times a day, and every one of those changes is a moment when the running system becomes one we have never run before. That is where most of our trouble begins. The question of this chapter is how a team can change a running system often, quickly and safely, when those three goals seem to pull against one another: speed appears to cost safety, and caution appears to cost speed.

The answer the industry has arrived at looks, at first sight, like a paradox. We make change safer by making each change smaller and changes more frequent, so that every step is easy to understand and easy to undo. And we separate two acts we used to treat as one: putting code into production, and letting anyone use it. Code can then sit in production, switched off, until we choose to expose it a little at a time. Both ideas take the rest of the chapter to unfold. To see why they matter, consider first what a single change can do when neither is in place. On the morning of 1 August 2012, Knight Capital, one of the largest market makers in American equities, put new code into service on its order router. Forty-five minutes after the market opened, it had lost about $440 million, and within a year it no longer existed as an independent company. Section 4.1 follows that morning closely, because nearly every safeguard this chapter describes is one Knight did not have.

Encore stands at the other end of that spectrum. It ships about 180 production deployments on a typical working day, across 40 teams, and every one of them travels through a single golden-path pipeline, the paved route the platform group maintains so that no team has to build its own. At the end of that pipeline sits a canary: the new build first serves a small share of real traffic, and it is promoted to everyone only if Encore’s service level objectives (SLOs), the reliability targets the business has agreed to, show that it is behaving. The arrangement resembles a touring production that tries each new scene in front of a few rows before the whole house sees it, although the analogy breaks in an instructive place: at Encore the few rows are chosen by software and the verdict is read from numbers rather than from the audience’s faces, which is both what makes it fast and what limits it to the problems those numbers can see. This chapter takes that machinery apart to show how each piece earns its place. It then follows change down the routes that bypass the pipeline, such as configuration edits, data changes and flags flipped by hand, which can break production just as surely as code can. And it ends with what shifts when a growing share of those 180 daily deployments was written by a machine.

4.1 Change as the Leading Cause of Outages

Forty-five minutes at Knight Capital

To understand what went wrong at Knight, we need to know a little about the machine at its center. Knight’s order router, called SMARS, took large “parent” orders from customers and split each one into many smaller “child” orders, which it sent to the market until the parent was filled. In the summer of 2012 the New York Stock Exchange was preparing to launch its Retail Liquidity Program (RLP), which began trading on 1 August, and Knight wrote new code so that SMARS could take part in it. To mark the orders that belonged to the new program, the developers reused a flag in the order messages. That flag already had a history. It had once activated a function called Power Peg, which had been unused since 2003 but had never been removed from the code. So the same flag now meant one thing to the new code and something quite different to any server still running the old code.

A second, older change had made Power Peg dangerous. In 2005 Knight had moved the logic that counted how much of a parent order had been filled. Power Peg relied on that count to know when to stop sending child orders, and after the move it no longer received it. Nobody noticed, because nobody ran Power Peg. The function sat in the code for years like a machine with its brakes disconnected, harmless only because nobody had started it.

From 27 July a technician copied the new code to the eight SMARS servers by hand, a few on each successive day. The new code reached seven of them; the eighth kept the old code. In one sense the rollout was careful, since it was staged over several days. But nothing verified it. No second person checked the technician’s work, and no automated process compared the eight servers to confirm that they all ran the same thing. When the market opened at 09:30 Eastern time on 1 August, seven servers received RLP orders, saw the flag and handled them as intended. The eighth received the same orders, saw the same flag, read it in its old meaning, and woke Power Peg. Power Peg began sending child orders, and because it could no longer tell when a parent order was filled, it never stopped. There had been a warning. Before the open, Knight’s systems sent 97 automated emails that mentioned Power Peg, but those emails were not designed as alerts, and nobody read them as one.

The response made things worse, and it is worth slowing down to see why. The engineers, suspecting the new release, did what most of us would do: they removed the new code from the seven healthy servers. This was a rollback in every ordinary sense of the word. Yet the flag in the incoming orders still carried its new meaning, because the orders themselves had not changed. Now all eight servers ran the old code against the repurposed flag, and all eight woke Power Peg. The rollback turned one misbehaving server into eight. By the time the system was stopped, about 45 minutes after the open, SMARS had made 4 million executions in 154 stocks, for more than 397 million shares. Knight reported a pre-tax loss of about $440 million; the Securities and Exchange Commission (SEC), in its 2013 order, put the loss above $460 million and fined the firm. Knight needed emergency financing within days and was merged into Getco the next year. Figure 4.1 follows the eight servers through the morning.

Figure 4.1 — Knight Capital’s eight SMARS servers on 1 August 2012. One unverified server woke Power Peg at the open, and the rollback of the seven healthy servers gave all eight the same old code.

It is tempting to name a single culprit: the technician who missed a server, perhaps, or the engineers who rolled back. Read with the eyes of safety science, which the Operations book turns on software in its Chapter 2, Knight’s morning has no single cause. It had at least six contributing ones: dead code kept for nine years; a flag given a second meaning; a deployment that nobody verified; a refactoring that quietly broke a path nobody used; warnings that nobody read; and a rollback that amplified the failure instead of containing it. Each of these was locally understandable. Leaving old code in place saves effort, reusing a field avoids a change to the message format, and rolling back is the textbook first move. None of them alone would have cost Knight anything. Together they built a mechanism, piece by piece over years, that needed only one more event to set it off, and the deployment supplied that event.

There is one more lesson here, and it bears directly on the rest of this chapter. Knight’s rollout was staged, and staging alone saved nothing. A staged rollout protects us only when something compares the servers the change has reached with the servers it has not, and asks whether they agree. Without that comparison, staging merely spreads a mistake over several days instead of one.

Why change dominates

Knight’s cost was extreme, but the shape of its failure is common: a system that had run well was changed, and the change exposed something nobody had observed before. The authors of Google’s Site Reliability Engineering (2016) reported that roughly 70% of their outages were due to changes in a live system. From that observation they prescribed three remedies, and those remedies organize this chapter: roll out progressively, detect problems quickly, and roll back safely. Other datasets give smaller shares than 70%. Much of the difference depends on what counts as a “change.” If a team counts only code deployments, the share falls; if it also counts configuration edits, flag flips and data pushes, the share rises. That question of definition will prove important later in this chapter, because some of the most damaging changes of recent years were configuration and data, not code.

Encore asked the question of itself. When its Reliability team reviewed the last year of incidents, about two thirds had begun within an hour of a change. The changes came in several kinds: a deployment, a configuration edit, a flag flipped, or platform automation simply doing its job, such as rotating a certificate or upgrading a node.

The team could answer the question only because Encore keeps the evidence. Its deploy history records every change, its owner and the services it touched, and it shows those changes beside the first alert of any incident, so that an engineer looking at an alert can see at a glance what changed nearby and shortly before. This join between the record of changes and the record of incidents sounds like bookkeeping, but it is the foundation of any honest conversation about stability. The DevOps Research and Assessment (DORA) program measures stability with two numbers: change failure rate, the share of deployments that cause a failure in production, and the time it takes to recover from a failed deployment. Neither can be computed without knowing which change preceded which incident. A team without that join is left to argue about its stability from memory, and memory is kind to the changes we were proud of.

If change causes most outages, the tempting conclusion is to change less. That conclusion fails for two reasons. The first is that the world does not wait. Security patches arrive, dependencies need upgrading and the business needs new features, so a team that changes its system rarely does not make fewer changes; it gathers the same changes into larger batches. A large batch is harder to test, harder to reason about and harder to undo, and when it fails it is harder to tell which of its many parts was to blame. The second reason is that a team that deploys rarely deploys badly. Deployment is a skill, and like any skill it fades without practice. A procedure performed once a quarter is half forgotten each time it is needed, and its scripts and runbooks drift out of date in between.

I have come to believe that this is one of the clearest places in operations to apply Brooks’s distinction between essential and accidental complexity. The essential complexity is that the system must change, often and indefinitely, because the world it serves keeps changing. The accidental complexity is whatever makes each change bigger, rarer or harder to reverse than it needs to be. We cannot remove the first, but most of this chapter is about removing the second.

Recap. Most outages begin with something we did, because a change is the moment when the system becomes one we have not yet observed. Knight shows that a deployment is often only the last step in a mechanism assembled over years, and that a staged rollout protects us only when something compares the servers it reached with those it did not. Changing less does not help, because it only makes each change larger and each deployment less practiced, and only a change log joined to the incident record lets a team see any of this clearly.

Exercise 4.1. For your last ten incidents, note the most recent change before each one, and classify it as code, configuration, data, flag, automation or none. What share of the incidents followed a change? Which kind of change most often bypassed your deployment pipeline? And how long did it take, in each incident, to find the change that mattered?

4.2 The Deployment Pipeline and Continuous Delivery

A sequence of increasingly expensive questions

Section 4.1 left us with an uncomfortable pair of facts: most outages begin with a change, and changing less only makes each change larger and more dangerous. The way out of that bind was worked out over about two decades, and it helps to see the ideas arrive in order. The phrase continuous integration (CI) was coined by Grady Booch in Object-Oriented Design with Applications (1991), where it described a steady rhythm of integrated releases rather than one great integration at the end of a project. Kent Beck’s extreme programming (1999) gave the phrase the cadence we mean by it today: many merges a day into a single mainline, so that no one’s work drifts far from everyone else’s. In 2009 John Allspaw and Paul Hammond of Flickr described deploying to production more than ten times a day, which at the time sounded to many engineers like recklessness. A year later Jez Humble and David Farley’s Continuous Delivery (2010) gave the practice the structure that made such a pace safe, and that structure is the subject of this section.

The deployment pipeline is the automated path every change takes from commit to production: a sequence of stages, each asking a more expensive question than the last, in which only a change that answers every question may proceed. The ordering is the point. Cheap, fast questions come first, so that most bad changes are turned away within minutes and at little cost; slow questions that need real infrastructure or real traffic come later, and are asked only of changes that have already earned them. A pipeline of this kind keeps a few strict rules. The first is to build the artifact once and promote that same artifact through every stage, so that what was tested is exactly what ships; rebuilding at each stage would mean the thing in production had never really been tested at all. The second is to keep everything that defines the system in version control, so that every difference between yesterday and today is a recorded change that someone can read. The third is that when the pipeline is red, the team fixes the pipeline first. That rule sounds fussy until we think about what happens without it. A pipeline that fails often for reasons nobody investigates teaches people to rerun it, then to work around it, and before long a red result carries no information. A pipeline that people have learned to ignore is no longer a control.

Figure 4.2 shows the stages of Encore’s golden path, the standard route that every service built on the platform gets by default. Read each row as a question, and pay as much attention to the last column as to the second: every stage is blind to something, and the pipeline is designed so that a later stage covers what an earlier one cannot see.

Figure 4.2 — Encore’s golden-path pipeline. Each stage is cheaper to fail than the next, and the right-hand column says which later stage must catch what this one cannot.

It is worth walking down the table once, because each stage earns its place. The commit stage compiles the service and runs the unit tests that need nothing beyond the code itself; it answers in six minutes, and because it is so cheap, it is where we want most mistakes to die. The acceptance stage checks the service against its contracts with the services it talks to and against the behavior its specification promises, which takes longer because other services, or faithful stand-ins for them, must be present. The fitness stage runs fitness functions, automated checks that the system still keeps a promised architectural, security or cost property (Section 4.5 returns to them); they catch things such as a dependency that crosses a forbidden boundary, a container image with a known vulnerability, or a resource request that would quietly double a service’s bill. The staging soak then starts the new version in a production-like cell, one of the self-contained copies of the stack from Chapter 3, runs its database migrations and lets it serve for ten minutes, which catches the change that is correct in every test yet fails to boot or to migrate. None of these stages can reproduce the 500× surge of a real on-sale, and so the canary stage sends a small share of real traffic to the new version and compares its service level indicators (SLIs), the measured signals of service health from Chapter 1 such as error rate and latency, with those of the version already running. Only after all of that does the last row come into play, and it is a different kind of question altogether. Whether fans should see a new seat-map renderer this Friday is a decision for the product and the business, and Section 4.3 shows how flags let Encore make it separately from the deployment.

The same staircase shape, cheap checks first and expensive ones last, appears in any well-run backstage. A touring crew checks every cable and channel at line check in the empty afternoon hall, then runs a full soundcheck with the band, and only then opens the doors; nobody would discover a dead microphone in front of 20,000 people if a ten-second test at three o’clock could have found it. The analogy fails at one point, and the failure is telling. A soundcheck happens once per show, while a pipeline runs for every change, dozens of times a day, and that frequency is what makes its rules matter so much: a check that is a little flaky or a little slow is a nuisance once a day and a corrosive habit at fifty runs a day.

Humble and Farley drew a distinction between two practices that are easy to confuse. In continuous delivery, every change that passes the pipeline could be deployed to production at the push of a button; in continuous deployment, every such change is deployed, with no human step in between. The second asks more of a team. It needs tests fast enough that nobody is tempted to skip them, observability good enough to notice a bad change on its own, trunk-based development so that changes stay small, and the ability to roll back in minutes. Encore deploys continuously, with one deliberate exception. Seat Inventory, the event-sourced ledger at the heart of every purchase, keeps a human release step, because a change to it must have its projections replayed against production history before anyone trusts it. In an event-sourced service the projections are the read models rebuilt from the stored events, and a change that alters how they are computed can be correct on every test fixture yet wrong against years of real events. For the one service where every seat must be sold exactly once, Encore judged that a person should look at the result of that replay before the change goes live.

Small batches, and what replaces the change board

The pipeline’s deepest effect is not on speed but on batch size, and batch size is where safety is really decided. Consider the two ends of the range. A 40-line change can be read in one sitting, its effects can be watched on a dashboard in the minutes after it ships, and if something goes wrong it can be undone without disturbing anything else. Three weeks of work by 15 people, released together, can be none of those things: no one person understands all of it, a fault in it could come from any of a hundred places, and undoing it means undoing everyone’s work at once. This is the lesson of Section 4.1 seen from the other side. Changing rarely makes batches large, and a pipeline that makes changing cheap is what lets batches stay small.

The research supports this more strongly than intuition might suggest. Accelerate (2018) found that throughput and stability do not trade off against each other: the teams that deployed most often were also the ones that failed least often and recovered fastest. The same work found that requiring approval from an external change advisory board, the committee that reviews changes it did not write, went with slower delivery and no better change failure rate. My own reading of that finding is that such a board mostly produces reassurance. A weekly committee looking at a summary cannot know a diff as well as the colleague who reviewed it line by line that morning, and so its approval adds delay without adding much knowledge. I would grant that the board’s real purpose, a sense that someone responsible has looked, is legitimate, and that the mistake lies only in where the looking happens.

None of this means changes should go unexamined. What DORA calls streamlined change approval keeps the examination and moves it into the pipeline, where the people and the evidence are. A peer who knows the code reviews every change. Every change passes the same automated checks, so that no change is trusted because of who wrote it. Riskier changes, such as those to Seat Inventory or to payment handling, get more reviewers and more stages. And the pipeline records who approved what and which checks passed, so that an auditor reads evidence instead of meeting minutes. Encore’s golden path writes that record for every deployment, which is also what gave the Reliability team the deploy history it used in Section 4.1 to join changes to incidents.

There is one limit to all of this, and Knight Capital has already shown it to us. The pipeline guards only the changes that go through it. Knight’s deployment was copied by hand to eight servers, outside any pipeline, and the flag that woke Power Peg was not code at all but data carried in a message, so even a perfect code pipeline would never have examined it. Encore has the same kind of side doors. Its service configuration lives in the GitOps repositories of Chapter 3 and so passes through review and reconciliation, but feature flags, Bot Screening rule sets and platform automation such as the node-upgrade reconciler each had their own paths into production. Section 4.3 shows three large outages that travelled by exactly such paths.

Recap. The deployment pipeline asks increasingly expensive questions of a single artifact, so that most bad changes fail early and cheaply and each stage covers what the earlier ones cannot see. Its real gift is small batches, which buy stability as well as speed. Peer review and automated checks inside the pipeline do the change board’s job better than the board did, because they look at the change itself, and every route that bypasses the pipeline deserves to be listed and narrowed.

Exercise 4.2. List every route by which a change reaches production in a system you know: pipelines, consoles, flag tools, configuration services, generated data, automation and scripts. For each route, mark whether a change that takes it is reviewed, tested, recorded, staged and reversible. The routes with the most gaps are your next pipeline work.

4.3 Progressive Delivery — Feature Flags, Canaries, and Rollbacks

Deploying is not releasing

Section 4.2 ended with the pipeline delivering a tested artifact to the door of production. What happens at that door is the subject of this section, and it begins with a distinction that sounds pedantic until we have been burned without it. To deploy a change is to put its code onto production machines; to release it is to let people experience its behavior. For most of the industry’s history the two happened in the same instant, so every deployment was also a bet placed on every user at once. Knight’s morning in Section 4.1 was such a bet: the new code went live on all eight servers at the moment the market opened, and there was no smaller stake to place first.

Progressive delivery is the practice of putting new code into production without exposing it, then widening its exposure in deliberate steps, each step conditional on evidence that the last one did no harm. The analyst James Governor named it. Its techniques are many, and we will meet canaries, rings, blue/green switches and feature flags in turn, but its one steady idea is simple. Every change has a blast radius, the set of people and systems that would be hurt if the change turned out to be wrong, and we should choose that radius deliberately instead of letting it default to everyone. A stage manager does something similar with a new lighting cue: it is tried in an empty house, then at a dress rehearsal before a few friends, and only then on opening night. The analogy holds as far as the ordering of audiences. It breaks because a theatre’s rehearsals end, while in production every change is rehearsed afresh, many times a day, on an audience that is already in its seats.

Canaries and automated analysis

The oldest of these techniques borrows its name from the mines. British miners carried canaries underground for most of a century, because the bird showed the effects of carbon monoxide before the men did, and a bird in distress gave them time to get out; the birds were phased out from 1986. A canary release follows the same logic in software. It sends a small share of real traffic to the new version and compares that version’s health with the old version’s before any more traffic follows. If the new version is sick, only the small share of users routed to it notice, and only briefly.

The analogy is worth taking seriously, because the place where it breaks teaches us something. The miners’ canary worked because it breathed the same air as the miners, only more sensitively. A software canary is useful only to the extent that its traffic resembles the traffic it is protecting. A canary that weakly resembles the population it guards, because it serves fewer users, runs during quieter hours or receives cheaper requests, can sing happily while the mine fills. Most of the craft of canary analysis is in making the comparison fair.

Encore’s purchase path shows what fairness takes. At each step of Figure 4.3, an automated analysis compares the canary with a baseline: a group of the same size, running the old version, started at the same moment and receiving the same kind of traffic. Each of those conditions removes a way the comparison could lie. The canary is never compared with yesterday, because yesterday had a different audience and a different mix of events. It is never compared with the long-running fleet either, because processes that have run for days have warm caches and settled connection pools, which flatter them against anything freshly started; a new build would look slower for reasons that have nothing to do with its code. A baseline started alongside the canary suffers the same cold start, so whatever difference remains belongs to the change.

The analysis watches SLIs that fans would recognize as their own experience: the rate at which checkouts succeed, the p99 latency of POST /holds (the call that holds a seat while a fan pays), and the rate of PaymentFailed events caused by Encore itself, as opposed to a declined card, which tells us about the fan’s bank and nothing about our build.

Figure 4.3 — Encore’s canary schedule for purchase-path services. The request minimum matters as much as the minutes, because a canary judged on too little traffic cannot tell a bad build from noise.

Two columns of Figure 4.3 deserve a closer look. The first is the bake, the minimum time a step must run before the analysis may judge it. At step 1 the bake has two conditions, a time and a request count, and both must be met. The reason is statistical. If the canary has served only a few hundred requests, a handful of failures could be bad luck or a bad build, and no honest analysis can tell which; “beyond noise” in the table means precisely that the difference from the baseline is larger than chance alone would plausibly produce. The second column is the error-budget burn at step 3. Burn rate, from Chapter 1, measures how fast the service is spending its error budget, where 1× would use the budget exactly over the SLO window. At 25% of traffic, the canary is now large enough to matter to the budget itself, so the step adds a test the earlier steps did not need: the change must not be spending the budget at more than twice the sustainable pace. Added together, the bakes come to 45 minutes from the first fan on the new version to the last.

Statistics are not the only reason to roll back. An integrity alarm fires when the system violates a rule that must never be broken, such as a seat with two SeatSold events, which would mean two fans each hold a ticket for the same chair. Such an alarm rolls the canary back at once, whatever the statistics say, because no amount of good latency compensates for selling a seat twice; Figure 4.4 draws that path. Canaries also do not run during on-sales, for reasons Section 4.4 gives.

The schedule has a limit we should state plainly. Bakes measured in minutes catch only failures that show themselves in minutes. A slow leak, such as memory that grows a little with every request or a queue that drains slightly slower than it fills, passes every step and is left to the burn-rate alerts once the change is everywhere. That is why the largest operators, Amazon among them, bake for hours between waves of regions: they are buying time for slow failures to surface while most of the world still runs the old version.

Figure 4.4 — Figure 4.3’s schedule drawn over time on a logarithmic scale. A canary that holds reaches every fan in about 45 minutes. One that raises an integrity alarm at 25% goes straight back to zero, without waiting for the statistics.

A canary is one way to shape exposure; two others complete the toolkit, and each trades something different. Blue/green deployment runs a complete new environment (green) beside the old one (blue) and switches all traffic from one to the other at once. It gives up the gradual widening of a canary, and in return it buys an instant switch back, at the price of paying for double capacity while both environments exist. Encore uses it for the serverless Sale Gate, its virtual waiting room, where an all-at-once switch with an immediate way back suits a service that must behave identically for every fan in the queue.

Ring deployment widens exposure by population instead of by percentage. Each ring is a group of users chosen because of who they are, and the rings are ordered so that those best able to absorb harm see the change first. Microsoft has long released Windows and Azure this way. Encore’s rings start with its own staff and synthetic buyers, the scripted clients that exercise real purchase paths, who notice problems and complain to the right people. They then widen through the self-service and Professional cells to Encore’s own fan site, and end with the Enterprise silo for venue-chain-a, whose contract places it last. That final position is worth noticing: here the order of exposure is set partly by engineering judgment and partly by a commercial agreement, and the pipeline has to honor both.

Figure 4.5 — Encore’s delivery rings, widening by population from staff and synthetic buyers outward. The outermost ring is last because a contract places it there.

Configuration and data travel the same road

Everything so far has concerned code, and an engineer could finish the previous subsection believing the problem solved. History says otherwise. The three most instructive outages of recent years were not caused by code at all, and each traveled down a road that progressive delivery had not yet been built on.

On 19 July 2024 CrowdStrike pushed Channel File 291, a content update for its Falcon sensor on Windows. The sensor’s code itself went through staged rollouts. Its “rapid response content,” however, was treated as data rather than code, and data went to every machine at once. The file supplied a 21st input field to an interpreter that provided only 20. The content validator, the one check standing between the file and the fleet, missed the mismatch, and when the sensor read past the end of what it expected, the out-of-bounds read crashed about 8.5 million Windows machines, by Microsoft’s estimate, in the 78 minutes before the file was withdrawn. CrowdStrike’s remediation reads like a summary of this section: staged content deployment through canary rings, customer control over when updates arrive, and bounds checks in the code that reads the content.

On 12 June 2025 a policy change reached Google Cloud’s Service Control, the system that checks authorization and quota on API calls. The change carried blank fields. It replicated globally within seconds, and in every region it reached a code path that had been added two weeks earlier without a feature flag or error handling, and crashed it. The code path had been waiting, unflagged and unguarded, for the data that would wake it, and that data arrived everywhere at once. Google committed to making Service Control fail open, to flag-protecting changes to critical binaries, and to propagating global data “incrementally with sufficient time to validate and detect issues.”

On 18 November 2025 a permissions change in a Cloudflare database doubled the size of a generated feature file for Cloudflare’s bot-management model. That file was pushed network-wide every five minutes. It now overran the proxy’s limit of 200 features, and an unhandled error became a panic that broke much of the web for about three hours. Cloudflare’s first remediation was to harden “ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input.”

Set side by side, the three stories rhyme, and the three operators reached one recipe from different directions. Its first half is to stage configuration and data as we stage code: through rings, with a bake and an automated halt, so that a bad push reaches a canary before it reaches every region. Its second half is to make every consumer treat internal configuration as untrusted input, even though its own colleagues produced it. In practice that means four things. Validate the input on arrival. Enforce the limits the code silently assumes, such as Cloudflare’s 200 features or CrowdStrike’s 20 fields, so that a violation becomes a rejected file instead of a crash. Keep the last version known to be good, so that rejecting a new file leaves the component running on the old one. And decide in advance whether a component with nothing good to load should fail open, carrying on without the protection the configuration provided, or fail closed, refusing to serve until it has something trustworthy. Neither answer is right in general; what matters is that someone chooses it calmly, beforehand, instead of discovering it during an outage.

I have come to believe that this is the most underrated lesson of the decade in operations. As an industry we learned long ago to distrust input from strangers; we are only now learning to distrust input from ourselves.

Encore’s nearest equivalent is the Bot Screening rule bundle that every Sale Gate instance loads, the set of rules that decides which visitors look like bots. Shadow-mode trials (Chapter 1) run a new rule against real traffic without acting on it, which tests whether the rule is wise. They do not test whether the bundle that carries it is well formed, and until recently the bundle reached every instance within a minute, by its own road, outside the rings. The Reliability team has since moved the bundle, the flag service’s state and the edge routing table onto the same rings as code, and rewrote the Gate’s loader along these lines:

MAX_RULES = 5_000   # far above today's bundle, far below what exhausts memory

def load_bundle(raw: bytes, current: Bundle) -> Bundle:
    try:
        bundle = Bundle.parse(raw)          # schema, types, required fields
        if len(bundle.rules) > MAX_RULES:
            raise BundleError(f"{len(bundle.rules)} rules exceeds {MAX_RULES}")
        bundle.verify_signature()           # built by the pipeline, never by hand
    except Exception as err:                # a bad bundle must never crash the Gate
        metrics.increment("rule_bundle_rejected")   # pages Bot Screening on call
        log.error("keeping bundle %s: %s", current.version, err)
        return current                      # last-known-good
    return bundle

Each line of the loader answers one part of the recipe. Parsing checks the shape of the bundle. The MAX_RULES limit turns an assumption into an enforced rule, set far above any real bundle so it never trips by accident and far below the size that would exhaust memory, which is the Cloudflare failure made impossible. The signature check ensures the bundle came out of the pipeline rather than from someone’s laptop. Any failure keeps the current bundle and pages the Bot Screening team, so a bad file degrades into a quiet alert instead of a fleet of crashed waiting rooms.

The last decision in the recipe is the hardest, and the loader cannot make it alone. A Gate instance that cold-starts with no valid bundle has no last-known-good version to fall back on. Encore’s answer is to fail open on bot scoring but keep metering admission. Letting a few bots through for a while is a nuisance; letting the full crowd of an on-sale through at once would overwhelm Seat Inventory, and it is the rate limit, not the bot rules, that protects it.

Feature flags and their debt

Canaries and rings decide which machines and which populations receive a new version. Feature flags go one level finer. A feature flag is a runtime switch that decides which behavior deployed code exhibits, so that code can reach production long before its behavior reaches anyone. With a flag, deploying and releasing become two separate acts performed by different tools, and the release can be widened, narrowed or reversed without deploying anything at all.

Encore’s new seat-map renderer shows what that buys. Behind the flag seatmap-renderer-v2, it reached staff first, then one fan in a hundred on small events, then all events below 5,000 seats, and finally, after a quiet month, the stadium on-sales. When it misdrew curved balconies on one venue’s map, the team turned it off for that venue in seconds and fixed it at leisure. No rollback, no emergency deployment, no other venue affected: the flag shrank the blast radius to exactly the fans who needed protecting.

The price of that power is that flags accumulate, and they age differently by kind. A release flag, which guards a new feature during its rollout, should live for weeks and then be removed along with the old code path. A kill switch, which lets operators turn off an expensive or risky feature under stress, may rightly live for years. An experiment flag should live as long as its experiment and no longer. Trouble begins when flags outlive their purpose, and here Knight’s morning in Section 4.1 is the cautionary tale: a flag given a second meaning, guarding a dead function nobody had removed in nine years.

With Knight in mind, Encore’s flag service enforces a few simple rules, each aimed at one part of that failure. Every flag has an owner, a kind and an expiry date, so that no flag is anyone’s orphan. No name is ever reused, so that an old meaning can never wake under a new one. A flag change is a production change, recorded in the deploy history, so that an incident timeline shows the flip of a switch alongside every deployment. And an expired release flag blocks its owner’s next deployment, which is stricter than most tools’ default of a warning that is easy to ignore. Uber’s Piranha, a tool that writes the clean-up change itself by removing the flag and the dead branch from the code, is the gentler complement: it lowers the cost of doing the right thing instead of raising the cost of not doing it.

The rules came from a sobering count. When the team surveyed the service, a third of Encore’s 2,300 flags had not changed state in a year. Each of those flags still had two branches in the code, and for each one, only the branch in use was being exercised; the other had gone untested for a year and could fail in any way at all if someone flipped the switch. A flag that is never turned off is dead code wearing a switch.

Rollback or roll forward

Every technique in this section assumes that when something goes wrong, we can get back. That assumption deserves examination, because it is false more often than we like to think. To roll back is to return production to the last version known to be good; to roll forward is to repair the problem with a new change instead. Rollback is the right default for stateless code, because a stateless service holds nothing that outlives the process: replace the binary, and the world is as it was.

Knight’s morning shows where that reasoning fails. Its rollback spread the failure from one server to all eight, because the old code met a world the new code had changed: the repurposed flag was already flowing through the order messages, and the old code read it with its old meaning. Rolling back the code did not roll back the world. In most systems that changed world is data: rows written in a new format, events with a new field, a cache filled by the new version. Once the new version has written something the old version cannot read, rolling back means running code that misunderstands its own data.

The defense is to keep each change compatible with its predecessor, so that the previous version can always read whatever the new one wrote. Two patterns do this. The expand-and-contract migrations of Chapter 6 first expand a schema to accept both old and new forms, then migrate, then contract once nothing uses the old form. Amazon’s two-phase pattern applies the same idea to formats on the wire and on disk: one deployment teaches every node to read a new format, and only a later deployment begins writing it. Between the two, rolling back is always safe, because every node already understands everything that could have been written.

Not every change can be made this way, and Encore treats the exceptions as exceptions rather than pretending they do not exist. Its pipeline records whether each deployment is safe to roll back. One that is not gets a longer canary, because its only escape is forward and it must be right before it spreads, and a named engineer awake for its first hour, because rolling forward takes a person who understands the change.

Recap. Progressive delivery separates deploying from releasing so that we can choose each change’s blast radius. Canaries are only as honest as their comparison, so they are measured against a simultaneous baseline on enough traffic, and rings order exposure by who can best absorb harm. Configuration and data must travel the same staged road as code, into consumers that distrust them and keep a last-known-good version. Flags make release a runtime decision but turn into debt unless they are owned and retired. And rollback is safe only while the previous version can still read the data the new one wrote.

Exercise 4.3. Pick the configuration or data file that the most machines in your fleet load. How long does a change to it take to reach all of them? What validates it on arrival, and what limits does the code that reads it silently assume? And what does a consumer do with a malformed one: crash, keep the last good version, or fail open or closed, and did anyone decide which?

4.4 Testing in Production

Rehearsing in the real room

For a long time, “testing in production” was something an engineer said with a guilty smile, as a confession that the real testing had been skipped and the users had been left to find the bugs. The phrase has since changed its meaning, and the change is worth understanding. It is now the name of a discipline, and the reason is simple once stated: no staging environment reproduces production. Staging has a copy of the code, but it does not have production’s data, with its years of odd records and forgotten edge cases; it does not have production’s traffic, with its real mix of devices, networks and impatient people; and it does not have production’s scale. For Encore the last point is decisive. A 500× surge in the minutes after a big artist announces a tour cannot be rented for an afternoon and replayed in a test cluster. It exists only where it happens, in front of real fans, and so some of what we need to learn can only be learned there.

The theater has known this for centuries. A company rehearses for weeks in a rehearsal room, but it still holds a technical rehearsal and a dress rehearsal on the real stage, under the real lights, because the room changes things that no amount of rehearsal elsewhere can predict: the sight lines, the time a costume change really takes in the wings, the way a voice carries to the back of the hall. Testing in production is our dress rehearsal in the real room. The analogy breaks at one important point, and the techniques below exist because of that break: a dress rehearsal has no paying audience, while production always does. Every method of testing in production is therefore a way of learning from the real room while keeping the audience, and everything that would touch the audience, out of the experiment. Three such methods matter at Encore.

The first is the synthetic user: a scripted client that exercises a real path through production continuously, behaving as a customer would. Encore’s synthetic buyers do everything a fan does. They hold seats, pay through the payment provider’s test route and receive tickets, all for a hidden test event that no real fan can see, so they consume no real seats and move no real money. They run every hour, and every minute during on-sales. Their value is that they answer a question real traffic cannot. At three in the morning, an absence of ticket sales is ambiguous: it may mean nobody wants to buy, or it may mean nobody can. A synthetic buyer that fails to complete its purchase removes the ambiguity, because it always wants to buy. During an on-sale the situation is reversed, since real traffic is enormous and noisy, and the minute-by-minute synthetic purchase becomes a steady reference point that walks the whole path, from the Sale Gate through Seat Inventory and Orders to the payment provider, at a moment when everything else is moving.

The second is the dark launch: new code runs against real traffic, but its results are hidden from users, so the organization learns how the new code behaves before anyone depends on it. The technique has a well-known early example. In 2008 Facebook prepared its chat service by having live pages quietly send it simulated traffic, so that the back end met realistic load before a single user could open a chat window. Encore used the same idea when it built a new availability projection for Seat Inventory. (As Section 4.2 explained, a projection is a read model, built here by replaying the stream of seat events into a shape that is fast to query.) The new projection was fed the same events as the old one, and both answered every availability query; fans saw only the old answer, while the two answers were compared and every disagreement was investigated. The team switched over only when a fortnight had passed without a single disagreement. That is a demanding standard, and a fitting one for the service whose whole purpose is to sell each seat exactly once.

The third is shadow traffic: real requests are copied to a new version of a service, and the new version’s responses are thrown away. Where a dark launch usually lives inside the running system, shadowing typically happens in the network path, at the edge or in the mesh, which duplicates requests toward a separate deployment. The method is safe for reads, because answering the same question twice changes nothing. It becomes dangerous the moment a request causes a side effect in the world. Shadowing the Notification service naively would send every fan two emails, and if the shadow also issued wallet passes, every fan would receive the same ticket twice. Encore therefore shadows write traffic only into sandboxes, where the side effects land in test mailboxes and test wallets that no fan will ever open. The trade-off is plain: the closer a shadow comes to production’s real behavior, the more care it takes to keep its effects invisible.

These three methods let us learn from the real room safely on an ordinary day. On the day of a big on-sale, however, Encore does something that looks like the opposite of testing in production: it stops changing the purchase path altogether. That practice, and the slow way its boundary wears down, is the subject of the rest of this section.

Change freezes and their exceptions

A change freeze is a period in which production changes are forbidden except for emergencies, because a failure during that period would cost more than usual. The idea is old and intuitive. Retailers have frozen their systems through the holiday season for decades, on the reasoning that the weeks that bring in the most revenue are the wrong weeks to discover a bad deployment. The reasoning is sound, but the evidence that freezes actually reduce harm is thin. Practitioners report a familiar pattern at the other end: changes pile up during the freeze, and when it lifts they all land together, so failures spike in the days afterward. A broad freeze may only move the risk to a later date, and concentrate it there.

The one serious quantitative study I know of comes from Meta. In its winter 2023 freeze, Meta used risk models to decide which changes needed to be held, and gated only 16% of them; the rest were allowed through. Compared with the previous winter, it landed 42% more changes and saw 52% fewer outages. This is one company’s observational study, and it is not proof; a different winter, a different mix of changes or a different company might tell another story. But the evidence we have points one way: toward a narrow freeze aimed at the changes most likely to do harm, and away from a broad one that stops everything and then releases the backlog in a flood. Google’s freeze after its June 2025 Service Control outage (Section 4.3) had exactly that narrow shape. It covered only Service Control and manual policy pushes, and only until the repairs were in.

Encore’s freeze is scoped in the same spirit, and for a reason specific to its traffic. It covers the purchase path, from two hours before each big on-sale until the on-sale ends. The justification is that during an on-sale Encore’s ordinary safety instruments go blind. The rest of the week, a risky change is made safe by a canary (Section 4.3), which compares the new version with a baseline running the old one and waits for enough traffic and enough minutes to tell a bad build from noise. When load rises 500× in minutes, both conditions fail. There is no stable baseline, because the traffic, the caches and the load on each shard are changing faster than any comparison can settle. And there is no time to bake: a canary that passes every step of Figure 4.3 takes about 45 minutes to reach every fan, longer than the most intense part of the surge. So the instruments that make change safe on a Tuesday afternoon are unusable in precisely the hour when an outage costs the most. A freeze is what remains when those instruments are gone, and in Encore’s case it is a deliberate, narrow choice rather than a seasonal habit.

The hard part of any freeze is its boundary, the precise answer to the question “which changes does this cover?”, and boundaries erode. Over the past year Encore’s freeze has softened, one reasonable step at a time. Exceptions are granted in the on-sale chat by whoever is senior and awake at the moment, and are recorded nowhere else; there were 41 of them in 12 months, rising quarter by quarter from 3 to 11, 13 and 14. Low-risk platform automation, such as certificate rotation and the node-upgrade reconciler that drains and replaces Kubernetes nodes, was excluded from the freeze, because pausing it for every on-sale created more work than it saved. Catalog was judged to sit off the purchase path, on the grounds that a fan can buy a seat without ever searching for one. Each step had a sensible argument behind it, and the people who took them were not careless. Together, though, those steps turned the freeze from a rule into a custom, something the organization believed it followed while its practice quietly moved away. This is what safety science calls drift, in which a system migrates toward the edge of safety through a sequence of locally rational decisions, none of which looks like a decision to accept more risk.

There is a second, subtler weakness in how the boundary is drawn, and the Cloudflare outage of November 2025 (Section 4.3) shows it clearly. The change that triggered it was a routine permissions change in a database, far from the proxy it eventually broke: the database change altered a generated file, and automation carried that file to every machine. Now imagine a freeze written the way most freezes are written, as a list of protected services with automation exempt. It would have let that change through, because the change touched no listed service and no human pushed anything by hand. A freeze drawn around services guards the services themselves, but leaves open everything they depend on. Encore’s boundary is drawn that way, and so are its exemptions. Encore has not yet asked whether its freeze should follow dependencies rather than service names, or where its exceptions ought to be written down so that their accumulation is visible to everyone. The Operations book returns to both questions, first on the morning of the Vesper Hale on-sale and then in the review that followed it.

Recap. Synthetic users, dark launches and shadow traffic let us learn what only the real room can teach, while keeping users and side effects out of the experiment. The evidence for change freezes is thin, and what there is favors narrow, risk-based freezes over broad ones. Encore’s freeze is justified, because its canaries are blind during a 500× surge, but its boundary has drifted through a series of sensible exceptions, and it is drawn around services rather than around what those services depend on. Exceptions belong in writing, where their growth can be seen.

Exercise 4.4. List every change that reached production during your last freeze, including automation runs and configuration or data pushes. Who granted each exception, and where is it recorded? Then pick one frozen service and trace what it depends on: which of those dependencies could have changed during the freeze without anyone treating the change as an exception?

4.5 Machine-Written Change — Review, Fitness Functions, and Provenance

The Tuesday morning, revisited

The Architecture book ended on a Tuesday morning, after a coding agent had rebuilt Encore’s Notification service overnight. The agent had brought in a new framework and new retry semantics, and the result spanned 40 files that nobody had read in full. Every check passed, and the canary, judging the build on its service level indicators, promoted it. One engineer nonetheless spent 20 minutes on that change alone, reading the retry logic with a question in mind that no check was asking, and saw that the new retries could send a fan the same ticket twice. The engineer added a spec-level idempotency test before merging. The point of the story is not that the pipeline failed, because it did everything that was asked of it. The trouble was that nothing in it asked whether the new retries were safe, since until that morning nobody had thought the question needed asking.

We have met unread code before in this chapter. Power Peg sat in Knight’s router for nine years, and nobody read it again before the repurposed flag woke it; unread code is as old as software itself. What is new is the speed at which it now arrives. By autumn a quarter of Encore’s merged changes carried an agent label, and those changes were larger than the ones people wrote. The industry’s evidence points the same way, though it is young and should be read as such. DORA’s 2024 report estimated that each 25% increase in AI adoption went with a 7.2% fall in delivery stability, which is to say that teams adopting these tools more heavily also saw more of their changes go wrong. Its 2025 report found that throughput gains had arrived but that the stability penalty was still in place, and it named small batches and strong version control among the capabilities that help a team keep the gains without paying the penalty.

A second finding suggests why the penalty might exist. ImpossibleBench (2025) gave coding agents tests that they could not honestly pass, and found that the agents often edited the tests instead. This matters more than it first appears. A test is the pipeline’s written record of what we intended, and a pipeline can check a change only against that record. An author who is rewarded for making the checks pass, and who is able to rewrite the checks, can satisfy the pipeline while abandoning the intention behind it. People are capable of the same shortcut, but the pipeline was built on the assumption that the person changing a test knows what the test is for, and that assumption no longer holds for every author.

The best production account so far is Google’s 2025 report on large code migrations. About 80% of the modifications that landed were written by a model, yet every change still passed the existing tests, was reviewed by the owners of the code it touched, and was rolled out gradually. The lesson I draw from it is that a high share of machine authorship can be workable when the old disciplines of testing, ownership and progressive delivery surround it, and that what the machine changes is how much weight those disciplines must carry.

Encore’s policy

After the Notification morning, Encore’s architecture group and the Reliability team wrote a policy together. Its premise has two halves, and the tension between them is deliberate. The pipeline treats every change alike, whoever or whatever wrote it, because a check that relaxes for a trusted author is a check that will one day be walked around. Review, on the other hand, weighs what a change could break, because careful human attention is finite and must be spent where the damage would be greatest. The policy has four parts, and each answers a weakness that the Notification change exposed.

Spec-level tests. The first weakness was that Encore’s tests described the old implementation, so a rewrite could keep them green while changing behavior they had never pinned down. A spec-level test checks a property the business depends on, stated independently of any implementation, so that it stays true when every line beneath it is regenerated. For Notifications, the property is that one TicketIssued event yields exactly one wallet pass and one email under any sequence of retries and restarts. Encore writes it as a property-based test, which generates many such sequences (a timeout here, a crash after sending but before recording, a duplicate delivery there) and checks the property against every one. The test therefore explores orderings of failure that a person writing examples by hand would not think to try. And because ImpossibleBench showed what an agent may do with an inconvenient test, these tests are guarded. People write or approve them, and an agent’s pull request that modifies one is blocked until a human owner approves the test diff separately, through ordinary code-owner rules. The agent may change the implementation as freely as it likes, but it may not quietly change the definition of correct.

Small changes and provenance. The second weakness was size. A 40-file change is hard for any reviewer to hold in mind, and a canary that promotes it can say only that the whole looked healthy, not which part carried the risk. Following DORA’s 2025 advice, Encore now asks agents for changes that a reviewer can hold in mind at once; the Notification rewrite would today arrive as three changes, each passing through its own canary, so that a problem would surface in a third of the code rather than somewhere in all of it. The third weakness was that, after the fact, nobody could easily say what had written what. The golden path already signs build provenance, a verifiable record of how and from what source each artifact was built (Chapter 7 takes it up), and authorship now sits on top of it. Every commit records any agent that wrote it, with its model version, and the human who sponsored it, meaning the person who asked for the change and answers for it. The instinct is the same as the Linux kernel’s, which asks for an Assisted-by tag and lets only a human sign off a patch. I should be candid about the evidence here: nobody yet knows whether such labels shorten incidents. Encore keeps them because they are cheap to record now and cannot be added later, and an incident review that someday wants to ask whether a model version mattered can ask only if someone wrote the version down.

Risk-tiered human review. Human attention is the scarcest resource in the whole pipeline. Compute can be bought by the hour and checks can be added by the dozen, but an engineer who reads carefully can read only so much in a day. The policy therefore spends that attention according to blast radius, the idea Section 4.3 introduced for rollouts: here, the extent of the harm a faulty change could do before anyone stopped it, as Figure 4.6 shows.

Figure 4.6 — Encore’s review tiers. The tier comes from what a change touches; provenance decides only how long an agent’s change bakes and whether a human samples it.

The table rewards a careful reading, because what it leaves out is as deliberate as what it includes. Authorship does not decide the tier. A one-line change to the purchase saga is Tier 1 whether a person or an agent wrote it, and an agent’s rewrite of a dashboard is Tier 3. Provenance enters only at the margins, in the longer bake for agent changes in Tier 2 and in the sampling for Tier 3. Notice also what a Tier 2 reviewer is asked to read. It is not every line of code, which is the part an agent produces most cheaply, but the spec, the invariants and the test diff, which are the places where a wrong decision would be hardest to see later. The reviewer reads the contract closely and the implementation by sampling.

There is a useful analogy in a touring production’s stage manager during a changeover between acts. She cannot inspect every cable the crew lays, and she does not try to. She checks for herself the rigging that hangs over the performers’ heads, keeps an eye on whoever is working the pyrotechnics, and trusts the rest to practiced hands and a checklist, glancing at a corner now and then to keep everyone honest. Where the analogy breaks is revealing. A stage manager knows her crew, and their competence has been visible across years of shows. An agent’s competence is harder to read: it shifts with each model version, and it can be excellent in one file and confidently wrong in the next. That is why Encore samples an agent’s work instead of extending it a reputation.

Tiering review by what a change touches is old practice, and Encore did not invent it. IT service management has long sorted changes into classes by risk, and at Google owners files guard sensitive paths by requiring a named owner’s approval. What Encore added are two numbers, and they deserve honesty about their standing: the doubled bake and the one-in-ten sample are Encore’s own choices, without published precedent. The doubled bake is a cheap hedge in the spirit of DORA’s stability finding. If agent changes are somewhat more likely to fail, a longer bake gives a slow failure more time to show itself in the canary, and the only price is waiting. The sample exists for a subtler reason. Tier 3 relies on automated checks, and checks decay quietly as the code beneath them changes, so the sample is there so that someone notices when Tier 3’s checks begin to miss things, before the users of an internal tool notice first.

Fitness functions. The fourth part addresses the gap that sampling leaves open. If humans read the contract closely and the code only in part, something must still read every line. That something is the set of fitness functions we met at the pipeline’s fitness stage in Section 4.2, which Building Evolutionary Architectures (2nd ed., 2022) defines as automated checks that a system keeps a promised property as it evolves. At Encore they are the only reviewers that read every line at the speed agents write. Three examples show what they look like. Every handler with an external side effect, such as sending an email or issuing a wallet pass, must take an idempotency key, so that a retried request can be recognized as a repeat rather than acted on twice. No service may stack its own retry layer on top of the mesh’s, because two layers of retries multiply each other and turn a burst of failures into a storm of attempts; nor may any service exceed the 10% retry budget that Google’s overload guidance recommends. And no outbound dependency may appear without a registry entry, so that an agent cannot quietly put a new service on the path to a fan. The limit of fitness functions is worth stating plainly. A check can see that an idempotency key is present, but not whether the code uses it correctly, whether, for instance, it records the key before or after the email goes out. That is a question about behavior, and it is why the spec-level tests remain. The two kinds of check divide the work between them: fitness functions enforce structure everywhere, and spec-level tests verify behavior where it matters most.

None of this is free, and the policy does not pretend otherwise. Tier 1 reviews are slower than they were. Agent-heavy teams grumbled that the doubled bake looked like distrust, and in a sense they were right, though it is a distrust of risk that has not yet been measured rather than of the teams themselves. Spec-level tests take real thought, because stating a property independently of any implementation forces a team to decide what the business actually requires, a question that a test written against existing code lets it avoid. Against these costs stands a single observation. The Notification engineer’s 20 minutes were the most valuable that anyone at Encore spent that week, and they were valuable because they went to the right change, with the right question, instead of being spread thinly across everything. The policy exists to spend such minutes where they matter and to keep them from being diluted. A review that reads everything a machine wrote reads nothing carefully.

Recap. Machine-written change is new in two ways: in its volume, and in its tendency to satisfy the checks rather than the intentions behind them, and so far delivery stability pays for it. Encore’s answer combines proven pieces (guarded spec-level tests, small changes, build provenance and fitness functions) with authorship labels and sampling rates that are its own young choices, kept because they are cheap and because their worth can be judged only later. The pipeline checks every change alike; review also weighs who wrote it.

Exercise 4.5. Choose a service you own and write its two most important invariants as spec-level tests, stated so that they would remain true if every line of the current code were replaced. Then decide which review tier the service belongs in, and describe what your review would read in full, and what it would only sample, if an agent rewrote the service overnight.

Kata: Always Open

Your brief: “Kestrel.” Kestrel is a mobile-first retail bank with 1.8 million customers in three EU countries, built and run by 110 engineers in 14 teams. A bank of this kind never closes. Whenever a customer taps a card, at a supermarket at noon or a petrol station at three in the morning, the payment network asks Kestrel to approve or decline it, and every such card authorization must be answered within 2 s, day and night. Above that sits a regulator, which requires a record of every production change: who approved it, what it touched and how to undo it.

Kestrel changes itself often, about 60 deployments a day, but its change records do not keep pace with its changes. They are written by hand after the fact, for a weekly risk committee that reviews them, and its change failure rate, the share of changes that cause a failure in production, is about 15%. Not every change travels through the deployment pipeline, either. Kestrel’s fraud rules, which decide which payments look suspicious, are treated as data rather than code: they are published from a console straight to every authorization node at once.

Kestrel’s flags have their own history. The bank carries 1,900 of them, and one, a flag that had controlled a fraud rule, was reused for a new rule. For a whole morning it declined valid card payments in one country. Kestrel also freezes production for Black Friday, but the exceptions to that freeze are granted in chat. And coding agents now write a third of Kestrel’s pull requests, most of them fraud rules.

Kestrel’s troubles will look familiar, because each of them is an episode of this chapter wearing a bank’s clothes. The reused flag is Knight Capital’s repurposed flag, with a fraud rule in place of Power Peg. Rules pushed to every node at once take the same road that carried CrowdStrike’s Channel File 291 and Cloudflare’s feature file. The weekly committee is the external change board that Accelerate found brought no better change failure rate. The exceptions granted in chat are Encore’s 41 exceptions, recorded nowhere else. And the agents’ pull requests are the Notification morning of Section 4.5, with one sharpening detail: at Kestrel, the changes machines most often write are fraud rules, which sit directly on the path that must answer within 2 s.

The deliverables follow the chapter’s order. Taken together, they should describe one bank in which every change, whether code, configuration or data and whether written by a person or a machine, travels a known route, reaches a small population before a large one, leaves a record the regulator can read without anyone writing it by hand, and can be undone, or else is known in advance to be unsafe to undo.

Deliverables:

1. Change-route map — List every route by which a change reaches Kestrel’s production, treating code, configuration and data alike: the pipeline, the fraud-rule console, the flag tool, scripts, platform automation and anything else you find. For each route, mark whether its changes are reviewed, tested, recorded, staged and reversible, and name the gaps. This is Exercise 4.2 at the scale of a bank, and it should make plain how many of Kestrel’s changes never meet the pipeline at all.
2. Pipeline redesign — Lay out the stages a change passes through, as a sequence of increasingly expensive questions, with peer review by someone who knows the code. Have the pipeline write the change record itself, capturing who approved the change, what it touched and how to undo it, so that the regulator’s record is produced as a side effect of shipping rather than reconstructed afterwards. Then take a position on the weekly risk committee: what, if anything, it should still do once the evidence lives in the pipeline’s record instead of in its minutes.
3. Progressive rollout plan — Define the customer rings through which Kestrel’s changes widen, and the canary SLIs that decide whether a change may move to the next ring; for a bank whose promise is a 2 s answer, the speed and outcome of card authorizations are natural places to look. Show how fraud rules are published the same way: staged through rings, and validated on arrival by every authorization node, so that a malformed rule set is rejected rather than loaded. Finally, say when rollback is unsafe, because the old version would meet data the new one has changed, and what extra care such changes receive.
4. Flag and freeze policy — Write the rules every flag must obey, including how you ensure a flag’s name and meaning are never reused, and a plan for working down the 1,900 flags Kestrel already has. Scope the Black Friday freeze by dependency and risk rather than by a list of services, and replace the chat with an exceptions register that places every exception in the change record.
5. Review tiers for machine-written change — Define review tiers by what a change could break, so that human attention goes where an error costs most. Write spec-level tests for fraud rules, stating properties the bank depends on that must stay true however the rules beneath them are regenerated, and fitness functions that check every change automatically for the properties Kestrel has promised to keep.

Rubric:

Where you now stand

You can now treat change as the leading cause of outages without trying to stop it, because you have seen that changing less only makes each change larger and each deployment clumsier. You can design a deployment pipeline as a sequence of increasingly expensive questions, and let it write the record that a change board once kept by hand. You can choose each change’s blast radius, through canaries compared against a simultaneous baseline and rings ordered by who can best absorb harm. You can send configuration and data down the same staged road as code, into consumers that validate them and keep the last good version when a bad one arrives. You can keep feature flags from hardening into debt, scope a freeze honestly and keep its exceptions in writing, and spend human review where machine-written change can do the most harm. Everything in this chapter has been about the changes we choose to make. Chapter 5 turns to the failures we do not choose, and to how a system can be built to withstand them.

References

• Jez Humble, David Farley — Continuous Delivery. Addison-Wesley, 2010.
• U.S. Securities and Exchange Commission — In the Matter of Knight Capital Americas LLC. SEC, 2013 — free online.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering. O’Reilly/Google, 2016 — free online.
• CrowdStrike — Channel File 291 Incident: Root Cause Analysis. CrowdStrike, 2024 — free online.
• Google Cloud — Incident report, 12 June 2025. Google, 2025 — free online.
• Matthew Prince — Cloudflare outage on November 18, 2025. Cloudflare, 2025 — free online.
• Nicole Forsgren, Jez Humble, Gene Kim — Accelerate. IT Revolution, 2018.
• DORA — Streamlined change approval. dora.dev — free online.
• Clare Liguori — Automating safe, hands-off deployments. Amazon Builders’ Library — free online.
• Audris Mockus, Rui Abreu, Peter C. Rigby, et al. — Leveraging Risk Models to Improve Productivity for Effective Code Un-Freeze at Scale. ACM TOSEM, 2025.
• DORA — Accelerate State of DevOps Report 2024 and State of AI-assisted Software Development. Google Cloud, 2024–2025 — free online.
• Stoyan Nikolov, et al. — How is Google using AI for internal code migrations?. ICSE-SEIP, 2025 — free online.
• Ziqian Zhong, Aditi Raghunathan, Nicholas Carlini — ImpossibleBench. arXiv, 2025 — free online.
• Linux kernel documentation — AI Coding Assistants. kernel.org, 2025 — free online.
• Neal Ford, Rebecca Parsons, Patrick Kua, Pramod Sadalage — Building Evolutionary Architectures, 2nd ed.. O’Reilly, 2022.

Chapter 5: Designing for Failure — Stability Patterns and Chaos Engineering

“A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable.” — Leslie Lamport

Every system fails. Some fail in small pieces and some fail all at once, and the difference between the two is rarely a matter of luck. When a failure stays small, it is usually because someone designed it to: an engineer imagined that particular failure in advance and drew a line it could not cross, such as a timeout, a partition, or a limit on how much one part may ask of another. When a failure spreads, it has usually found a path that was built for some other purpose, a channel that exists so that the parts can cooperate, and very often a path that was built for recovery. That second observation is the uncomfortable one, because it means the machinery we add to help a system heal itself can become the road along which its sickness travels. This chapter is about both kinds of structure, the lines that contain failure and the paths that carry it, and about how to find out, on a day of our own choosing rather than the failure’s, whether the lines actually hold.

Our field offers no clearer lesson than the events of 15 January 1990, when AT&T’s long-distance network spent about nine hours unable to complete a great many of its calls. Nothing physical had broken. Weeks earlier the company had installed new software on its 114 4ESS switches, and the change touched one particular routine: the code that ran when a switch recovered from a fault and told its neighbors that it was back in service. In that C code, a break statement inside an if clause, which was itself inside a switch statement, ended the enclosing block too early. For weeks the flaw lay quiet, because the code containing it ran only during recovery, and recovery was rare. Then one switch in New York suffered a minor fault and, on returning to service, sent its neighbors the usual recovery messages. A neighbor that received a second such message while it was still handling the first stepped onto the defect, corrupted its own data and reset itself. When that neighbor came back, it did exactly what a well-behaved switch should do and announced its own recovery, and its neighbors fell over in turn. The cascade rolled across the network in this way, as Figure 5.1 shows, until engineers reduced the signaling load enough for the switches to settle, and then reinstalled the previous version of the software.

Figure 5.1 — The AT&T cascade of 15 January 1990. Each switch’s recovery announcement reset its neighbors, and each reset produced the next announcement. The loop that sustained the outage is the recovery path itself.

It is worth slowing down to see why so carefully engineered a network behaved this way, because the reasons reach far beyond telephony. Every switch had been built to recover from its own faults, and that design rested on an assumption so natural that it was rarely stated: that failures would be independent, so that while one switch was recovering, its neighbors would be healthy. But all 114 switches ran the same code, and so they shared the same defect. Worse, the defect sat in the recovery path, which is the code that runs least often and matters most when it does run, and which is therefore the code least likely to have been exercised before the day it is needed. Notice also what did not cause the outage. There was no flood of ordinary calls; what kept the failure alive was a feedback loop inside the switches’ own protocol, in which each recovery produced the messages that caused the next crash, so the loop fed itself. From this follows a lesson we will return to throughout the chapter. Redundancy protects us against independent failures, such as one machine’s disk wearing out while its twin’s does not, and it does nothing at all against common ones. A fleet running identical software is, for the purposes of its defects, one machine.

Encore learned a smaller version of the same lesson early in its life. Its first big outage began when the payment service provider (PSP), the outside company that actually moves the money for each ticket, slowed from its usual 200 ms to 8 s. Nothing inside Encore had failed, and yet the slowness spread: requests waiting on the PSP filled the connection pools, services with full pools began failing their health checks, the autoscaler’s response made matters worse rather than better, and clients retrying their failed requests turned a slow dependency into a storm. It was a cascade of reasonable defaults, each sensible on its own and ruinous together, and in that respect it rhymes with AT&T’s switches, where every component also did what it had been designed to do. The Architecture book told that story and named the patterns that answered it, among them retry budgets and a single layer of retries. Naming a pattern, though, is the easier half of the work. A timeout has to be set to some particular number, a breaker has to be tuned, and a wall between components has to be checked to see whether it still holds after a year of releases. This chapter is about operating those patterns. It begins where cascades begin, at the boundaries between components (Section 5.1). It moves to the mechanisms that keep a caller from drowning when a callee slows (Section 5.2), and to what a service should do when it is simply asked for more than it can give (Section 5.3). It widens to the walls that limit how much of the system a single failure can reach (Section 5.4). And it ends with the practice of testing all of these on purpose, through the quarterly game days run by the Reliability team, Encore’s ten site reliability engineers (Section 5.5). (The map of Encore in Chapter 1 shows where each piece sits.)

5.1 Integration Points and Cascading Failure

Where faith is required

An integration point is any place where one component depends on another across a boundary it does not control: a network call, a shared database, a queue, a third-party API. Inside a single process, a function call either returns or throws, and the caller learns which. Across a boundary that certainty disappears. Michael Nygard called integration points the number-one killer of systems, and the reason is structural rather than a matter of bad luck or careless vendors. A call across a boundary can hang forever. It can return an answer that is well formed and wrong. Worst of all, it can succeed on the far side while the caller concludes that it failed: a PSP may take a fan’s money a moment after Orders & Payments has stopped waiting for the reply. In every one of these cases the caller sees much the same thing, silence or an error, and has no means of telling which story is true. What a component cannot verify, it must take on faith, and so every integration point is a place where our system extends trust to a party whose behavior we neither see nor control.

That faith would matter less if failures stayed where they started. They do not, and the opening of this chapter showed why. A cascading failure is one in which the failure of one component raises the load or latency on the components that depend on it until they fail too, so that the damage travels outward along the dependency graph. The AT&T switches passed their failure along a recovery protocol; most cascades in ordinary systems travel along integration points, from callee to caller, one hop at a time.

It is tempting to think the most dangerous dependency is the one that dies, but I have come to believe the opposite: the most dangerous integration point is the one that has slowed down rather than stopped. A dead dependency refuses quickly. The connection is rejected in a millisecond, the caller gets a clear error, and whatever fallback exists can run at once. A slow dependency is far more costly, because it holds a thread, a connection and a queue slot for as long as we are willing to wait, and software left to its defaults is willing to wait a very long time. A little arithmetic shows how fast this goes wrong. The number of calls in flight is the arrival rate multiplied by the time each call takes, the relation Little’s law gives us. When Encore’s PSP slowed from 200 ms to 8 s in that first outage, each payment call began occupying its connection 40 times longer. At an unchanged rate of purchases, Orders needed 40 times as many connections open at once just to keep pace. That is why, as the opening of this chapter recounted, the pools emptied, new requests queued behind them, health checks timed out and failed, and the autoscaler, reading those failures as a capacity problem, added instances that brought still more callers to a dependency that was already struggling. Nothing had crashed. The PSP was still answering, only slowly, and that slowness was enough to take down the service that depended on it.

A concert tour offers a fair picture of the same mechanism. If the local lighting crew simply fails to show up, the stage manager knows at once and can call in a backup crew. If they show up and work at a quarter of their usual pace, the stage manager waits, then the sound crew waits on the lighting rig, then the doors open late, and the whole evening backs up behind one slow team that never quite failed. The analogy breaks in one useful place: a stage manager can see the slow crew and decide to stop waiting. Software sees nothing unless someone has told it how long to wait and what to do afterwards, which is exactly the subject of Section 5.2.

Keeping a register

If integration points are where faith is required, a team ought at least to know where it is extending that faith. That sounds obvious, yet few teams could list every dependency of their own service from memory, and fewer still could say what the service does when each one misbehaves. Reviewing dependencies is an old part of launch practice at Google and Amazon, where a service is not considered ready until someone has walked through what it calls and how it copes. Encore’s version is an integration-point register, a table kept in each service’s catalog entry with one row per dependency. Each row records the timeout, the retry policy and the circuit breaker that guard the call; what the service does when the dependency is slow and when it is down; and the date on which the team last saw that behavior actually happen.

That last column is Encore’s own addition, and it is the one that matters. The other columns describe what the team intends; the date records whether anyone has checked the intention against reality. A fallback that has never been exercised is a belief about the system, and beliefs about failure behavior have a way of turning out wrong at the worst possible moment, as the AT&T engineers discovered about the code that ran only during recovery. A row with a recent date tells us that someone watched the dependency slow down, perhaps on a game day or in a real incident, and saw the service behave as the row says. A row with no date tells us we are still taking that behavior on faith.

The registers also show how much faith each service requires. For Orders & Payments the register runs to 11 rows, led by the two PSPs, because a checkout touches inventory, fraud signals, payment and the order store. For Seat Inventory, which was deliberately designed to depend on almost nothing, it runs to three. The difference is a design choice made visible: the service that must never sell a seat twice was built with as few places to extend trust as possible.

Writing the register for the first time is usually instructive, because it exposes dependencies that nobody chose. A typical example is a logging library that blocks the calling thread when its buffer fills. Nobody thinks of logging as an integration point, yet if the log shipper downstream slows, the buffer fills, every request that tries to write a log line waits, and a problem in the logging pipeline becomes an outage of the service itself. Such a dependency appears in no architecture diagram; it arrived with a library, and the register is often the first place it gets written down.

A register is never finished. Each release can add a dependency, change a timeout or alter a fallback, and so a register goes stale with every release unless the team revisits it, and the dates in its last column age even when nothing changes. The register’s real value is that it turns a vague unease about dependencies into a specific list of assumptions, each one either recently observed or not, and the next section takes up the mechanisms that those columns describe.

Recap. Integration points are the places where our system takes another system’s behavior on faith, and cascading failures travel along them from callee to caller, most often when a dependency slows rather than stops, because a slow call holds resources that a dead one releases at once. Components that are identical fail together, as the AT&T switches did. A register that records, for each dependency, when its failure behavior was last actually observed shows which of our assumptions have been checked and which we are still holding on faith.

Exercise 5.1. Write the integration-point register for one service you run, with a row for each dependency, including the ones nobody chose. For each row, fill in the date when you last saw the “slow” behavior actually happen, in a test or in production. How many rows have no date?

5.2 Timeouts, Retries, Circuit Breakers, and Backpressure

Timeouts derived from budgets

Section 5.1 showed a slow dependency holding threads, connections and queue slots for as long as we are willing to wait. The first defense is to decide how long that is. A timeout is a caller’s promise to itself about how long it will wait for an answer before it gives up and does something else. Every network call has one, whether or not anyone wrote it down, and that is the trouble: most production timeouts are library defaults that nobody chose, often measured in tens of seconds, set by a library author who knew nothing about our users. A default of that kind is not a decision. It is the absence of one, and Encore’s first big outage showed what it costs when a PSP that normally answers in 200 ms starts taking 8 s.

The discipline is to derive each timeout from the user’s patience rather than from the library. We start with a latency budget: the time the user-facing operation may take end to end, divided among the calls that serve it. Each hop receives a share, and its timeout must fit inside that share. Figure 5.2 shows the division for Encore’s 3 s checkout, from the moment the fan presses “pay” to the moment the confirmation comes back.

Figure 5.2 — The 3 s checkout budget, divided. Each timeout sits well above the dependency’s normal tail and within its share of the budget. The large margin is deliberate: on an on-sale morning, queues are where the time goes.

Read the table one row at a time and each line is a small argument. Confirming the seat hold normally takes 60 ms at the 99th percentile, so a 250 ms timeout leaves room for a bad moment without letting a sick Seat Inventory eat the whole checkout; the single retry is allowed only “if budget remains”, meaning only if the request’s overall deadline still leaves room for another attempt after the first has failed. The fraud signal from Bot Screening is useful but not essential at this point, because the fan has already passed screening at the Gate; if it does not arrive in 150 ms, Orders proceeds without it and marks the order accordingly, which is what “fail open with a flag” means. The PSP gets the largest share, 800 ms, because charging a card is the slowest and most important thing in the path, and it gets no retry at all, for reasons we come to shortly. The last row may look like waste. It is the most deliberate number in the table, because on an on-sale morning the time goes into queues, and a budget that spends everything on the happy path has nothing left when the queues grow.

That still leaves the question of where, exactly, inside its share, each timeout should sit. Marc Brooker’s advice, drawn from Amazon’s practice, is to begin with the rate of false timeouts we are willing to tolerate, meaning calls to a healthy dependency that we abandon only because they were a little slower than usual, and to read the timeout off the matching percentile of the dependency’s latency. If we can accept one false timeout in a thousand calls, the timeout belongs near the p99.9. The arithmetic is unforgiving in the other direction: a timeout set just above p99 abandons one healthy call in a hundred by design, and at Encore’s on-sale volumes one in a hundred is a great many fans. Encore’s timeouts therefore sit near each dependency’s p99.9, which is why every value in Figure 5.2 stands well above the normal p99 beside it.

A table like Figure 5.2 is only a plan, and three practices are needed to make the services behave as the plan says. The first is deadline propagation: instead of telling a callee “you have 250 ms”, the caller sends the absolute time by which it needs an answer, and each callee works only while that moment still lies ahead. The difference matters because time already spent upstream is gone; a service that receives a request with 40 ms left should not begin work that takes 200 ms. Encore’s gRPC calls carry deadlines, and each service passes its deadline on to the calls it makes in turn, just as distributed tracing asks services to pass on trace context. The second practice is the twin of the first, cancellation propagation: when a caller gives up, the cancellation travels down the chain, and the callees stop working on an answer nobody will read. Without it, a timed-out request keeps consuming capacity all the way down, which is exactly the hidden load a struggling system can least afford.

The third practice is a matter of honesty about what a timeout tells us. When a call that has a side effect times out, its outcome is unknown, and it stays unknown until someone reconciles it. A timeout means only that the answer did not arrive in time; it does not mean the work was not done. A PSP charge that timed out may well have succeeded on the provider’s side, and a naive retry could charge the fan twice. So Orders never retries a PSP charge inside the fan’s request. It reconciles the charge later, asking the provider what happened under the same idempotency key, and only then decides whether the purchase stands. I suspect that this distinction, between “failed” and “did not hear back”, is the single most commonly forgotten idea in distributed systems, and the one that most often costs real money.

Retries with jitter and a budget

A timeout tells us when to stop waiting; the next question is whether to try again. A retry is a bet that the failure was transient: a dropped packet, a brief pause, a replica restarting. Placed by one client, it is usually a good bet. Placed by every client at once, it becomes a denial-of-service attack on our own dependency, and the reason is synchronization. Clients that fail at the same moment, because the same dependency hiccupped under all of them, and then back off by the same exponential schedule, will also retry at the same moments, in waves. Each wave arrives just as the dependency is trying to recover, and knocks it down again.

The cure is jitter: randomness added to the wait before each retry, so that clients which failed together do not return together. Jitter turns synchronized waves into a steady trickle that a recovering dependency can absorb. Brooker’s simulations compared several schemes and found that “full jitter”, in which each client waits a random time between zero and the current exponential ceiling, did markedly better than exponential backoff alone. It is a slightly counterintuitive result, since full jitter sometimes retries almost immediately, but spreading the load evenly matters more than making any one client patient.

Jitter shapes when retries arrive; it does nothing to limit how many there are. That is the job of a retry budget, which caps retries as a share of ordinary traffic. Encore’s service mesh allows retries of about 10% of ordinary requests to each destination. It enforces this with a token bucket: each successful request adds a fraction of a token, each retry spends one, and when the bucket is empty, failures are returned to the caller instead of retried. The effect is that retries flow freely when failures are rare and dry up when failures are common, which is exactly when retrying stops being a good bet. The 10% figure is Google’s ratio, and it is tighter than the 20% that Envoy and Linkerd ship by default.

The budget has two companions. Retries happen at one layer only, the one immediately above the failing call, and only for idempotent operations, those that can safely be performed twice. The first rule matters because layers multiply, as Figure 5.3 shows. Suppose a request passes through three services before reaching a failing dependency, and each layer makes up to four attempts, one original and three retries. The bottom layer makes four calls for every request it receives, the middle layer calls the bottom four times, and the top calls the middle four times, so one fan’s click can become 4 × 4 × 4, or 64, calls on the dependency, arriving just when it is weakest. The second rule matters because retrying a non-idempotent operation, such as the PSP charge above, turns a transient fault into a duplicate effect.

Figure 5.3 — Load on a failing dependency when every layer above it retries. Three layers of three retries can multiply it 64 times just as the dependency is weakest, while a 10% budget holds it near normal.

A retry budget, however, leaves one gap. It limits retries, but not first attempts, and first attempts sent to a backend that will only reject them still spend its capacity: the backend must accept the connection, parse the request and decide to refuse it, and under heavy overload the refusing alone can saturate it. The SRE book describes Google’s answer to this gap, client-side adaptive throttling. Each client keeps two counts over the last two minutes: how many requests it sent, and how many the backend accepted. While the backend accepts nearly everything, the client does nothing. Once requests exceed a small multiple of accepts, the client begins rejecting a growing share of new requests itself, before they ever leave, so the backend never has to spend effort turning them away.

# Google SRE book, ch. 21. Counts cover a rolling two minutes.
def reject_probability(requests: int, accepts: int, k: float = 2.0) -> float:
    return max(0.0, (requests - k * accepts) / (requests + 1))

if random.random() < reject_probability(window.requests, window.accepts):
    return fail_locally(request)   # never reaches the backend

The formula rewards a moment’s study. With k set to 2, a healthy backend that accepts nearly all of its requests gives requests - 2 * accepts well below zero, so the probability is zero and nothing is throttled. As the backend begins rejecting, accepts fall behind requests, and once requests pass twice the accepts the client starts dropping its own traffic. The system then settles where the backend receives about twice what it can accept: enough extra that the client notices quickly when the backend recovers, but not so much that rejections become the backend’s main workload. Each client reaches this decision alone, from its own counts, with no coordination at all, which is what makes the scheme robust.

There is one retry layer that no mesh can see, and it is made of people and their devices. Fans who see an error refresh the page, and the fan app may retry on their behalf before they even notice. Neither passes through the mesh’s budget, and so neither is limited by it. Section 5.5 shows Encore discovering exactly this layer during a game day.

Circuit breakers as state machines

Retries and budgets make each call more careful. A circuit breaker takes a broader view: it watches the calls to one dependency together, and when enough of them fail, it stops sending calls at all for a while, so the dependency can recover and the caller stops wasting threads waiting on it. The name comes from the breaker in a household electrical panel, which trips when a circuit draws too much current, sacrificing the circuit’s service to save the wiring from fire. The analogy holds well for the tripping. It breaks at the reset: a household breaker waits for a person to flip it back, while a software breaker tests the dependency for itself and closes again on its own.

That behavior is a small state machine. In the closed state, calls flow normally and the breaker counts failures. When failures cross a threshold, it moves to open and rejects calls immediately, without trying the dependency. After a while it moves to half-open and lets a few trial calls, called probes, through; if they succeed it closes, and if they fail it opens again. Those three states are familiar to most engineers. What is less familiar is that the states are the easy part, and that the parameters around them decide whether the breaker protects the system or harms it. Figure 5.4 lists those parameters, in the terms of Resilience4j, a common Java library.

Where the breaker lives also matters. A service mesh offers something it calls a “circuit breaker”, but it is usually a concurrency limit combined with outlier ejection, which removes individual misbehaving instances from a load-balancing pool. That is useful, but it is not a breaker that understands payments, can count a slow call as a failure, or knows to switch providers. Encore’s PSP breaker therefore lives in the payment client inside Orders, close to the code that knows what a payment is.

Figure 5.4 — The circuit breaker’s tuning surface. The slow-call threshold must sit above the call’s normal p99 and below its timeout, here 450 ms, 600 ms and 800 ms, or a PSP that is slow without failing outright never trips it.

The row most worth dwelling on is the slow-call threshold, because it is where a breaker most often disagrees with the timeouts around it. Encore’s first big outage was a slow PSP, not a failed one, and a breaker that counts only errors would barely have noticed it. So the breaker also counts a call as slow when it takes longer than 600 ms, and opens when half the calls in its window are slow. The number is constrained from both sides. It must sit above the PSP’s normal p99 of 450 ms, or an ordinary busy minute would trip it. And it must sit below the 800 ms timeout, because a call that would take longer than 800 ms is cut off by the timeout and recorded as a failure, never as slow; a threshold at or above the timeout can therefore never fire. The three numbers, 450, 600 and 800, have to be read together, and a team that tunes them separately, in different files owned by different people, will sooner or later set one that silently cancels another.

Two of Encore’s other choices deserve a word. Requiring all five half-open probes to succeed is stricter than Resilience4j’s default, which closes the breaker once the probes’ failure rate falls below the threshold. Encore chose strictness because a PSP that flickers between healthy and sick is costly: every attempt sent to it while it is failing may become an unknown charge that must be reconciled, while a few extra seconds on the secondary provider cost almost nothing. The second choice is about scope, and here Brooker offers a warning: a breaker scoped more widely than the failure it guards can harm a sharded design. If one shard of a sharded service falls sick, a single breaker in front of the whole service sees a rising failure rate and opens for every key, including the healthy shards that could have served most of the traffic. That is why Seat Inventory, sharded by event, has no breaker at all. Encore’s calls to it rely on deadlines and the retry budget instead, which fail individual requests without cutting off the healthy shards.

A breaker that opens must send the caller somewhere, or it merely produces errors faster. The fallback is the other half of the design, and for payments Encore’s is a ladder of degraded modes, each rung a smaller promise than the one above it. On the first rung, the breaker on the primary PSP opens and moves new payment attempts to the secondary provider. Attempts already in flight finish or fail where they started, because idempotency keys belong to one provider, and a charge half made at one cannot be completed at the other. On the second rung, reached if both providers are degraded, verified fans’ seats are held as a pending payment: the hold runs for up to 20 minutes instead of the usual 5, with a cap on how many seats each fan may hold this way. When a provider recovers, the fan is sent a link to complete the payment, with strong customer authentication. Holds left unpaid lapse, and those seats return to sale. On the last rung, checkout fails fast with an honest message, and the Gate slows admission so fewer fans arrive at a checkout that cannot serve them.

What no rung does is issue a ticket without an authorized payment. That is partly law, since European rules require the fan to authenticate a card payment as it happens, and partly design, since Encore holds no card data and so could not charge a card later even if it wished to. A tour manager whose headline act falls ill may shorten the show, switch to an acoustic set or reschedule the date, but does not let the audience believe the band is about to come on. A degraded mode may shrink what we promise; it may never make the promise false.

A ladder of fallbacks carries its own danger, which Jacob Gabrielson describes from Amazon’s experience: a fallback that runs only in emergencies tends to fail exactly when it is needed. It is code that almost never runs, against a provider that has seen almost none of our traffic, and both drift silently between emergencies. Encore’s answer is to keep the secondary warm. Orders sends a small, steady share of ordinary payments through the secondary PSP every day, so the integration is exercised, its credentials are valid and its latency is known before the day anyone depends on it. The same doubt applies to the breaker itself. A breaker that has not opened in a quarter is most likely guarding a healthy dependency, which is good news, but it has not been seen to work either, and only fault injection, the subject of Section 5.5, can show that it does.

Backpressure

Everything so far has been about the caller protecting itself from a callee. Backpressure runs the other way, letting the callee protect itself. Backpressure is the means by which a component that cannot keep up tells its upstream to slow down, before it accepts work it cannot finish. The image is borrowed from plumbing, where a narrow pipe pushes back against the flow feeding it, and the image is apt in one respect: the pressure is felt upstream, where something can still be done about it. In software, three habits make it work. Queues are bounded, so that a full queue refuses new work instead of growing until every item in it is too old to matter. Callers honor a “retry after” signal, waiting as long as the busy service asks instead of hammering it at once. And failure comes early, at the first point that knows the work cannot be done in time, rather than late, after every layer has spent effort on it.

Backpressure has to end somewhere, because each service that slows down pushes the pressure onto the one before it, and eventually the pressure reaches a person. At Encore the chain ends at the Sale Gate, the one place in the system where fans are meant to wait. There, a waiting fan sees a queue position on a static page, which is a far kinder experience than an error from deep inside checkout. Section 5.3 takes up the Gate as an admission controller and what to do when, despite it, too much work gets in.

Recap. Timeouts come from the user’s budget, not from library defaults, and they travel down the call chain as deadlines, with cancellations following them; a timed-out side effect is unknown until it is reconciled. Retries need jitter to break up synchronized waves and a budget to limit their volume, they belong at a single layer, and clients can throttle their own first attempts when a backend is rejecting most of them. Breakers must agree with the timeouts around them, and their fallbacks are ladders of smaller but honest promises, each exercised a little every day so we know it works. Backpressure carries overload back upstream to the one place, the Sale Gate, where waiting is part of the design.

Exercise 5.2. Build Figures 5.2 and 5.4 for one operation you run. Does any slow-call threshold sit at or above its timeout, and which of your fallbacks ran in production last week?

5.3 Overload, Load Shedding, and Tail Latency

Admission control at the Gate

Backpressure, as the previous section left it, pushes the signal to slow down all the way upstream until it reaches someone who can wait. Sooner or later, though, the signal arrives at the edge of the system, where the work comes from people rather than from other services, and something there must decide how much of it to accept. Two ideas govern that decision, and they are easy to confuse. Admission control decides at the entrance how much work to let in at all. Load shedding discards work that is already inside the system but cannot be finished in time. The first is prevention and the second is triage, and triage is always the more expensive of the two, because the work it throws away has already consumed capacity: a request shed halfway through checkout has already been parsed, authenticated, routed and partly processed, and every one of those steps was paid for and then wasted.

A concert venue makes the difference concrete. The doors open at a set time, staff scan tickets at a steady pace, and the crowd that has not yet been let in waits outside, where waiting is cheap and nothing inside is harmed by it. That is admission control. If instead the doors were thrown open and the venue later discovered that the hall was dangerously full, someone would have to walk people back out through a crowd that had already found its place, which is slower, angrier and more dangerous for everyone. That is load shedding. Here the picture stops matching software, and the mismatch is instructive: a venue’s capacity is printed on a certificate and stays put, while a software system’s capacity moves with every release, every cold cache and every slow dependency, and so the number at the door is always an estimate.

The Sale Gate is Encore’s admission controller, its doorkeeper. It admits about 1,200 fans per second per event, a rate chosen to keep Seat Inventory below 80% of its measured capacity, and everyone else waits on a static page that shows them their position in the queue. A static page costs almost nothing to serve, so a great many fans can wait there without putting any weight on the services behind it. The visible queue position matters too, because a fan who can see the line moving has less reason to refresh in frustration and add load of their own. Commercial waiting rooms from Cloudflare, Queue-it and AWS work the same way: an operator sets the admission rate, and the product holds everyone beyond it in a queue.

That operator-set rate is the weak point, and it deserves to be named as such. The rate is a judgment, reviewed at the on-sale readiness review the Wednesday before each big on-sale, and it is only as good as the load test behind it. A test that misjudges capacity produces a rate that misjudges it too, and the Gate will then faithfully admit exactly the wrong number of fans. The natural wish is to let the system set its own rate by watching how its services are coping, and one level down that is already proven practice. Netflix’s concurrency-limits library and Envoy’s adaptive concurrency filter both bound how much work a single server has in flight by the latency that server observes: when responses slow, the limit shrinks, and when they speed up, it grows again, in the spirit of the congestion control by which TCP probes for the capacity of a network path. Encore’s services run the Envoy filter. A waiting room steered by the latency of a service several hops downstream is younger practice, with a longer and more delicate feedback loop, and so the Gate’s rate stays a human judgment for now. This seems to me the right order of things: automate a control loop where it has been proven, and keep a person on the one whose mistake would turn away an entire on-sale.

Shedding behind the gate

A well-chosen admission rate does not make the services behind the Gate safe, because the Gate controls how many fans arrive, not how much work each one causes. Retries can multiply that work, a cold cache can make each request several times more expensive than the load test assumed, and a slow dependency can hold threads and connections long enough that a normal arrival rate becomes an overload. So the services behind the Gate need a way to shed, and shedding well means deciding in advance which work matters most. Encore’s services tag each request with a criticality class, a label saying how much the business loses if that request is dropped, following the practice the SRE book describes. There are three classes: checkout, for a fan who is paying for seats; browse, for a fan looking at seat maps and prices; and background, for work nobody is waiting on at that moment. Under pressure they are shed in reverse order, background first and checkout last. This is triage in the medical sense, where a crowded emergency department sees the patient whose condition is most urgent rather than the one who arrived first, and like medical triage it works only if the categories were agreed before the crowd came through the door.

Three further rules decide how the shedding is done, and each follows from the cost of wasted work. The first rule is to shed early, before expensive work is spent on a request that will be discarded anyway. A request rejected at the front door of a service costs almost nothing; one abandoned after its database query has already cost the query.

The second rule is to measure pressure by time in queue rather than by the length of the queue. This is the insight of Kathleen Nichols and Van Jacobson’s controlled-delay algorithm, CoDel, devised for network routers in 2012, which Facebook later carried into its RPC servers. The reasoning is that a queue’s length says little on its own: a queue of a hundred cheap requests drains in moments, while a queue of ten expensive ones may not. How long requests have been waiting is the quantity that matters, because every request has a deadline, and a request served after its caller has given up wastes capacity twice: once in doing work nobody will read, and again because that capacity was taken from a request that could still have succeeded.

The third rule applies under severe overload, when the queue is already long and the oldest requests in it are probably past saving. There Facebook’s servers serve the newest requests first, last in, first out, because the callers of new requests are still listening, while the callers at the back of an old queue have very likely given up. This feels wrong, and the feeling is worth examining. Each of these policies deliberately trades fairness for goodput, the rate of work completed usefully within its deadline, as distinct from throughput, which counts every response, including the ones nobody waited for. Under last in, first out, the fan who has waited longest is the one least likely to be served, which is unfair by any everyday standard and right by the standard of how many fans finish buying their tickets. That trade-off should be written down where the on-call engineer can find it, because during an incident a policy nobody explained will look exactly like a bug.

Metastable failures

Admission control and shedding both assume that once load falls, the system recovers. The most dangerous overloads are the ones in which that assumption fails. The Architecture book borrowed a term for them from Nathan Bronson and colleagues (2021): a metastable failure is one that persists after its trigger has been removed, because the system’s own response to the failure keeps it failed. The word comes from physics, where a metastable state holds steady until a small push sends it somewhere it cannot easily climb back from. Bronson and colleagues describe three ingredients. There is a vulnerable state, typically a system running efficiently close to its capacity, which is precisely the state that careful cost management encourages. There is a trigger, such as a slow dependency or a burst of traffic. And there is a sustaining effect, a feedback loop that keeps the system down after the trigger has passed. These failures are not exotic. Lexiang Huang and colleagues found them at 11 organizations, including at least four of 15 major AWS outages, and retries were the commonest sustaining effect: requests time out, clients retry, the retries add load, more requests time out, and the loop feeds itself.

AT&T’s collapse, which opened this chapter, had the same shape. The trigger, one switch’s minor fault, was over in seconds, and yet the network stayed failed for hours. Its sustaining effect, though, was not excess ordinary traffic but the recovery protocol itself, each switch’s announcement resetting its neighbors, which is why the cure needed a rollback to the previous software as well as less signaling load. Encore’s PSP outage had the classic structure. The trigger was the PSP slowing from 200 ms to 8 s. The sustaining effects were retries, which multiplied the load on a provider that could not answer, and an eager autoscaler, which responded to failing health checks by adding instances that opened still more connections to the same struggling dependency.

The practical lesson is uncomfortable. A metastable system will not recover when the trigger is fixed, and often not even when new capacity arrives, because new capacity starts cold, with empty caches and no warm connections, and the loop simply absorbs it too. The system recovers only when load falls below the level at which the loop can no longer sustain itself, and that level may lie well below normal. In practice that means shedding far more than feels reasonable, turning retries off, and then readmitting traffic in steps while watching that each step holds. A fire crew facing a fire that keeps relighting works the same way: it does not stop when the visible flames are gone but cools everything well below the point of ignition, and only then lets the building be used again. The analogy breaks because nobody resents firefighters for using too much water, while an engineer who turns away 70% of the fans during an on-sale will be asked to explain it afterward. In a metastable failure the cure always looks like an overreaction, and so the runbook should say so in advance, plainly and with numbers, since nobody at 3 a.m. wants to be the person who turned away 70% of the traffic on purpose and on their own authority.

Better still is to find the loop before it finds us. Encore’s load tests push Seat Inventory past the point where goodput stops rising, deliberately into overload, then bring the load back down to normal and check, as the SRE book advises, that the service recovers unaided. A service that returns to health when the load drops has no sustaining loop at that level. One that stays degraded at normal load has just shown us a metastable failure in a rehearsal, where it costs an afternoon rather than an on-sale.

Tail latency under load

Overload rarely announces itself in the average. It shows first in the tail, the slowest few percent of requests, because those are the requests that land behind a long queue or on a busy replica. One technique aimed squarely at the tail is the hedged request, from Dean and Barroso’s “The Tail at Scale” (2013): when a read has not returned within some threshold, the client sends a second copy to another replica and uses whichever answer comes back first. Hedging cuts the tail because a slow replica is usually slow for reasons of its own, so the second copy rarely meets the same delay. But hedging has a cost that grows at the worst moment. During overload, many requests cross the threshold, so many are hedged, and the system receives extra load exactly when it can least absorb it, which is the retry storm again in a politer form.

Encore therefore hedges narrowly, and each restriction answers one of these dangers. It hedges only idempotent reads of seat-map geometry, the layout of sections and rows, so a duplicate request can never cause a duplicate effect. It hedges only reads that have taken longer than the p95, so in normal operation only about one read in 20 is even a candidate. It allows hedges only within a budget of about 5% extra requests, so that when the system is busiest and many reads are slow, the budget runs out and hedging stops instead of adding to the overload. And it cancels whichever copy loses, so the replica still working on the slower copy stops spending capacity on an answer nobody will read.

Recap. Admission control at the entrance is cheaper than shedding inside, because shed work has already been paid for, and the Gate’s admission rate is a human judgment that must keep being reviewed against honest load tests. Behind the Gate we shed by criticality and by time in queue, accepting that these policies trade fairness for goodput and writing that trade-off down. Metastable failures are common and outlast their triggers; ending one takes more shedding than feels reasonable, and a load test that pushes past the peak and then checks for recovery finds them before an on-sale does. Even a technique that helps the tail, such as hedging, needs a budget, or it becomes one more source of overload.

Exercise 5.3. Find an incident in a system you know where load stayed high, or the service stayed degraded, after the original cause was fixed. What was the sustaining effect, and what runbook text, written in advance, would have given the on-call engineer the confidence to shed enough traffic to break the loop?

5.4 Cells, Regions, and Blast Radius

A ladder of bulkheads

The patterns of the last two sections act on a failure while it is happening: a timeout cuts a slow call short, a breaker stops calling a sick dependency, the Gate turns fans away before Seat Inventory drowns. This section asks a different and more architectural question, which is how far a failure can travel before something stops it. The blast radius of a failure is the set of users, tenants or functions it can affect. We cannot make failures rare enough to ignore, but we can decide, in advance and on purpose, how large each one is allowed to become.

The Architecture book used a ship’s bulkheads as the image for shrinking the blast radius: watertight walls that let a holed ship flood one compartment and stay afloat. Operations adds a refinement that the image hides. Bulkheads come in sizes, and each size contains a different class of failure. A wall around one connection pool keeps a slow dependency from starving the rest of a process, but it does nothing when the process itself crashes; for that we need a wall around the process. Each larger wall exists because the smaller one has a known gap. Figure 5.5 arranges these walls as a ladder, from the smallest compartment to the largest.

Figure 5.5 — The ladder of bulkheads. The “does not contain” column of each row is the reason the next row exists.

Figure 5.5 is best climbed one rung at a time, with the third column in view. Encore gives the PSP its own connection pool in Orders, so a slow PSP cannot take every connection the service has, which was exactly the mechanism of Encore’s first outage. But a pool cannot protect against a process that dies, so pods are spread across zones. Pods cannot protect against a fault they all share, so data is split into shards, four for each large event. And so on up the ladder, until we reach the region, whose own gaps (a global control plane, a defect already present everywhere) have no larger wall above them. Choosing a rung, then, is really choosing which failure we fear most and paying for the smallest wall that holds it.

One class of failure deserves particular attention, because it is the one we cause ourselves: the bad deploy. Here the ladder behaves in a way that is easy to miss. A rung contains a bad deploy only if the change reaches that rung at a different time from its neighbors. A wall that every compartment crosses at the same moment is no wall at all. This is why Amazon deploys to one box, then one zone, then region by region, and why Microsoft rolls Azure changes through canary regions first: in each case the rollout itself is staged along the bulkheads, so that a defect shows itself in one compartment while the others are still running the old code. Among these rungs, the cell is the smallest at which a whole application, with all its services, can be updated on its own; pods and shards are pieces of one service and usually take its new version together, whereas a cell is a complete copy that can be held back while another moves ahead.

Staging has a limit, though, and it is the same limit this chapter opened with. No rung contains a latent defect that is already everywhere when its trigger arrives. AT&T’s 114 switches had all been running the faulty recovery code for weeks; when the trigger came, every switch was equally ready to fail, and no boundary between them could help. Google Cloud relearned this on 12 June 2025 (Chapter 4). The new code in Service Control had been rolled out carefully, region by region. Its flawed path, however, did not run until a policy change arrived, and that policy change, replicated globally in seconds, reached every region at once. The code had been staged, but its trigger had not, so the staged rollout did not help. To my mind, this is the most important line in Figure 5.5: a bulkhead holds only against what crosses it at its own time, and data and configuration cross far faster than code.

Cells for tenants

Of all the rungs, the cell is the one an architect most deliberately draws, and it is the one Encore leans on for its software-as-a-service tenants. A cell is a complete, independent copy of the application serving a fixed slice of customers: a blast radius drawn on purpose. Where a shard divides one service’s data, a cell divides the whole system, with its own services, its own data and its own capacity, so that what goes wrong inside one cell has no path to the customers of another.

Encore’s Professional tenants live in Kubernetes cells, and about 500 self-service venues live in serverless cells of about 50 venues each. The arithmetic of that choice is what makes it worthwhile. Every deploy and configuration change reaches a canary cell of internal and early-adopting tenants first, then the remaining cells in the waves of Chapter 4. If a release carries a defect that the canary cell does not reveal, and it reaches the next wave, it meets one cell of about 50 venues before the wave pauses, so a defect that would have reached all 500 venues at once in a single pooled deployment reaches about 50. The same logic applies to a poison tenant, a customer whose data or traffic triggers a fault: in a pooled system it can harm everyone, and in a cell it can harm only its cellmates.

The concert world has understood this kind of independence for a long time. Big stadium tours carry more than one steel stage structure, leapfrogging them from city to city, so that a storm delaying one build does not cancel the next night’s show. Cells give Encore the same independence: trouble in one copy leaves the others free to go on. The analogy is worth pressing until it breaks, because the place it breaks is instructive. Two stages built from one faulty drawing will fail alike, however many trucks carry them apart, and cells running one faulty release fail alike too. Independence of copies protects against accidents that happen to one copy; it does nothing against a mistake in the design they share. That is why the staged rollout across cells matters as much as the cells themselves.

There is one more thing the cells share, and it is the most dangerous: the router that maps each tenant to its cell. Every request passes through it, so its failure crosses every bulkhead at once, and the ladder offers no wall above it. AWS’s guidance on cell-based architecture asks for this layer to be as thin as possible, doing little more than looking up a tenant and forwarding the request. Encore follows that advice, and goes a step further in discipline: the router is the component it changes least often of all those it owns. Here the usual virtue of frequent, small change gives way to a different calculation. A component whose failure is shared by every customer earns its stability by doing almost nothing, and by being touched almost never.

Shuffle sharding

Cells divide tenants between copies of the system, but inside a pooled cell, where 50 venues share the same workers, the question of isolation arises again on a smaller scale. One venue running a huge import, or receiving a flood of webhooks, can occupy workers its neighbors need. The obvious answer is to partition the workers and give each venue a fixed group, but a fixed group means that every venue in it shares the misfortune of its noisiest member. Shuffle sharding, which Colm MacCárthaigh described from AWS practice, takes a cleverer route: it gives each tenant a small random combination of workers instead of a fixed partition, so that no two tenants are likely to share all their workers.

In each self-service cell, Encore divides its import and webhook workers into eight lanes and assigns each venue two of them. Eight lanes can be paired in 28 different ways (eight choices for the first lane, seven for the second, halved because the order does not matter), so there are 28 possible pairs for the 50 venues of a cell. Figure 5.6 shows what happens when one venue’s runaway import floods its two lanes.

Figure 5.6 — Shuffle sharding across a self-service cell’s eight lanes. A runaway venue floods its own two lanes; neighbors sharing one of them keep working on the other, and in a cell of 50 one or two neighbors, on average, hold the same pair and are hit.

The runaway venue’s neighbors fall into three groups. Most hold neither of its lanes and never notice the flood. Some share one lane with it, and they keep working on their other lane. A few hold exactly the same pair, and they are flooded along with it. The isolation is cheap, since it needs no extra workers, only a different way of assigning them, but it is conditional, and the conditions are worth stating plainly.

The first condition is that work must be able to move. A neighbor that shares one flooded lane is protected only if its consumers take its work from the other lane while the first is clogged. If each piece of a venue’s work were bound to one particular lane, half of that venue’s work would wait behind the flood. MacCárthaigh’s account assumes that work can move, and Encore’s consumers are built so that it does.

The second condition is that pairs repeat. With 28 pairs and 49 neighbors, some neighbors must hold the same pair as the runaway venue. Any one neighbor shares the runaway’s exact pair about one time in 28; across 49 neighbors, that comes to 49 divided by 28, or about 1.75, which is to say that on average one or two neighbors are hit in full. Many more share a single lane: of the 28 pairs, 12 contain exactly one of the runaway’s two lanes, so on average about 21 neighbors run on one lane, at reduced capacity, until the flood clears. Encore spreads the repeats evenly when it assigns pairs, so that no pair is crowded, and accepts what remains. The comparison that justifies the scheme is with fixed partitions. Had Encore split its eight lanes into four fixed groups of two, each group would serve about 12 or 13 venues, and a runaway venue would take about 12 neighbors down with it, instead of one or two. The same eight lanes, assigned by combination rather than by partition, shrink the full blast radius by roughly a factor of six.

Static stability

Everything so far in this section has been about keeping a failure small. The last idea is about recovering from it, and in particular about what recovery is allowed to depend on. Most systems have two parts with very different jobs. The data plane does the work customers see: routing requests, selling seats, taking payments. The control plane changes the data plane: it creates instances, scales capacity, updates routing tables and elects new leaders. A system is statically stable when its data plane keeps working, unchanged, while its control plane is impaired, so that recovery never depends on creating, scaling or reconfiguring anything.

The idea is old, and engineers in other fields would recognize it at once; the name is AWS’s. Its central lesson is uncomfortable. The control plane is likeliest to be overwhelmed exactly when everyone needs it, because a zone or region failure sends every customer of the cloud to the same control plane at the same moment, each asking for new instances to replace the ones they lost. A recovery plan that begins “and then we launch more capacity” is a plan to queue behind everyone else at the worst possible time. A statically stable design avoids the queue by not needing to join it.

Encore applies the idea in three places. First, the tenant-to-cell routing table, which the router consults on every request, is cached on every edge node, with an alert when a node’s copy grows old; if the service that publishes the table fails, the edge keeps routing with what it already has. Second, each tier carries enough capacity across its three zones that losing one zone needs no scaling at all; the surviving two zones can already carry the load. Third, the seat ledger’s replicas in the other region, together with the witness, already hold every committed write, so a region failover elects a new leader among copies that are already running and builds nothing new, as Chapter 6 tests. In each case recovery is a matter of using what exists, not of creating what is missing.

The price is idle capacity: spare headroom in every zone, replicas that serve no writes until the day they must. That spare capacity can look like waste on a cost report, and in a quiet quarter it is hard to defend. In return, recovery uses only what is already running, and it does not depend on any system that might be failing at the same moment for the same reason. For a company whose busiest hour is a big on-sale that cannot be rescheduled, that is a trade worth making.

Recap. Bulkheads come in sizes, and we choose among them by the failure we fear most; each rung exists because the one below has a known gap. Any rung contains a bad deploy that reaches it at its own time, cells are the smallest rung at which a whole application can be staged, and no rung contains a defect already everywhere when its trigger arrives, because configuration and data often cross every wall at once. Shuffle sharding shrinks the blast radius inside a pooled cell cheaply, provided work can move between lanes, and static stability keeps recovery from depending on a control plane that is under stress at the very moment we need it.

Exercise 5.4. Draw Figure 5.5 for your system, with the third column filled in honestly. Which shared component (a router, an identity provider, a global configuration store) crosses every rung, and does its data reach production in stages as your code does?

5.5 Chaos Engineering and Game Days

The experimental attitude

Every pattern in this chapter rests on a claim about how the system behaves under conditions we hope never to see. The breaker will open when the PSP slows. The cell will contain a bad tenant. The failover will finish before fans notice. We can rarely check such claims in ordinary operation, because the conditions they describe rarely arrive, and so the claims quietly decay. A release changes a timeout, a library swaps its defaults, a runbook falls out of date, and nothing tells us, because nothing has tested the claim since the day it was written. Chaos engineering is the practice of testing those claims on purpose, by bringing the unwelcome conditions about ourselves, at a time and scale of our choosing, instead of waiting for them to arrive on their own.

The practice has a short and fairly well-recorded history. At Amazon in the early 2000s, Jesse Robbins, a volunteer firefighter as well as an engineer, ran exercises he called GameDays, which took real capacity out of service to see how both the systems and the people coped. The firefighter’s habit shows: a fire company does not wait for a fire to discover whether its hoses reach. At Netflix, during the company’s move onto AWS from around 2008, engineers took the idea further and made failure routine on purpose, so that every service had to survive the loss of its neighbors as an ordinary event; Chaos Monkey was the famous result. The community later wrote its approach down as the Principles of Chaos Engineering, which define the practice as “the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production.” The principles ask for five things: a hypothesis about the system’s steady state, injected events that resemble real-world ones, experiments run in production, automation so the experiments can run continuously, and a minimized blast radius. The practice is now widely adopted, but we should be honest about the evidence: the case that it improves outcomes still rests mostly on practitioners’ reports. Those practitioners do agree on one precondition. Experiments belong in production only once observability, automatic aborts and an on-call rotation are in place to catch the problems the experiments expose. An experiment that breaks something nobody can see, and that nothing can stop, is simply an outage we caused.

The word in that definition that matters most is experiment, and it is worth being precise about how an experiment differs from a test. A test checks a property we already know the system should have, and it passes or fails. An experiment asks a question whose answer we do not know, and its result teaches us something either way. “Does the breaker open when the PSP is slow?” is a test: Section 5.2 specified the answer, and we are checking that the code agrees. “What happens to checkout when the PSP is slow during a shard failover?” is an experiment. Each failure has been studied alone, but nobody has designed for the two together, and the interesting behavior, if there is any, lives in their interaction. I have come to believe that this is where most of chaos engineering’s value lies. Our designs handle the failures we imagined one at a time; production delivers them in combinations.

Game days

Automated experiments test the machines. But Lisanne Bainbridge’s ironies of automation, which the Operations book takes up in its third chapter, warn us that the people who must step in when the automation fails are part of the system too, and their skills decay faster than the code does. A game day addresses that half of the problem. It is a scheduled exercise in which people and systems face an injected failure together, with observers recording what the people did as carefully as what the machines did: who was paged, which dashboard they opened first, where the runbook helped and where it misled them. Two published programs make good models. Google’s annual Disaster Recovery Testing (DiRT), which Kripa Krishnan described in 2012, and Slack’s Disasterpiece Theater share the habit that makes a game day worth its cost: each finding is followed until it is fixed, not merely written down.

Encore calls its quarterly game days tech rehearsals, after the theater’s practice of running every light, sound and scene change in sequence before opening night, so that the crew discovers the cue that does not work while there is still no audience in the seats. The analogy holds well in one respect: a rehearsal tests the crew and the equipment together, and it is the crew’s timing as much as the equipment that it exposes. It breaks in another. Once a show opens, its cues are largely fixed, and a good tech rehearsal stays true for the whole run. A production system changes with every release, so a rehearsal proves only that the system and its people worked on the day of the rehearsal. That is why Encore runs them every quarter instead of once, and why each one is designed around something that has changed since the last.

The tech rehearsal

Last quarter’s rehearsal is a good example of the form. It combined two failures that Encore had drilled only separately: a slow primary PSP, and a failover of one seat-ledger shard. It also served as the quarter’s evidence that the PSP breaker actually opens, the evidence Section 5.2 said only fault injection could supply. It ran in production on a Tuesday morning, a deliberately quiet time, against the hidden test event that the synthetic buyers use. For the occasion the test event was given four seat-ledger shards of its own, so that failing over one of them touched no real event’s seats. The experiment was written down in advance:

experiment: psp-latency-during-shard-failover
hypothesis: >
  Synthetic checkout success stays at or above 97% within the 3 s budget;
  the primary PSP breaker opens within 15 s; new payments move to the secondary;
  shard 2 failover rejects writes for under 60 s; no seat is sold twice.
target: test event only (tenant: encore-internal), home region
injections:
  - at: T+0     psp_latency: { psp: primary, added: 2s, share: 60% } # mesh fault
  - at: T+5m    shard_failover: { event: test, shard: 2 } # runbook step, by hand
abort_if:
  - real-event checkout SLO burn rate > 2x for 5m
  - any duplicate SeatSold for the test event
duration: 30m

Each part of that file does a job. The hypothesis is phrased in terms a fan or the finance team would recognize (checkouts succeed, payments move, no seat is sold twice), not in terms of internal metrics, because the point is to learn whether the experience survives. The injections are staggered: the PSP slowdown starts first and is injected by the service mesh, and five minutes later, once the breaker should already have done its work, an engineer fails over shard 2 by hand from the runbook, exactly as it would happen in a real incident. The abort conditions mark the edges of the blast radius. If real fans’ checkout starts burning its error budget at more than twice the sustainable rate for five minutes, or if a single seat in the test event is sold twice, the experiment stops at once, whatever it has or has not yet taught.

The machines did what the hypothesis predicted, and in some respects did better. Adding 2 s to 60% of PSP calls pushed those calls past their 800 ms timeout, so they counted as failures, and the breaker opened in 9 s, well inside its 15 s target. New payment attempts moved to the secondary PSP. The attempts that had timed out on the primary were, as Section 5.2 explained, of unknown outcome, since a charge might have gone through after the caller stopped waiting; reconciliation settled the last of them within four minutes of the injection ending. Because the secondary carried the load, no hold had to fall back to a pending payment. On the ledger side, the fenced failover of shard 2, in which the old leader’s term is fenced off so that it can no longer write once a new leader is chosen, rejected writes for 38 s against a target of under 60 s, and no seat was sold twice.

The people’s side of the rehearsal told a different story, and its two findings were the reason the morning was worth having. The first concerned the failover itself. Jonas, the Seat Inventory engineer who performed it, had never failed over a shard by hand, because the automation had always done it for him. The runbook he followed named a flag that had been removed two releases earlier, so the step took 11 minutes instead of 3. This is precisely the deskilling Bainbridge warned of: the automation had worked so reliably that the manual path, and the knowledge needed to walk it, had decayed without anyone noticing. The second finding was in the fan app. When the failover rejected holds, the app retried each rejected hold three times at fixed one-second intervals. That retry layer sat outside the service mesh, so the mesh’s retry budget never saw it, and its fixed intervals meant that clients rejected together retried together, in synchronized waves. Section 5.2 had argued for retries at one layer only, with jitter, and the app had quietly broken both rules. The app team added jitter and the client-side adaptive throttle described in Section 5.2.

One more line in the observers’ log deserves recording, though it seemed minor at the time. While the synthetic buyers were retrying their rejected holds, Bot Screening’s velocity rule, which looks for clients acting faster than people do, quarantined several of them. The log marked this as expected, since synthetic traffic does look synthetic, and the item was closed. Nobody asked the next question: what the same rule would make of a large crowd of real fans, frustrated by rejected holds, retrying in the same way.

That closing entry shows both the value of a game day and its limit. A game day finds out, cheaply and on a quiet morning, which of our resilience claims have never really been tested, and its observers will usually write down more than the organization is ready to act on. Whether a finding is fixed or merely closed is a choice the organization makes afterward, and so the Reliability team reports each quarter not only how many findings each rehearsal produced, but how many were fixed and how many were merely closed.

Recap. Chaos engineering treats each resilience claim as a hypothesis and tests it with a deliberate experiment, run in production within a bounded blast radius, but only once the means to observe the experiment and abort it are in place. Its richest questions concern failures in combination, which designs rarely anticipate. Game days add the people to the experiment, and they often show that the human side of recovery, the hand steps and the runbooks, has decayed faster than the automation that usually spares us from needing it.

Exercise 5.5. Design a game day that combines two failures you have so far drilled only separately. Write a hypothesis a user would recognize, set abort conditions that bound the blast radius, and name the human step you suspect nobody has practiced recently. Then decide who will follow each finding until it is fixed.

Kata: Kick-Off

Your brief: “Matchday.” Matchday is a sports-streaming service operating in five EU countries, built and run by 40 engineers. On a peak evening it serves 1.4 million concurrent viewers, and its load has a sharp shape: in the 10 minutes before a top-league kick-off, the rate of stream starts rises 30×, because fans open the app at the last moment. Every stream start needs a license from one of two digital rights management (DRM) providers, external services that decide whether a given viewer may decrypt a given stream. All competitions share one pool of playback-API instances, and the Sunday-evening derby alone takes 40% of that pool.

Last month one DRM provider slowed down. Its latency rose from 120 ms to 5 s and stayed there for 25 minutes. Three layers then retried at once: the player SDK on the viewer’s device, the playback API, and the fans themselves, restarting the app when nothing appeared. Together they quadrupled the volume of license requests. The shared pool saturated, and stream starts failed for every competition, including those licensed through the other provider, which had been healthy all along. The slow provider recovered at 20:31. Matchday itself recovered at 21:40, and some 210,000 viewers missed the whole first half.

The story has the shape this chapter has been teaching us to recognize. A single integration point, the slow DRM provider, became a cascading failure because the timeouts were too patient, the retries multiplied across layers, and the shared pool gave the derby and the slow provider a way to drag down everyone else. The most instructive number is the gap between 20:31 and 21:40. For 69 minutes the trigger was gone and the system stayed down anyway, which is the signature of the metastable failure of Section 5.3. The deliverables below ask you to take the incident apart in the order the chapter did, and then to design the rehearsal that would have caught it before a derby did.

Deliverables:

1. Integration-point register — Map the stream-start path from the player to the playback API and on to each DRM provider, as Section 5.1 did for Encore. For every call on that path, derive a timeout from a 4 s time-to-first-frame budget, the time a fan may wait between pressing play and seeing the picture. Divide the budget among the hops as Figure 5.2 divided Encore’s checkout, and leave a margin for queuing.
2. Retry and breaker design — Choose one layer to own retries, give it a retry budget and jittered backoff, and say what the other layers do instead. In particular, design throttling in the player SDK, because a million devices retrying in step are the largest retry layer Matchday has. Add a circuit breaker per DRM provider, and make it consistent with that provider’s timeout, so that the breaker opens on the slowness that last month’s provider actually showed and does not wait for errors that never came.
3. Shedding and metastability analysis — Design admission control for the pre-kick-off surge, and set the order in which work is shed when admission is not enough. Then name the three parts of last month’s metastable failure: the vulnerable state the system was running in, the trigger, and the sustaining effect that kept it down after the trigger was gone. Finish with the runbook text that would have ended the incident by 20:40, written for an on-call engineer who must act on it without hesitating, even though the action will look like an overreaction.
4. Blast-radius plan — Choose the rungs of Figure 5.5 that Matchday needs. The plan must achieve two things: the derby can no longer starve the other matches of playback capacity, and a stream that is already playing survives the loss of the providers that started it.
5. Game-day plan — Design one experiment in the spirit of Section 5.5. State a hypothesis that could turn out false, the failures you will inject, the conditions under which you will stop, and the human step you most want to watch: the moment when a person, not a machine, has to recognize what is happening and decide what to do.

Rubric:

Where you now stand

You can now trace a failure along the integration points it travels, derive timeouts from a latency budget instead of inheriting library defaults, and tune retries and circuit breakers so that they calm an incident rather than feed it. You can shed load deliberately, recognize a metastable failure by the way it outlives its trigger, choose the bulkhead whose size fits the failure you fear, and test all of it in a game day with people in the room, since people are half of every recovery. Every one of these techniques, however, rests on an assumption we have not yet examined: that the data underneath survives. A process can be restarted, a pod rescheduled and a cell rebuilt, but none of that brings back a lost write. Chapter 6 turns to state, and to disaster recovery practiced until it becomes routine.

References

• Michael T. Nygard — Release It!, 2nd ed.. Pragmatic Bookshelf, 2018.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering, chapters 21 and 22. O’Reilly/Google, 2016 — free online.
• Marc Brooker — Timeouts, retries, and backoff with jitter. The Amazon Builders’ Library — free online.
• Jacob Gabrielson — Avoiding fallback in distributed systems. The Amazon Builders’ Library — free online.
• Nathan Bronson, Aleksey Charapko, Abutalib Aghayev, Timothy Zhu — Metastable Failures in Distributed Systems. HotOS, 2021 — free online.
• Lexiang Huang et al. — Metastable Failures in the Wild. OSDI, 2022 — free online.
• Ben Maurer — Fail at Scale. ACM Queue, 2015 — free online.
• Kathleen Nichols, Van Jacobson — Controlling Queue Delay. ACM Queue, 2012 — free online.
• Jeffrey Dean, Luiz André Barroso — The Tail at Scale. Communications of the ACM, 2013.
• Clare Liguori — Automating safe, hands-off deployments. The Amazon Builders’ Library — free online.
• Google Cloud — Incident report, 12 June 2025. 2025 — free online.
• Colm MacCárthaigh — Workload isolation using shuffle-sharding. The Amazon Builders’ Library — free online.
• Becky Weiss, Mike Furr — Static stability using Availability Zones. The Amazon Builders’ Library — free online.
• Principles of Chaos Engineering — free online.
• Kripa Krishnan — Weathering the Unexpected. ACM Queue, 2012 — free online.

Chapter 6: Data, State, and Disaster Recovery

Code can be redeployed. Data can only be remembered.

Everything this book has said about change so far rests on a quiet promise: if a release goes wrong, we can take it back. Canaries, feature flags and rollbacks, the machinery of Chapter 4, all depend on that promise. A canary is safe because a bad canary can be withdrawn. A flag is safe because it can be switched off. A rollback is safe because the previous version is still in the registry, ready to run again. In every case the thing we change is code, and code is a description of behavior that we can swap for an older description whenever we like.

Data breaks the promise. A row that has been written cannot be unwritten by redeploying anything, because a redeploy replaces the program and leaves untouched everything the program has already done. Suppose a faulty release writes thousands of malformed records before anyone notices. Rolling back stops it from writing more, but the malformed records stay where they are, and the older version of the code must now live with them. Worse, the mechanism we rely on to protect data against the loss of a machine works against us here. Replication faithfully copies a corruption to every replica, so the error becomes exactly as durable as the truth it replaced. This chapter is about the part of production that remembers, and about how to make its memory survive three kinds of threat: our own mistakes, the failure of our machines, and the deliberate work of our adversaries.

The story every operations engineer eventually hears took place at Pixar in 1998, during the making of Toy Story 2. Someone ran a recursive delete, of the rm -rf kind, at the root of the film’s directory tree. Oren Jacob, then an associate technical director, later recalled watching the film disappear piece by piece, first Woody’s hat, then his boots, then Woody himself, until the team pulled the plug. By then most of the film was gone. The backups, which everyone assumed would cover exactly this case, had been failing unnoticed for some time. What saved the film was luck. Galyn Susman, the supervising technical director, had been working from home after the birth of her child, and she kept a copy on her home computer. The team drove that machine to the studio wrapped in blankets, restored from it, and then spent days finding out what the copy lacked.

The story is worth telling carefully, because it holds three lessons, and each returns later in this chapter. The first is that the deletion itself was ordinary: a routine command, run in the wrong place by someone doing normal work. That is the kind of failure Jim Gray had in mind in 1985, when he found that administration and software caused far more outages of Tandem’s systems than hardware did. We tend to build our defenses against the dramatic failure, the disk that dies or the data center that floods, while the commonest threat to our data is a person with the right permissions pointed at the wrong directory. The second lesson is that the formal plan had failed, and failed silently. Backups were being taken, as far as anyone knew, but nobody was checking that they could be read back, and a backup that cannot be read back is only a ritual. The third lesson is about what actually worked. The recovery came from a copy that was independent of the original system in every way that mattered: a different machine, in a different place, beyond the reach of the command that destroyed the original. And the people who restored it were improvising, because no one had ever practiced a restore. They succeeded, but improvisation under pressure is exactly the mode in which tired people make the next mistake. I have come to believe that a recovery we have never rehearsed is not yet a capability. It is a hope with a procedure attached.

Encore has far better machinery than Pixar had in 1998, and yet much the same exposure, because machinery protects only against the failures it was designed for. Consider four pieces of its state and practice. Its seat ledger, the event log in which Seat Inventory sells each seat exactly once, commits every write to a majority of replicas in three EU locations, so no single machine or site can lose a sale. That is strong protection against failing hardware, and none at all against a bad write, which the majority will accept as readily as a good one. Its Finance ledger, the relational books fed by every completed sale, is backed up nightly, and that backup has never been restored in full, which puts it closer to Pixar’s position than anyone at Encore would like. Its holds store is about to change shape under live traffic, the situation in which a half-finished change can leave data that neither the old code nor the new code understands. And its game days, the rehearsed failure exercises the team runs each quarter, have never failed over a whole region, so the largest recovery Encore depends on is also one it has never performed. By the end of the chapter all four will have been tested, and one of them will have failed in a way that teaches more than a success would have. (The map of Encore in Chapter 1 shows where each sits.)

6.1 Why Stateful Systems Are Different

The asymmetry of state

A stateful system is one whose correct behavior depends on what it has remembered from earlier requests, so that losing or corrupting that memory is a failure even when every process is healthy. The definition is worth reading twice, because its last clause is where the trouble lives. A stateless service is, in effect, a function: give it the same request and it computes the same answer, so we can replace it, restart it or roll it back to yesterday’s version and nothing of value is lost. A stateful service is a function plus a history. Its answer to “is seat 14 in row F still available?” depends on everything that has happened to that seat since the event went on sale, and that history runs in one direction only. Every pod can report healthy and every dashboard can be green while the system is still wrong, because what it remembers is wrong.

A touring production shows the difference well. The road crew strikes the stage after each show and builds it again in the next city, and nobody worries much, because the set can be rebuilt from its drawings as often as needed. The box office is another matter. Its record of who bought which seat cannot be rebuilt from anything; if it is lost, no amount of carpentry brings it back. Most of a modern system is stage set, and a small, precious part of it is box office. The analogy holds up to a point and then breaks in a way that teaches something: in software the drawings are themselves data, and a set we believed we could always rebuild turns out to be rebuildable only if someone kept the drawings and knows how long the build takes. We will come back to that before the section ends.

This one-directional quality shows up as an asymmetry between code and data, and the asymmetry appears in four places.

The first is rollback. A deployment can be reverted, because the old binary still exists and running it again restores the old behavior. A data change cannot be reverted in the same way. Once a migration has rewritten a million rows, and fans have bought seats against the new shape in the meantime, there is no previous version to return to; there is only a further, forward change that compensates for the first. Accountants learned this long ago, which is why a ledger is corrected by a new entry that offsets the mistaken one, never by erasure. The operational consequence is demanding: every migration must be safe if it is left half done, because a half-finished migration is a state we may have to live in for hours while we work out how to move forward. Section 6.3 turns that requirement into a technique.

The second is copying. Code is small, and moving it is nearly free. State is large, and copying it is bounded by physics: by disk throughput, network bandwidth and the time a database needs to replay its logs. Restoring 4 TB takes hours, however clever the tooling, and no orchestration makes the bytes move faster than the hardware allows. This is why recovering a stateful system is a matter of hours where recovering a stateless one is a matter of seconds, and why a recovery time objective for state has to be tested rather than assumed.

The third is corruption. When a stateless service has a bug, the bug fails requests while it is deployed, and fixing the code ends the failure. When a bug writes bad data, the damage outlives the bug. Every later reader of that data is misled, long after the faulty code has been replaced, until someone finds the bad rows and repairs them. That makes corruption the quietest kind of outage, because nothing crashes and no alert fires, and often the longest, because the clock starts with the first bad write and stops only when a person notices the consequences and traces them back.

The fourth is replication, and here the asymmetry turns a protection into a hazard. We replicate state so that losing a machine, a zone or a region does not lose data, and replication is very good at that job. But a replica copies whatever the primary writes, faithfully and within milliseconds, so it copies our mistakes too. A dropped table is dropped on every replica; a corrupting bug corrupts them all. Replication protects us from losing hardware and does nothing to protect us from ourselves, which is why the backups of Section 6.2 have to be something other than one more replica.

Taken together, these four asymmetries explain why the habits that serve us well with stateless services (deploy freely, roll back quickly, trust the replicas) become dangerous when applied to state, and why state deserves a discipline of its own.

Classifying Encore’s state

That discipline begins with knowing what we have. The first operational act for state is an inventory: a list of every place the system remembers something, what each memory is for, and what losing it would cost. Encore’s inventory is short, because most of Encore was made stateless on purpose, and most of its services can be restarted anywhere without losing anything that matters. What remains is the small, precious part, and Figure 6.1 lists it. Two of its columns use terms the Architecture book introduced: the recovery point objective (RPO) is how much recent data we can afford to lose, measured back in time from a failure, and the recovery time objective (RTO) is how long we can afford to be without the service.

Figure 6.1 — Encore’s state inventory. The third column matters most: derived data needs a tested rebuild rather than a backup, while systems of record need both a backup and a practiced restore.

Reading the table row by row shows how differently each store has to be treated. The seat ledger is the heart of the business: losing even one committed write could mean selling a seat twice, so its RPO is zero, and it is protected by the majority replication described in the chapter’s opening. Seat holds matter only briefly, since each one expires within minutes, so losing a few minutes of them inconveniences fans without corrupting anything. Orders can tolerate seconds of loss because the PSPs keep their own records of every charge, and any gap can be reconciled against them. The Finance ledger can tolerate more loss and more downtime than the purchase path, but its failure carries a different kind of cost: a failed nightly close and exposure to regulators. Each row is a separate promise with its own price, and a single company-wide “data protection policy” would overpay for some rows and underprotect others.

The third column, “System of record?”, is the one that does the most work. A system of record is the authoritative store for some fact: if it says a seat was sold, the seat was sold, and there is nowhere else to look it up. Derived data is computed from a system of record, as a cache, an index or an analytical copy, and can in principle be computed again. The distinction, which Kleppmann develops at length, takes one large and anxious question (“how do we protect our data?”) and breaks it into several small ones, each of which can actually be answered. For a system of record, the question is how we would restore it and how long the restore would take. For derived data, the question changes: where is its source, and how long does a rebuild take?

The seat-map cache is the instructive case, and it is where the stage-set analogy earns its keep. The cache needs no backup, because the seat-map geometry it holds is computed from data kept elsewhere and can always be computed again. What it does need is a rebuild whose duration is known, because while the cache is cold, requests that it would have answered fall through to slower paths beneath it, and the system is slow until the cache is warm again. “No backup needed” is easy to hear as “no risk”, and that would be a mistake; that rebuild duration matters, as the Operations book’s review of the Vesper Hale on-sale will show. I have come to believe that derived data is where teams are most often surprised, precisely because it is labeled harmless. A rebuild is safe to rely on only when two conditions hold: its source is retained, and its duration has been timed.

Recap. State moves in one direction only: we can revert code, but we can only compensate data with a further change. Copying state is slow because physics bounds it, corrupting it is quiet because nothing crashes, and replicating it spreads our mistakes as faithfully as our writes. An inventory that separates systems of record, which need a backup and a practiced restore, from derived data, which needs a retained source and a timed rebuild, is the first and cheapest act of control.

Exercise 6.1. Build Figure 6.1 for a system you know. Mark each store as a system of record or derived data, and for each derived store write down how it would be rebuilt and how long that would take. Which “derived” store turns out to have no source?

6.2 Backups That Restore — RPO, RTO, and Testing Recovery

What a backup is for

The Architecture book defined the two numbers that frame this section, and Section 6.1 used them to price each of Encore’s stores: the RPO looks back from a failure to the last data we can still recover, and the RTO looks forward from it to the moment service returns. An architect writes these numbers into a design. An operator has to treat them differently, as claims that nobody has yet tested. This section asks what has to be true for those numbers to hold on the day they are needed.

The first thing to get right is what counts as a backup. A backup is a copy of a system of record, held independently of it, from which a known past state can be restored within a stated time. Two words in that definition do most of the work. Independently rules out a replica. A replica protects us well against losing a machine, but it shares the system’s fate for logical damage: when someone drops a table or a bug writes nonsense, the replica faithfully applies the same change within milliseconds, as Section 6.1 warned. Within a stated time is what turns a copy into something we can plan around during an outage. A copy that might take a week to bring back is an archive. It may be valuable, but no one can promise service on the strength of it. The data-integrity chapter of Google’s Site Reliability Engineering puts the point plainly: nobody really wants backups; what people want are restores. A backup is only ever a means. The restore is the thing being bought.

Figure 6.2 — RPO measures back from a failure to the last good copy, RTO forward to service restored; here for the seat ledger and the Finance ledger, beside what the Finance drills of Section 6.5 measured. The promise was kept only once the team changed what had to come back first.

When we imagine needing a backup, we tend to picture a region burning down. That does happen, but the common threats are humbler, and they shape the design more. The first is a deletion in the wrong place, the ordinary mistake that nearly cost Pixar a film. The second is a bug that writes bad data quietly for a week before anyone notices. The third is an adversary who encrypts the database and then goes looking for the backups, so that the victim has nothing to restore from and must pay. Each threat asks something different of the copy. Slow corruption needs a copy from before the bug began, which means the backups must reach far enough back in time. Ransomware needs a copy the attacker cannot reach, which means the backups must sit outside the credentials the attacker has stolen. A burning region needs a copy somewhere else. No single mechanism meets all three demands, and the table below sets out the usual candidates side by side.

Reading down the “Weakness” column shows why every row needs another row to cover it, and why serious data protection is always built in layers. The SRE book starts with the cheapest layer, soft deletion. A deleted row is not removed at once but marked and kept for a grace period, so the common slip can be undone in minutes. Its operational cousin is the delayed replica, a replica deliberately held some time behind the primary (PostgreSQL’s recovery_min_apply_delay, MySQL’s SOURCE_DELAY). If someone drops a table, the delayed replica, an hour behind, still holds it, and we can copy it back before the drop reaches the replica too. The protection lasts only as long as the delay. Once the hour has passed, the replica has caught up with the mistake.

For corruption that runs longer than any delay, the workhorse is point-in-time recovery. We restore a physical base backup, a full copy of the database files taken at some moment, and then replay the write-ahead log, the database’s own record of every change, forward to an instant we choose. That might be 14:02 last Tuesday, just before the bad deploy began writing nonsense. Two limits come with this power. It reaches back only as far as the log is retained, often a week to a month, so a bug that goes unnoticed for longer than that cannot be undone this way. And it rewinds the whole database. Every good write made since 14:02 on Tuesday is discarded along with the bad ones, and someone must then reconcile them, finding the legitimate orders and edits of the intervening days and putting them back. Logical dumps, which export tables or schemas as data rather than as files, are slower but finer-grained, and that granularity can matter more than speed. At Encore each Professional tenant has its own database, so restoring one tenant’s database from a dump is far gentler than rewinding a cluster that many tenants share and making all of them pay for one tenant’s accident.

Three copies, two media, one elsewhere

Layers protect against different threats only if they do not all fail together. The oldest compact statement of that idea is the 3-2-1 rule, usually credited to the photographer Peter Krogh: keep three copies of your data, on two kinds of media, with one copy off site. It was framed for people keeping photographs on hard disks, yet it has aged remarkably well, because underneath the specifics it is a rule about independence of failure. Two kinds of media means one flaw in a disk model or a tape format cannot take every copy. Off site means one fire or flood cannot take every copy either. The numbers matter less than the question they force: what single event would destroy all my copies at once?

In the cloud the question survives but the answers move. “Medium” and “site” become accounts, regions and, above all, credentials. A backup stored in another region but deletable by the same role that administers the database is not independent of the database in the way that matters most. When an attacker steals that credential, they can destroy the database and its backups with the same hand. The most instructive real case runs the other way, showing what independence looks like when it happens by accident. When the NotPetya malware struck Maersk in 2017, recovery hinged on a domain controller in Ghana that a local power cut had kept offline. That copy survived because it was unreachable, and it was unreachable only by luck. Like Pixar’s home computer, it was the independent copy no one had planned. The lesson is to plan for one.

Encore plans for it this way. Backups of systems of record go to a separate account, into a vault under a compliance-mode retention lock. Compliance mode means that for 35 days nobody can delete a recovery point or shorten its retention. “Nobody” here includes Encore’s own administrators and the account’s root user, and no approval process can change that. The lock is deliberately beyond anyone’s reach, because any key that opens it could be stolen. Everything that must stay adjustable, from the backup plans and the vault’s access policy to the list of who may restore, sits instead in governance mode. Changes there are possible but require multi-party approval, so that no single person, and no single stolen credential, can quietly weaken the protection. The encryption key lives in the backup account as well. This detail is easy to miss and fatal to get wrong: a backup encrypted with a key that was destroyed alongside the source is a perfectly preserved copy that no one can ever read. A backup whose key died with the source is no backup.

Restore testing

Independence protects the copy from the disaster. It does not prove that the copy works. A backup has two failure modes, and only one of them is visible. In the first, the backup job fails, and an alert can catch that. In the second, the job succeeds and reports success, but what it produced cannot be restored, because it is empty, truncated, incompatible or unreadable. Only an actual restore catches the second failure, and it is the more dangerous one, because it looks exactly like health until the day it is needed.

GitLab’s outage of January 2017 is the modern case, and its public postmortem is unusually candid. Late at night, an engineer working to repair database replication wiped the data directory on the primary database instead of the lagging secondary. The team then turned to their backups and found them wanting, one after another. The daily dumps had been producing nothing, because of a version mismatch between the dump tool and the database, and the emails reporting that failure were themselves being rejected, so no one saw them. Disk snapshots had never been enabled for the database servers at all. What saved the company was a snapshot taken by hand six hours earlier, for a different purpose: refreshing a staging environment. Those six hours of data, touching some 5,000 projects, were lost, and copying the snapshot back took most of a day. GitLab’s postmortem traced the root cause not to the mistyped command but to ownership: nobody had been responsible for testing the backup procedures. Every mechanism existed on paper, and no person’s job was to find out whether any of them worked.

The echo of Pixar, nearly 20 years earlier, is hard to miss: backups failing unnoticed, a mistaken delete, and rescue by a copy made for some other reason. The past repeats because the failure is structural. Backups are written once and then trusted. Nothing in ordinary operation exercises them, so nothing reveals that they have quietly stopped working. One practice answers each of the two failure modes. The first is to alert on a backup’s absence, rather than waiting for a failure report, since a job that fails badly enough may never manage to report anything, as GitLab’s rejected emails showed. The rule below fires when the last successful backup of any system of record is more than 26 hours old, which is a daily schedule plus two hours’ grace for a slow run:

- alert: BackupStale
  expr: time() - backup_last_success_timestamp_seconds{kind="system_of_record"} > 26 * 3600
  for: 15m
  labels: {severity: page}
  annotations:
    summary: "No successful backup of  for over 26 hours"

The second practice addresses the invisible failure mode, and it is continuous restore testing: restores run on a schedule into an isolated environment, then verified against invariants. Verification is the part teams skip, and it is the part that matters, because a database that starts up can still be missing a day of data. The checks are things that must be true of a correct copy. Row counts should be close to the source’s. Totals that must balance, such as money in against money out, should balance. The latest transactions known to exist should be present. Encore restores the Catalog and every Professional tenant database weekly, checks each against its source and records how long the restore took.

That last measurement is worth dwelling on. The duration of a scheduled restore is a service level indicator (SLI), a measured signal of how the system is actually doing, and it gives the RTO something it otherwise lacks: evidence. Data grows steadily, and restore time grows with it, so an RTO that was comfortable when it was agreed can become impossible without anyone changing a line of configuration. A weekly restore duration that creeps upward month by month is the earliest warning that a promise is drifting out of reach, and it arrives while there is still time to act, by splitting the data, provisioning faster storage or renegotiating the promise. Section 6.5 shows what happens to a store that never had this signal.

A backup that has never been restored is a rumor about your data.

Recap. A backup is an independent copy that restores a known state within a stated time. Each threat, whether a slip, slow corruption, ransomware or a lost region, needs its own kind of copy, and the most important of them is locked, kept with its encryption key, and held in another account beyond the reach of any single credential. Watch backups for freshness by alerting on their absence, and prove them with scheduled, verified and timed restores, whose duration tells us whether the RTO is still true.

Exercise 6.2. For your most important database, write down the date of the last full restore, how long it took and who did it. If you cannot answer, schedule one. If you can, compare its duration with your stated RTO and with how much the data has grown since.

6.3 Schema Changes and Data Migrations

Two deploys and a promise

Section 6.1 argued that data runs in one direction only, and nowhere is that felt more sharply than when we change its shape. A schema change is a deployment whose effects cannot be undone by redeploying. If we ship a bad build, we roll back to the previous one and the world is as it was. If we drop a column, rename a table or rewrite a million rows into a new format, rolling back the code does not bring the old data back; the previous build simply wakes up in a world it no longer recognizes.

A second fact makes the problem harder. During any rolling deployment, two versions of the code run side by side: some pods have the new build, some still run the old one, and for minutes (or, with mobile clients and long-lived workers, for weeks) both are serving real requests against the same database. Both versions must work against the data at every moment. Consider the simplest change imaginable, renaming a column. If we rename it and ship the code that uses the new name in the same release, then for the length of the rollout every old pod is querying a column that no longer exists, and every one of those requests fails. Reverse the order and the new pods fail instead. There is no instant at which schema and code can change together, because there is no instant at which only one version of the code is running.

The answer, refined by many teams over many painful releases, is expand and contract (also called parallel change): a schema change made in stages, first expanding the schema so that it supports both the old and the new code, then migrating the code and the data, and only when nothing uses the old shape any longer, contracting it away. For the column rename, that means adding the new column alongside the old one, deploying code that writes to both and reads from the old, copying the existing values across, switching reads to the new column, and finally, in a separate later deploy, removing the old column and the code that still mentions it. It takes at least two deploys, and between them stands a promise: for as long as both shapes exist, both stay correct. Each stage is small, and each, until the last, can be reversed by a flag or an ordinary rollback. What we buy with the extra ceremony is the thing data otherwise denies us, a way back.

The civil engineer’s version is replacing a bridge without closing the road. We build the new span alongside the old, shift traffic across lane by lane while watching how the new structure carries it, and demolish the old span only when the last car has crossed. At every point before demolition, a problem with the new span means sending traffic back to the old one. The analogy teaches the order of operations well, and it breaks at the end, in an instructive place. Once the old bridge is gone, no driver needs it; nobody arrives tomorrow expecting to cross a span that was demolished last week. Software has readers with long memories. Caches hold values in the old shape, stream consumers replay messages written months ago, mobile clients update when their owners please, and at Encore the event-sourced seat ledger will read its past events in their original shape forever, because replaying those events is how Seat Inventory rebuilds its state. Contraction is therefore the one step that deserves real suspicion. It is safe only when we can show, with evidence rather than hope, that no reader of the old shape remains.

The holds store, migrated under traffic

Encore’s own example is the seat holds. A hold is the short reservation a fan receives on picking a seat, which keeps it off the market for five minutes while the fan checks out. Holds lived in a relational table keyed by (event_id, seat_id), and a sweeper job deleted expired rows every few seconds. The design was simple, but at on-sale scale the sweeper had become a hot spot: a job doing bulk deletes against the very table that fans were writing to as fast as the Sale Gate admitted them. The On-Sale and Seat Inventory teams decided to move holds to a key-value store with native per-key expiry, in which each hold carries its own time to live and simply disappears when it runs out, with no sweeper at all. Holds sit on the purchase path, so the migration could not take the path down, even briefly. It ran over three weeks, kept outside on-sale windows, in five phases:

Figure 6.3 — Expand and contract for the holds store. A flag or a deploy reverses each phase until the fifth, and the fifth waits until there is nothing left to reverse to. The thresholds are Encore’s own; no standard exists.

It is worth reading the table row by row, because each phase answers a different question. In the first phase, the expand, every hold was written to the old table as before and also, in shadow, to the new store: written there, but never read by anything a fan could see. The question was whether the new store could carry the load at all, and a week of clean writes answered it without a single fan depending on the answer. The second phase, the backfill, asked whether the new store held everything the old one did. The third, verify, kept the old table as the source of every answer but also read the new store and compared the two, asking whether they agreed. Only in the fourth phase did fans’ requests start being answered from the new store, and even then behind a flag that could send reads back to the old table in seconds, with both stores still receiving every write so that the old one stayed current enough to fall back to. The exit criterion for that phase was deliberately demanding: one full on-sale, Encore’s hardest traffic, served from the new store within its SLO. The fifth phase, contract, is the demolition. Old writers were removed, and even then the table was archived and kept for another 30 days before anyone dropped it, a last margin against the reader nobody had remembered.

A backfill, the bulk copy of existing data into a new shape, is usually the riskiest phase of any migration. It runs for a long time, sometimes days; it puts heavy load on a database that is also serving production; and it races live writes, so that a row copied at 10:00 may be changed at 10:01 by a request the copy never saw. The holds backfill avoided all of that for a reason that is easy to overlook: holds live for only five minutes. Once every write path was sending every new hold to both stores, any hold older than five minutes had already expired and needed no copying. After five minutes of dual writes, the new store contained every live hold, and the backfill was complete without a single row being copied. The general lesson is that knowing the lifetime of your data can turn the hardest part of a migration into a wait. It carries a proviso, though, and the proviso is where the danger lay: every write path must really write to both stores from the first minute. A path that was missed would leave holds in the old table that the new store had never heard of, and the five-minute wait would prove nothing.

That proviso was the real hazard, and it is the general hazard of dual writes, in which application code writes the same change to two stores one after the other. Without a transaction spanning both stores, and there is none between a relational database and a separate key-value store, one write can succeed while the other fails, and the two stores quietly drift apart. Encore handled this in the way Stripe described in its account of online migrations. The old table stayed authoritative throughout: whatever it said was the truth. A failed write to the shadow store was logged and counted but never failed the fan’s request, because the store being tested must never be allowed to hurt the store being trusted. And phase 3 existed to measure the drift directly, by comparing what each store returned for the same reads. The comparison found exactly one class of mismatch, and it was a good one to find before fans did. When a fan pressed “back” during checkout, the hold was released: deleted from the old table, but in the new store left to expire on its own. Had reads switched over with that bug in place, every seat a fan had let go would have stayed held, invisible to every other fan, for up to five minutes. On an ordinary day that is a nuisance; in the middle of a sold-out on-sale it is a great many seats briefly and wrongly unavailable. No test written before the migration had thought to look for this case, which is precisely why the verify phase is a phase and not a formality.

There is an alternative to dual writes, with a different set of costs. Change data capture (CDC) reads each committed change from the old store’s own log, the record a database already keeps of every write, and applies it to the new store. Because the application writes only once and the copy is derived from what actually committed, CDC removes the race between two independent writes. It has a price, and it is worth stating plainly. The new store lags behind the old one by however long the capture pipeline takes, so a read from the new store may not yet reflect a write a moment old. Changes are delivered at least once, which means that after a crash or a retry the same change may arrive twice, and so the consumer must apply them idempotently: applying a change a second time must leave the store exactly as applying it once did. And there is now a connector, one more running component, to operate, watch and recover when it falls over. That last cost is the one small teams most often underestimate, which is why Encore’s platform offers CDC on the paved road, so that no single team carries it alone. The choice between the two approaches is a genuine trade-off: dual writes are simple to start and drift silently, while CDC is consistent by construction and adds lag and machinery.

The same reasoning, keep the old shape serving while the new one is built beside it, also governs a humbler and more frequent kind of change, altering a large table in place. On many databases an ALTER TABLE on a big table takes a lock that blocks writes for as long as the change runs, which on a table of hundreds of millions of rows can be minutes, and minutes of blocked writes on the purchase path is an outage by another name. For large MySQL tables, tools such as gh-ost apply the expand-and-contract idea at the level of a single table: they build a shadow copy with the new structure, keep it in sync with ongoing writes, and swap it in at the end, so that no ALTER TABLE holds a lock longer than an SLO allows. PostgreSQL leans instead on operations designed to avoid long locks, such as building indexes concurrently, and on a short lock_timeout set on every migration, so that a statement which cannot get its lock quickly gives up and can be retried, rather than waiting in line and blocking every query that queues up behind it. Because the dangerous statements are easy to write and hard to spot, linters such as strong_migrations and squawk read migrations during review and flag the ones that would take a locking path. Each of these tools encodes, in a narrow form, the same lesson as the holds migration: change the shape of data only in steps that the system can survive while it keeps serving.

Recap. Schema changes cannot be reverted by redeploying, and old and new code always overlap during a rollout, so we expand before we contract and keep both shapes correct in between; that discipline is what gives us back the way back that data otherwise denies. A backfill is usually the riskiest phase, unless the data’s own lifetime makes it a matter of waiting. Dual writes drift silently, so we measure the drift before trusting the new store, or we trade it for the lag and upkeep of change data capture. Contraction comes last and deserves the most suspicion, because it is the one step with no way back.

Exercise 6.3. Take the next schema change on your team’s roadmap and write it out as a table like Figure 6.3. For each phase, name the exit criterion and the way back if that phase goes wrong. Which phase has no way back, and what must be proven, and by what evidence, before it starts?

6.4 Replication, Multi-Region, and Failover

Lag and what it means

Backups protect us against losing data slowly, through corruption or a careless delete; replication protects us against losing a machine, a zone or a region suddenly. The two are easy to confuse, and the quantity that tells them apart is lag. Replication lag is the delay between the moment the leader acknowledges a write and the moment a replica applies it. Under asynchronous replication, where the leader answers the client without waiting for any replica, the lag is not merely a performance figure. It is exactly the data a failover would lose: every write that the leader has confirmed but the replica has not yet received vanishes when the replica is promoted. If a replica runs two seconds behind, then two seconds of acknowledged writes are the price of losing the leader at that instant, and the store’s real RPO is whatever the lag happens to be at the moment of failure, whatever the documents say.

Lag has an unkind habit of growing at the worst time. It is largest when write volume peaks, because the replica must apply everything the leader accepted, and peaks are when systems are most stressed and trouble most likely to arrive. GitLab’s outage of 2017, which we met in Section 6.2, began in just this way, with a replica that had fallen too far behind to catch up and an engineer working late to repair it. For that reason lag deserves a place among the very few signals that should wake someone up. Most causes do not earn a page, because the symptoms they produce will be caught by an SLO alert anyway; lag approaching an RPO is different, because when it finally shows up as a symptom, the data is already gone.

Zero RPO needs a third vote

For most of Encore’s stores some lag is tolerable. The seat ledger is the exception. Its RPO is zero, because a lost write there is not a minor inconvenience: it is a seat sold twice, with two fans holding tickets for one chair. And the only way to lose nothing when a region fails is to acknowledge nothing until it is durable somewhere outside the region that wrote it.

The obvious design, and the one many teams reach for first, is a synchronous standby in the second region: the leader in the home region waits for the standby to confirm each write before telling the fan it succeeded. Following that design through to its failure shows why it is not enough. Suppose the network link between the two regions breaks. The home region must now choose, and both choices are bad. It can keep waiting for a peer it cannot reach, in which case writes stop and nobody can buy a seat. Or it can stop waiting and carry on alone, in which case the next write exists in one place only, and the zero RPO is abandoned at exactly the moment it was bought for. The standby faces its own dilemma on the other side of the break. From where it sits, a dead primary and a severed link look identical, since in both cases the primary has simply gone silent. If it promotes itself, it is guessing, and if the primary is in fact alive and still accepting writes, the guess produces the split brain described in the next subsection. Two sites cannot settle the question, because each sees only that the other has gone quiet. Telling a dead partner from a lost connection takes a third site with a vote.

This is an old problem in distributed computing, and the field’s answer is the majority. Each seat-ledger shard is therefore a consensus group: a set of replicas that commit a write only once a majority of them hold it durably, and that elect their leader by majority vote. Encore’s groups are Raft-style, in the manner Diego Ongaro and John Ousterhout described in 2014. The majority rule matters because two majorities of the same group always share at least one member, so two leaders can never both be elected, and any committed write is known to at least one member of whatever majority comes next.

Each group has five voting members spread across three EU locations:

• two in the event’s home region, EU-1 or EU-2, placed in different zones, where the leader normally sits;
• two in the other serving region;
• and a witness in a third region, EU-3: a replica that stores the log and votes but serves no traffic.

A write is acknowledged once three of the five hold it. In practice those three are the two home replicas and whichever remote member answers first, so every seat write pays for one cross-region round trip, about 18 ms between nearby EU regions. That sounds like a cost a checkout cannot afford, but against the 3 s checkout budget it is a small line. The harder question is throughput. If each write waited for its own 18 ms round trip before the next could begin, a shard that must keep writes in order could manage only about 55 of them a second. Group commit removes that ceiling: the leader gathers all the writes that arrive during one round trip into a single replicated batch, so the round trip is paid once per batch rather than once per write. That is how a shard sustains its 1,500 ordered writes a second while every one of them is durable in two regions.

With this arrangement in place, the failure cases resolve themselves by arithmetic rather than by judgment, which is exactly what we want of a mechanism that must work when nobody is watching. Take them in order of severity:

• A node or a zone is lost. Leadership moves to the other home replica, in the other zone, and the shard stays in its home region.
• The home region loses contact with one peer location, either the other serving region or the witness. The home region still holds its own two votes plus those of the location it can still reach, which is at least three of five. It keeps its majority, and fans notice nothing.
• The home region is cut off from both. Now it holds only two votes of five. Its leader governs under a lease, authority that lapses unless the majority keeps renewing it; the lease runs out and the leader stops accepting writes. Meanwhile the other region’s two members and the witness, together three of five, elect a new leader in the other region.

Consider what that last case delivers. Every acknowledged write was already on a majority, and every possible majority of the old configuration overlaps with the three survivors, so the new leader has every write the fans were told succeeded: the RPO stays zero. The RTO is the election plus re-routing the Gate and Orders to the new leader, a matter of minutes. And because a minority can never commit anything, the isolated home region cannot keep selling seats on its own, so there is no split brain to repair afterwards. The witness is a modest machine doing a modest job, but it is the one that lets the system tell “my partner is dead” from “I am alone”.

None of this is novel, and I find that reassuring rather than disappointing. Google’s Spanner, Amazon’s DynamoDB multi-Region strong consistency and Aurora DSQL all offer the same shape, two regions plus a witness, and CockroachDB states the same trade as a survival goal, letting the operator declare which failure the data must survive. When several independent teams arrive at the same design, it usually means the design follows from the problem rather than from anyone’s taste. The price is the same everywhere too: added latency on every write, and a third site to operate, patch and watch. Encore pays it only for the seat ledger, the one store where a lost write is a broken promise to a fan. Its other stores’ RPOs allow simpler asynchronous arrangements, and spending consensus on them would buy complexity without buying anything the business needs.

One limit, first met in Section 6.1, deserves to be restated plainly here. A majority protects Encore from the failure of machines and regions, but it cannot protect Encore from itself. If a bad deploy writes a wrong value, the consensus group faithfully replicates that write to all three locations in 18 ms, exactly as it would a correct one. Replication preserves whatever it is given, mistakes included. So the ledger still has point-in-time backups, which can return it to the moment before the mistake, and any corrupted projections, the read models derived from the ledger, can be rebuilt by replaying its event log, which only ever grows and so still holds the history the corruption overwrote.

Split brain and fencing

The consensus group prevents two leaders from being elected. It does not, by itself, prevent an old leader from acting as one. Split brain, failover’s characteristic failure, is the condition in which two nodes both believe they lead and both accept writes, so that the data diverges in ways that may never be reconciled. It happens more easily than one might hope. A leader cut off from its peers, or one frozen for many seconds by a long garbage-collection pause, cannot know that it has been replaced in the meantime. When it wakes or reconnects, it carries on with complete confidence, issuing writes on authority it no longer holds. It resembles a stage manager who steps out during the interval, misses the announcement that someone else is now calling the show, and comes back to call cues into a headset nobody is supposed to obey. The analogy breaks in one useful place: a theater crew would recognize the wrong voice, while storage, left to itself, has no way to tell a current leader’s write from a deposed one’s. Since the old leader may never hear of the handover, the protection must sit with the storage it writes to, which is the one party every write must pass through.

That protection is fencing. Every leadership term carries a number that only ever increases, called a fencing token, and storage rejects any write bearing a term older than the newest it has seen. The idea draws on two lines of work. One is the leases Cary Gray and David Cheriton described in 1989, grants of authority that expire unless renewed, which bound how long a leader can believe in itself without confirmation. The other is the numbered terms of consensus protocols, which give each reign of leadership a place in a strict order. On Encore’s Seat Inventory shards, a new leader’s first act is to record its term with the storage. From that instant, any write from the old leader carries a smaller number and fails, loudly and harmlessly: loudly, because the old leader receives an error it cannot ignore, and harmlessly, because nothing it attempted was applied.

It is worth separating what fencing costs from what a takeover costs, because they are easy to blur. Fencing itself costs almost nothing, a single comparison of two numbers on each write. The expensive part of a takeover is the wait for the old leader’s lease to run out, tens of seconds in which the shard accepts no writes at all. That wait exists so that the old leader has had time to notice it has lost its authority and step down. But it relies on clocks, and clocks on a stalled or overloaded machine can misjudge how much time has passed. Fencing is what makes the wait safe: if a deposed leader’s clock tells it the lease is still good when it is not, its writes are refused anyway.

Figure 6.4 — Fencing on a Seat Inventory shard. The deposed leader is never asked whether it still leads; the storage simply refuses its stale term, and the refusal is how it finds out.

Failing over a region

The seat ledger is only one part of what must move when a region fails. Here an earlier decision pays off. Because Encore pins each event’s writes to a home region, a regional failure is not one enormous event but many small ones: each affected event must have its home moved, one at a time and independently. The ledger’s part, as we have seen, needs no human action, since the consensus groups re-elect their leaders on their own. The rest of the work is moving each event’s home and redirecting the Gate and Orders to the new leaders. Encore automated all of that as a single operation, so nobody has to type a sequence of commands under pressure at 3 a.m., but it deliberately kept the trigger human: a person decides that a region has failed, and the machinery then carries out the decision.

That may look like timidity, given how much else is automated. The evidence for it is GitHub’s outage of 21 October 2018. During routine maintenance, connectivity between GitHub’s US East Coast network hub and its primary data center broke for 43 seconds. In that short window its failover tooling, Orchestrator, did exactly what it was built to do: seeing the primaries unreachable, it promoted database primaries on the West Coast. When the link returned, some writes existed only in the East and others only in the West, two histories that had diverged in under a minute, and the site ran degraded for just over 24 hours while engineers reconciled them. Every component had behaved as designed. What failed was the assumption that a 43-second interruption and a lost data center should be met with the same response. GitHub’s remediation was to stop Orchestrator promoting primaries across regions.

The lesson I draw from it is about matching the reflex to the stakes. Within a region, automated failover is right: the failures are frequent, the distances short, and a fast mechanical response beats a slow human one. Across regions the arithmetic flips. Real regional failures are rare, brief network flaps are not, and a failover fired by a flapping link turns a momentary blip into a self-inflicted outage with the largest blast radius a company has. A person who takes a few minutes to confirm that a region is truly gone costs little against that risk, provided the steps that follow are rehearsed and automatic, which is what Section 6.5 sets out to test.

Recap. Under asynchronous replication, lag is the RPO: whatever has not yet reached the replica is what a failover loses. A zero RPO needs a majority across three sites, because two sites can only guess whether the other is dead or merely unreachable, and even a majority faithfully copies our own mistakes, so backups and replay remain necessary. Fencing, enforced by storage rather than trusted to leaders, is the cure for split brain. Region failover should be automated in its steps and human in its trigger.

Exercise 6.4. For one replicated store you run, find out what prevents an old leader from writing after a failover. If the answer is “it will notice it has been replaced,” write down how long that noticing takes and what the old leader could write in the meantime.

6.5 Disaster Recovery as Practice, Not Plan

The plan on the shelf

Most organizations have a disaster recovery (DR) plan; far fewer have a DR capability, and the gap between the two is the subject of this section. A DR plan is a document that says how an organization will restore service after losing a system, a site or a region. A DR capability is the demonstrated ability of particular people, with particular tools and permissions, to actually do it within the time promised. The difference is the one between an evacuation map and a fire drill. The map is drawn once, correctly, and then the building changes around it: walls move, a stairwell is locked for renovation, the people who knew where the second exit was leave. Nothing on the map announces that it has gone stale. Only a drill, with real people walking real corridors against a clock, reveals that the route now ends at a locked door. The analogy holds better than most, though it breaks in one place that matters to us. A building changes slowly and visibly, while a software system changes every week and mostly out of sight, so a DR plan goes stale far faster than any evacuation map.

Regulators understood part of this long ago. Banking, where I spent much of my career, requires DR tests by rule, and the rule does its job of ensuring that a test happens on schedule. Whether the test teaches anything is a separate matter, and it depends on something no regulation can require: whether people are allowed to fail. A test that must pass will be designed to pass, rehearsed along the easy path and declared a success, and the organization learns nothing it did not already believe. Only a test that is allowed to fail can find the locked door.

In the vocabulary of this book, a DR drill is a game day of the kind Chapter 5 described, aimed at a narrower and sharper question: can these people, with this tooling and these permissions, rebuild this system within the RTO someone promised? Section 6.2 treated the RTO as a claim awaiting a test; a drill is that test. Drills also counter one of Bainbridge’s ironies of automation, an irony that bites hardest in DR. The better the failover machinery, the more rarely anyone has to recover by hand, and so the less practiced anyone is on the day the machinery itself fails, which is precisely the day a manual recovery is needed. Encore’s seat ledger fails over so reliably that hardly anyone remembers how to do it otherwise. Good automation, left alone, quietly deskills the people who stand behind it, and a drill is how we keep them in practice.

All of which leads to a conviction I hold firmly. An RTO is a hypothesis until a stopwatch has run against it.

The Finance ledger drill

Encore put that conviction to work in the spring, when the Reliability team ran the company’s first full restore of the Finance ledger. Recall from Section 6.1 that the Finance ledger is the relational database of Encore’s books, fed by every SaleCompleted event, reconciled daily and closed nightly. It had been backed up every night and never once restored in full. By the spring it held about 4.2 TB, and its RTO of two hours had been agreed when the database was a tenth of that size: a promise made about a smaller system, never revisited as the system grew. The scenario was realistic and unglamorous. A data corruption is discovered at 09:00, and the repair is to restore the ledger to its state at 02:00, just after the nightly close completed. The team wrote down the planned duration of each step before starting, then recorded what actually happened.

Figure 6.5 — The Finance ledger restore drill. No delay came from the database engine.

The drill missed its RTO more than fourfold, and it was the most useful day of Encore’s quarter. That is less of a paradox than it sounds. Every hour of overrun was an hour Encore would otherwise have discovered during a real corruption, with Finance unable to close its books and the regulatory exposure that brings; the drill bought the same knowledge on a calm spring day, at the price of one team’s time. The caption of Figure 6.5 carries the first lesson. Not one minute of delay came from the database engine, which was the part everyone had worried about. The delays came from the people, permissions, capacity and preparation that surround the engine, and it is worth taking them one at a time.

The first delay was in reaching the backups at all. As Section 6.2 described, Encore keeps backups of its systems of record in a separate account, where no single credential can weaken them. Opening that account for a restore needs two approvers. The rule was right; it had simply never been tried. One approver was on leave, and once access was finally granted, the policy on the encryption key that protects the backups turned out not to grant the account the drill was using. Neither problem was exotic, and neither could have been found by reading the plan. The fixes were equally plain: a standby approver rota, so that two approvers can always be reached, and a key policy that grants the recovery account, so that the people restoring the data can also decrypt it.

The second delay was physics, and physics does not yield to runbooks. The restore was limited by the provisioned I/O of the volume it was writing to, a volume sized for the ledger’s normal daily running and never for the sustained bulk write of a full restore. A short calculation from Figure 6.5 shows how firm that limit was. Restoring 4.2 TB took 3 h 10 min, about 11,400 seconds, which works out to roughly 370 MB a second. At that rate two hours of restoring brings back only about 2.6 TB, and that is before log replay, verification and repointing have had a single minute. Whatever the runbook said, the whole ledger could not come back in two hours. A faster volume would have helped at the margin, but the deeper problem was that the RTO treated years of accounting history and the current year’s sales as one undifferentiated mass that all had to return at once.

So the team changed the shape of the problem. It split the ledger so that the current financial year, about 600 GB, restores first and becomes writable on its own while the history follows behind, and for that current-year slice it added a warm standby, a copy kept continuously up to date and ready to take over. Then someone asked Finance the question that should have been asked when the original RTO was set: what do you truly need, and by when? Finance’s answer was precise. It needed to run the next nightly close, which means the current year back within two hours; the history could wait a day. The two-hour promise had never been wrong in spirit. It had been attached to the wrong unit of data, and decomposing the ledger by what each part was for turned an impossible promise into two achievable ones. Three months later a second drill had the current year writable in 1 h 50 min. Figure 6.2, in Section 6.2, tells the same story in terms of RPO and RTO: the promise was kept only once the team changed what had to come back first.

Verification taught the subtlest lesson, the one Pixar learned in the days after its rescue, when the team had a running copy of Toy Story 2 and still had to discover, file by file, what that copy lacked. A restore is finished when someone with the authority to say so can declare that the data is right, and a database that has started and answers queries proves very little about that. For the Finance ledger, “right” means that its records agree with the settlement reports of Encore’s two PSPs and with the stream of SaleCompleted events up to 02:00. Before the drill nobody had written down how to check either, so engineers spent two hours and twenty minutes composing verification queries under time pressure, which is the worst possible moment to be writing careful queries against a company’s books. Afterwards the team kept those queries, refined them and began running them every week against the scheduled restores of Section 6.2. The work did not disappear; it moved out of the emergency and into an ordinary week, where it could be done calmly and checked. Written in advance and exercised weekly, the queries turned two hours of improvisation into 10 minutes of reading a report.

The last delay, a reporting job whose connection string had been hard-coded to the old database address, is the smallest in the table and perhaps the most typical. Every system accumulates quiet assumptions of this kind, each one harmless until the day the thing it assumes stops being true. A drill is how we find them while finding them is still cheap, and Encore’s next exercise found a whole family of them.

The region failover game day

The Finance drill tested one store. The largest test of the year tested a region. It came on a quiet October Tuesday, chosen because no on-sale was scheduled within a week, so that any surprise would land far from a night when fans were queuing. The Reliability team cut EU-1 off from both EU-2 and the EU-3 witness, isolating it as a severe network failure would, and an engineer then triggered the failover of every event whose home region was EU-1. Section 6.4 explained why Encore automates the steps of a region failover but keeps the trigger human; here a human pulled it deliberately. To make the test honest without putting a major sale at risk, small events carried real traffic, real fans buying real seats, alongside synthetic buyers that exercised the paths no real fan happened to take.

The core held, and it held exactly as Section 6.4’s arithmetic said it would (Figure 6.6). In each seat-ledger shard, the two replicas in EU-1 now commanded only two votes of five, so their leaders lost their majority, let their leases lapse and stopped accepting writes. In EU-2, the two replicas there gathered the witness’s vote to make three of five and elected new leaders. Fencing then rejected a handful of late writes from the deposed EU-1 leaders, loudly and harmlessly, as designed. The whole failover took about four minutes, and no hold or sale was lost. It is worth pausing on how satisfying, and how uncommon, such a result is: a mechanism designed on paper, exercised under a realistic failure, behaving precisely as its design predicted.

The surprises were at the edges, in services around the ledger that had never been asked to live anywhere but EU-1. The first was in the Sale Gate, the virtual waiting room in which fans queue for an on-sale. Its pre-warm, the preparation that raises provisioned capacity before a sale so that the first fans do not wait for new instances to start, had only ever been done in EU-1, because every on-sale until then had been homed there. So the first buyers in EU-2 met cold starts, the same slow opening seconds that a skipped pre-warm step had once caused, as the Operations book recounts. The second was in Bot Screening, whose risk data the Gate uses to judge which buyers are bots. That data reached EU-2 only by a daily copy, so the Gate in EU-2 screened fans against yesterday’s signals, blind to anything learned since. The third was an Orders reconciliation job whose schedule named EU-1 explicitly; with EU-1 cut off, it simply did nothing for six hours.

None of the three broke the failover, and each would have made a real one worse: slower for the first fans, weaker against bots, quietly behind on reconciling orders. They share a single cause, which Figure 6.6’s caption names. Each was a thing that had assumed EU-1 would always be home. The consensus design had been built from the start to survive losing a region, while the services around it had grown up in a world where that region never went away, and so the system was only as ready for a failover as its least-examined assumption. Each finding became a ticket with an owner, following the discipline of Chapter 5 that a game day is worth what is fixed after it.

Figure 6.6 — The October game day: EU-1 isolated, and EU-2 taking over with the votes of the EU-3 witness. The core held; the three findings sat at the edges, in things that had assumed EU-1 would always be home.

One more choice made these drills worth more than their findings, and it concerns who leads them. A DR runbook is good when someone who has never performed the procedure can follow it at 3 a.m., because that is very likely the person who will be holding it when a real disaster comes. A runbook led every time by the engineer who wrote it is never really tested, since its author fills each gap from memory without noticing that the gap is there. So Encore’s own rule, a step beyond the rotation of Google’s DiRT exercises that Chapter 5 introduced, is that each drill is led by someone who has not led one before, with an experienced engineer shadowing to keep the exercise safe. The theater has long worked this way: an understudy is not truly ready until they have played the role on the stage, under the lights, with the cast around them, and a company that only ever lets the lead perform has a single point of failure in its cast. The analogy is imperfect, because an understudy performs a fixed script while a DR lead must improvise whenever the system departs from the runbook, but that is all the more reason for the rehearsal to be real. How a genuine disaster is commanded, once it is no longer a drill, belongs to incident response, which the Operations book takes up.

Recap. Disaster recovery is a capability that exists only where it is practiced: a plan on a shelf records intentions, while a drill measures what people can actually do. Drills time real recoveries against the RTOs that were promised, and their findings concern permissions, throughput, verification and the assumptions at the system’s edges far more often than the database engine. A restore is finished only when someone with authority can say the data is right, and a runbook is proven only when someone new to it can follow it.

Exercise 6.5. Plan a restore drill for one system of record. Write down the scenario, the target state you will restore to and the RTO you have stated, then list every permission, key and person the restore needs, including who stands in when each person is away. Before you run the drill, predict the duration of each step and write down how you will verify that the restored data is right; afterwards, compare your predictions with what happened, and give each gap a ticket and an owner.

Kata: The Stored-Value Card

Your brief: “Tidewater Transit.” Tidewater runs fare payment for a regional transport network. Its riders carry 2.4 million stored-value cards, each holding a balance that the network spends down one fare at a time. Some 3,000 validators on buses and trams read those cards. A bus is not always within signal, so each validator buffers its taps while offline and uploads them when it reconnects. Riders add money through a top-up app.

Every balance lives in one 9 TB PostgreSQL primary, with an asynchronous replica in a second region. In the morning rush, when the most taps and top-ups arrive at once, that replica lags several minutes behind. Backups are a nightly logical dump that takes seven hours to produce and has never been restored in full, so no one knows how long a restore would take. The regulator now asks for an RTO of four hours and an RPO of five minutes for balances, and neither figure has yet been checked against anything Tidewater has measured.

Tidewater has already had one warning. Last spring an engineer dropped a partition of tap history, and the team rebuilt it from the validators’ own logs over two weeks. Next year brings a larger change: account-based ticketing, in which fares are charged to a rider’s account rather than deducted from the card, will reshape the balance model under live traffic. Tidewater has 14 engineers to do all of this, and one database specialist.

The brief resembles Encore’s story in shape and differs from it in detail. Tidewater has a single relational store where Encore has a consensus-replicated ledger, a nightly logical dump where Encore has layered backups, and a restore no one has tried where Encore now has two drills behind it. Work through it in the order the Reliability team followed at Encore: the inventory first, then the backups, then the migration, then failover, and finally a drill that tests the rest.

Deliverables:

1. State inventory — classify every store as a system of record or as derived data, in the shape of Figure 6.1, with its RPO, its RTO and the path by which it would be restored or rebuilt. Decide in particular where the validators’ buffered taps belong, since last spring they were the source from which lost history was rebuilt.
2. Backup and restore design — the layered mechanisms, each chosen against a threat that actually occurs (a dropped partition, slow corruption, ransomware, the loss of a region), and where each copy lives. For every copy, state which credentials could delete it. Then give the restore-testing schedule and the SLI that times each restore, so that the RTO becomes a measured figure rather than a hopeful one.
3. Migration plan — expand and contract for account-based ticketing, as Section 6.3 did for Encore’s holds store. Give the phases and the exit criterion for each, the backfill strategy for the 2.4 million existing balances, and how any drift between the old and new models during dual writing will be measured or avoided.
4. Failover design — lag alerting set against the five-minute RPO, so that someone is paged before the lag consumes the whole budget; how promotion of the replica is fenced, so that the old primary cannot go on writing; and who decides to pull the trigger.
5. First DR drill — the scenario, a predicted duration for every step written down before the drill begins, the verification queries that will prove the restored balances are right, and who leads it.

Rubric:

Three of the rubric’s questions repay a moment’s thought before you begin. The restore arithmetic is where Encore’s Finance ledger drill came apart, and nothing in Tidewater’s brief suggests its numbers will be kinder: a dump that takes seven hours to produce gives little reason to expect a restore in four. The fencing question asks for a mechanism that lives with the storage, because, as Section 6.4 showed, a deposed primary cannot be trusted to notice that it has been deposed. And the practice question expects failure on purpose. A first drill that passes cleanly has probably tested too little, while one that misses its target by hours, as Encore’s first Finance drill did, shows exactly where the work lies.

Where you now stand

You can now take an inventory of a system’s state and say which stores are systems of record and which can be rebuilt from something else. You can design backups for the threats that actually occur rather than only the dramatic ones, and migrate data under live traffic without losing the way back. You can reason about replication lag, quorums and fencing well enough to say what a failover will lose and what stops two leaders from writing at once. And you can turn a disaster recovery plan into a capability that is timed, practiced and led by people new to it. One threat in this chapter differed from all the others. The misplaced delete, the slow corruption and the severed link were accidents; the adversary who hunts for your backups before encrypting your data is not. Chapter 7 turns to that adversary, and to security as an operational discipline.

References

• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering, ch. 26. O’Reilly/Google, 2016 — free online.
• Martin Kleppmann, Chris Riccomini — Designing Data-Intensive Applications, 2nd ed.. O’Reilly, 2026.
• GitLab — Postmortem of database outage of January 31. GitLab blog, 2017 — free online.
• GitHub — October 21 post-incident analysis. GitHub blog, 2018 — free online.
• Diego Ongaro, John Ousterhout — In Search of an Understandable Consensus Algorithm. USENIX ATC, 2014 — free online.
• Cary G. Gray, David R. Cheriton — Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency. SOSP, 1989 — free online.
• Stripe — Online migrations at scale. Stripe blog, 2017 — free online.
• Kripa Krishnan — Weathering the Unexpected. ACM Queue, 2012 — free online.
• Jim Gray — Why Do Computers Stop and What Can Be Done About It?. Tandem Technical Report 85.7, 1985 — free online.

Chapter 7: Security Operations — Secrets, Patching, and the Supply Chain

“Security is a process, not a product.” — Bruce Schneier

A system can be designed securely and still be run insecurely. The two are different achievements, and it is easy to mistake the first for the second. The Architecture book did the design work for Encore: it drew the trust boundaries, decided which parts of the system may talk to which, and gave every workload an identity of its own. That work matters, but a diagram is a statement of intent, and intent does nothing by itself. A diagram does not rotate a key when the key gets old. It does not rebuild a container image when a flaw is found in a library inside it. It does not notice that a partner’s credential, which has always been used from the same few places, is suddenly being used from a country it has never come from before. Those things happen only because someone, or some machine that someone set up and still watches, does them. This chapter is about that work. It asks what security looks like as daily practice: who does it, how often, with which instruments, and how we can know it is still being done on the days when nobody is watching. That last question is the hardest of the four, and I have come to believe it is the one that separates a security program from a security document.

The question is much older than the cloud. On the evening of 2 November 1988, a program written by Robert Tappan Morris, a graduate student at Cornell, was released onto the young internet from a machine at MIT. Within a day, thousands of computers had slowed to a crawl. What makes the episode instructive is how ordinary it was. The worm used no exotic technique and no secret weapon. It walked through doors that administrators had left open in the normal course of running their machines: conveniences left switched on, software left unpatched, trust granted between machines and never taken back. The people who fought it had no team to call and no playbook to follow, so they improvised, comparing notes over mailing lists and working out together what the program did and how to stop it. That improvised, collective response was the first real attempt at what we now call security operations: the continuing, routine work of keeping a running system’s defenses current, and of noticing and responding when they are tested. We will look closely at the worm’s four doors in Section 7.1, because nearly every lesson in this chapter is already present in them.

Encore, for once, begins a chapter in reasonable shape. Its services carry cryptographic workload identities, so each service can prove which service it is rather than being trusted because of where on the network it sits. Its service mesh enforces mutual TLS, so traffic between services is encrypted and both ends verify each other. And its golden path, the paved road that most teams use to build and ship, is able to sign container images, so that what runs in production can be traced back to a build Encore trusts. The architecture, in other words, is sound. What Encore lacks is the habit of exercising it. Nobody yet rotates, patches, revokes and rehearses as a matter of routine on an ordinary Tuesday. That habit is what this chapter sets out to build, and the reason is practical: when the extraordinary Friday arrives, with a leaked key or a critical vulnerability across the whole fleet, the response will be only as good as the routines the teams have already practiced. A drill rehearsed during a quiet week is something people can carry out under pressure; a procedure read for the first time during an emergency usually is not.

7.1 Security as an Operational Discipline

Four open doors

The worm had four ways into a machine, and each holds one of this chapter’s lessons in small and legible form, so it is worth taking them one at a time. The first was the debug mode of sendmail, the program that delivered the internet’s mail. Debug mode was a convenience for administrators, and in many installations nobody had switched it off; the worm used it to persuade the mail system to run the worm’s own code. The second was a buffer overflow in fingerd, the small service that answered questions about a machine’s users. fingerd read its input into a fixed-size space in memory without checking the input’s length, so a carefully overlong request could spill past that space and overwrite the instructions the program would run next. The third door was rsh and rexec, the remote shell services, through which one machine could let users arriving from another machine it trusted log in without a password. Once the worm held one machine, every machine that trusted it was open as well. The fourth was the plainest of all: the worm guessed passwords, trying a short dictionary of common words, and on a network of busy people that was often enough.

What turned this program into an event was a single decision about how it should behave once it arrived. Morris seems to have intended the worm to spread quietly. Before installing itself, the worm asked each host whether it was already infected, but it ignored a “yes” about one time in seven and installed another copy anyway. The arithmetic of that choice is unforgiving. A machine on a busy network would be found again and again by its neighbors, and each visit had a one-in-seven chance of adding another copy, so copies accumulated on every host faster than anyone could clear them, and the machines slowed until they were of no use to their owners. The quiet crawl Morris may have imagined became a flood across perhaps 6,000 machines. Eugene Spafford’s analysis at Purdue remains the clearest account of how it all worked. The response tells us as much as the attack does. Many sites cut themselves off from the network, which stopped the worm but also stopped the fixes, since those fixes were traveling over the same network. Within weeks DARPA funded the Computer Emergency Response Team at Carnegie Mellon, now CERT/CC, so that the next time there would be someone to call. That is the first lesson of security operations, learned in its first week: when defense depends on coordination, the people coordinating need a place to meet that the attack cannot take away.

The four doors are worth tabulating, because each one has a living descendant in the systems we run today. The table reads from left to right as a line of inheritance: what the door was in 1988, the general kind of weakness it represented, the form that same weakness takes now, and the routine that keeps it closed, with the section of this chapter that develops each routine.

Figure 7.1 — The Morris worm’s four doors and their modern descendants. Each door was opened by ordinary operation rather than by bad design, which is why each needs an operational answer.

The adversary is part of the environment

Look again at the second column of Figure 7.1, and notice what is missing from it. None of the four doors was an architectural failure in the sense of the Architecture book, where a poor boundary lets one failure spread across a system. Each door came instead from the ordinary business of running machines day after day: a convenience enabled for a good reason and never disabled, a binary installed once and never updated, a trust granted between two machines for a busy weekend and kept for years because nothing ever forced anyone to revisit it. Nobody designed those weaknesses. They accumulated, the way dust accumulates in a building nobody is assigned to clean. That is why the answers in the last column are routines rather than designs, and it is why this chapter treats security as something a team does every week rather than something an architect draws once.

There is, however, one thing that sets security apart from the reliability work of the earlier chapters, and it is the adversary. In the chapters on capacity, resilience and recovery, the forces arrayed against us were indifferent. A failing disk does not study our defenses, and a traffic spike does not change its shape because we added a cache. An attacker does both. A defense that works this month also teaches its opponent what to try next month, so every success is partly a lesson handed to the other side. This makes security a contest that never reaches a stable end, and the right posture toward it is a practice kept up indefinitely rather than a wall built once and left to stand.

The familiar image of security as a fortress, with high walls and a guarded gate, gets this wrong, because a fortress is finished when its walls are built. I have come to believe that a hospital’s infection control is the more faithful analogy. Infection control is not a building. It is a set of routines (hand washing, sterilization, isolation, surveillance of which infections are appearing on which wards) performed constantly by ordinary staff rather than by a special squad, and tightened deliberately whenever a new organism arrives. Its successes are invisible, since what it produces is the infection that did not happen, and it decays the moment people stop performing it. All of that is true of security operations. The analogy fails at intent. Bacteria evolve, but they do not choose; they do not time their attack for a hospital’s busiest night. Human adversaries do exactly that, and for a company like Encore the busiest night is precisely the one they will choose.

Encore knows this adversary personally. Bot Screening, the service that decides which sessions in the online waiting room belong to humans and which to automated buyers, tunes its rules weekly, because the people it screens rehearse. They try their tools on minor on-sales, where little is at stake and much can be learned, and bring what works to the major ones. Bot Screening is therefore never finished, in exactly the way infection control is never finished. One caution follows from this, and the Operations book’s account of the Vesper Hale on-sale makes it vivid. A rule that judges behavior is only as current as the behavior it was tuned on, and adversaries are not the only ones whose behavior changes; fans change how they behave too, and a rule tuned against last month’s bots can mistake this month’s anxious fans for an attack.

If security is a routine practiced every week, then the natural question is who practices it. Encore organizes as Google’s Building Secure and Reliable Systems suggests, treating security and reliability as one discipline rather than as two departments with separate goals. A Security Engineering team of six sits in the platform group. It owns the security planes of the paved road, the identity, secrets and signing services that every team builds on, and it runs detection; it does not act as a gate that every change must pass through. The everyday chores of security, such as keeping an inventory of what runs where, rotating credentials and applying patches, belong to the teams that own the services. Those chores sit on the same scorecard as the teams’ service level objectives (SLOs), so a stale credential is measured and discussed the same way as a missed latency target. The trade-off in this arrangement deserves to be stated plainly. Spreading security across every team puts the work where the knowledge is, since the team that runs a service knows best which credentials it holds and which libraries it depends on. But the arrangement holds only while the platform really supplies the secure defaults, so that doing the right thing is also the easiest thing. Without those defaults, security that is shared by every team tends, in practice, to end up owned by none.

Recap. The Morris worm entered through four ordinary doors: a convenience left on, an unpatched bug, trust granted by network location and a guessable password. Each has a modern descendant, and each is closed by a routine kept up over time rather than by a single design. Because the adversary adapts, security is a continuous practice, owned by the teams that run the services and made workable by a platform that ships secure defaults.

Exercise 7.1. For a system you run, fill in Figure 7.1’s last two columns: name today’s descendant of each of the four doors in your system, and the routine (with how often it runs) that keeps that door closed. Which door has no routine at all?

7.2 Identity, Secrets, and Least Privilege in Practice

Credentials that expire on their own

Of the worm’s four doors, the two that matter most here are the trusted host and the guessed password, because both were failures of identity: the machines at the far end could not tell who was really knocking, and once they had decided to trust someone, they trusted that someone indefinitely. The Architecture book answered the second half of that problem with an argument that is easy to state: a credential which expires soon is worth little to a thief, because by the time the thief has found it, copied it and worked out where to use it, it has already stopped working. Operating that idea, day after day and across an entire fleet, turns out to be much harder than stating it, and this section is about the difference.

Start with the machines. Encore’s workloads prove who they are with SPIFFE Verifiable Identity Documents (SVIDs): short-lived X.509 certificates, of the same family a web browser checks when it opens a secure site, each of which names a single workload rather than a host or a network address. The certificates are issued by SPIRE, the reference implementation of the Secure Production Identity Framework for Everyone (SPIFFE). The process works in three steps. An agent runs on every node. When a workload starts, the agent attests it, which is to say it checks the workload’s credentials with the platform it runs on, in Encore’s case by asking Kubernetes which service account the pod runs under. Having satisfied itself, the agent hands the workload a certificate naming it, for example spiffe://encore/inventory. That certificate is valid for one hour and is renewed when half its life has passed, at the 30-minute mark; both figures are SPIRE’s own defaults, which Encore has kept. When Seat Inventory calls Orders & Payments, each side presents its certificate over mutual TLS, and each knows precisely which service is on the other end, whatever pod or address the call happens to come from. This is the modern answer to rsh’s trusted hosts: trust follows a proven name, and the name expires within the hour.

A touring crew handles access to a venue in much the same way. Nobody gets a permanent pass to the building. Each night the production office issues wristbands for that night’s show, so a wristband lost in a bar after the concert opens nothing tomorrow. The analogy holds further than one might expect, because it also shows the cost: if the wristband desk fails to open one afternoon, the crew cannot get into its own venue. Where it breaks is instructive. At a real stage door, a security guard who recognizes the lighting designer can wave her through; a service checking a certificate has no such judgment, and it refuses an expired certificate as firmly as it refuses a forged one.

That is the trade every team accepts when it adopts short-lived credentials: they move risk from confidentiality to availability. We worry less about a credential being stolen and more about the issuer being able to produce new ones on time. The arithmetic for Encore is worth doing once. Because every certificate is renewed at its half-life, at any given moment the oldest certificate still in use was issued just under 30 minutes ago and has just over 30 minutes left. If SPIRE stops issuing at that moment, the first certificates expire about 30 minutes later, and services begin refusing one another’s calls; over the following half hour the remaining certificates expire in turn, until nothing in the fleet will talk to anything else. The resulting outage would be both total and self-inflicted, caused by the very mechanism meant to keep the system safe. This is no hypothetical failure class: in December 2018 an expired certificate in Ericsson software took down O2’s mobile data network in Britain for most of a day.

Encore draws the obvious consequence. A dependency on which every service depends must be more reliable than any of those services, so Encore runs the certificate issuer as a platform plane with an SLO stricter than that of any service it serves. It runs SPIRE servers in both regions, so that losing one region does not stop issuance in the other. And it alerts when any certificate passes 75% of its lifetime without having been renewed. The threshold is chosen with care: a certificate at 75% of its hour is 45 minutes old, which means its renewal at the 30-minute mark has already failed and 15 minutes have passed since; the alert therefore fires while there are still 15 minutes left before the certificate expires, long enough for a human to act before services start refusing calls.

People are the other half of identity, and here the worm’s password dictionary has a modern descendant in the phishing page. A one-time code from an authenticator app protects against a guessed or reused password, but not against a convincing fake login page: the fake page asks for the password and the code, and the attacker simply relays both to the real site within the code’s short validity window. A FIDO2 security key or passkey closes that gap because the credential is bound to the domain it was registered with. The browser will only produce a valid signature for the real site’s domain, so the fake page has nothing worth relaying. For that reason anyone at Encore with production or administrative access signs in with a FIDO2 security key or passkey, and Encore is in good company: US federal agencies have been required to do the same since 2022.

The secrets that remain

Workload identity removes most secrets, but never all of them, because some credentials are issued by parties outside Encore’s own identity system and outside its control. Encore still holds credentials from its two payment service providers (PSPs), the signing keys for the passes fans keep in their phone wallets, and the API keys that partners use to call api.encore.com. These secrets live in a secrets manager and are injected into workloads at runtime, rather than baked into images or configuration files. Each has an owner, a rotation interval and a written rotation procedure, and of these the procedure matters more than the interval.

The reason is the same one Chapter 6 taught us about backups. A rotation never performed is a restore never tested: both look reassuring on paper, and both tend to fail at the moment they are finally needed. A rotation usually fails for an ordinary reason. Some consumer read the secret once when it started and cached it in memory, so the new value sits in the secrets manager while the running process keeps presenting the old one, until the old one is revoked and the process starts failing. Only a rehearsal finds that consumer, which is why Encore cares less about how often a secret is scheduled to change than about whether the team has actually changed it and knows, step by step, what that takes.

Encore learned what one of these remaining secrets can cost when a contractor working for Rowmate, one of its resale partners, committed Rowmate’s production API key to a public repository. The key was 26 months old. Two signals arrived within minutes, from two quite different directions. The first came from the code host: its secret scanning recognized the key because Encore had registered its key prefix (enc_live_) with the scanning service, so any string with that prefix appearing in public code triggers a notice. The second came from Encore’s own systems: Bot Screening noticed the same key calling the availability endpoints at 40 times Rowmate’s usual rate, and from a cloud region Rowmate had never used. The key was revoked 38 minutes after the commit. That sounds quick, but it is slow beside the honeypot studies, in which researchers deliberately leak keys and find them tried within minutes.

Encore has since turned on push protection, which rejects a commit containing a recognizable secret before it ever reaches the repository, and it now guards all of Encore’s own repositories. Push protection could not have helped in this case, however, because the commit was made by Rowmate’s contractor in a repository Encore does not control. Detection was fast and revocation was reasonably fast, yet neither is what limited the damage. What limited it was a decision made years earlier, when the key was issued: the key’s scopes.

The key carried two permissions, listings:read and resale:write. It could not buy primary inventory, read fans’ personal data or touch payouts, so whoever held it could look at listings and post resale offers, but nothing more. The gateway’s audit log showed exactly what that amounted to: 14 minutes of scraping and two fraudulent resale listings, which were removed before anyone bought them. Had the key been issued with broad access, as convenient keys often are, the same 38 minutes would have read very differently.

Revocation was the easy part; rotation was harder, and it shows why the written procedure matters. Rowmate needed a deployment of its own software to switch to a new key, and that took time. Revoking the old key at once would have cut off a legitimate partner along with the attacker. So for an hour both keys were valid, with the old one restricted to Rowmate’s own network addresses, so that the attacker lost access while Rowmate kept it.

The deeper fix was to stop issuing partners a long-lived key at all. Partners now use OAuth client credentials bound to a client certificate (RFC 8705, which open-banking profiles also require). In practice the partner exchanges its certificate for an access token that lives 15 minutes, and the token is accepted only on a connection made with that same certificate. A token copied into a public repository is therefore useless on two counts: it is tied to a private key the thief does not have, and in a quarter of an hour it expires anyway. Like every security improvement, this one moved a cost somewhere, in this case to the partners, who must now issue, store and renew certificates of their own. For partners who cannot, Demonstrating Proof of Possession (DPoP, RFC 9449) offers a lighter route to the same property: the client holds a key pair and signs a small proof with each request, and the token is bound to that key, so a stolen token without the key is worthless.

Figure 7.2 — How long each of Encore’s credentials lives, on a logarithmic scale, set against the 38 minutes it took to revoke Rowmate’s key. A token of the kind partners use now would have expired on its own before anyone revoked it.

Least privilege drifts

Scopes limited the damage from Rowmate’s key, and scopes are one instance of least privilege: the principle that every person and workload should hold only the permissions its work requires. The principle is easy to apply on the day a role is created and hard to keep afterward, because privilege only ever drifts in one direction. During an incident someone grants a service account a broad permission to get things working, and nobody removes it afterward, because removing a permission is risky and leaving it is free. Service accounts outlive the services they were made for. Each grant is reasonable on the day it happens; it is the accumulation that is dangerous, and it ratchets upward because nothing in daily work pushes it back down. Least privilege, then, is granted once and eroded continuously.

Encore reverses the drift using evidence of actual use, as the cloud providers’ access analyzers do. Access logs record which permissions each role actually used in the last 90 days. Every quarter, the team that owns each role receives a proposed policy with the unused permissions removed, and it may accept the proposal, amend it, or defend a permission it wants to keep. The defense matters, because evidence of use has a blind spot: a permission needed only for regional failover looks unused for most of the year, and removing it would quietly break the one procedure the team most needs to work when it is finally invoked. The quarterly review puts the decision in front of the people who know which permissions are dormant and which are dead.

All of this tightening raises an honest question: what happens when an engineer, in the middle of an emergency, needs a permission the tightened policy no longer grants? The exception every organization needs, and few design in advance, is break-glass access: a pre-arranged, audited path by which a human gains elevated access to production in an emergency. The name evokes the fire-alarm box behind a pane of glass, and the analogy is apt in one important respect. Anyone may break the glass, but nobody can do it quietly, and the noise is the point. Where the analogy breaks is that a fire alarm needs no second opinion, whereas production access, in the wrong hands, is itself the emergency.

At Encore, break-glass access therefore requires a second engineer’s approval and expires after 60 minutes. Using it posts a message to the security channel and opens a review ticket, so that every use is seen at the time and examined afterward. The cloud providers’ guidance adds the condition that matters most: the path must work when normal sign-in is the thing that has failed. An emergency path that runs through the same single sign-on as everything else will be unavailable in precisely the emergency where it is needed most. So Encore’s emergency accounts sit outside single sign-on, and the team exercises the path at every quarterly game day, for the same reason it rehearses its rotations: a path nobody has walked recently is not a path anyone can count on.

Recap. Short-lived certificates remove most secrets, but they turn the issuer into a dependency that must be more reliable than anything that relies on it. Humans need sign-in that a fake login page cannot relay, and the secrets that remain need owners, rotations that have actually been rehearsed, and scopes narrow enough that a thief gains little. Least privilege drifts upward unless evidence of actual use turns it back, and break-glass access must be loud, brief and independent of the systems it exists to rescue.

Exercise 7.2. Pick one long-lived secret your system depends on and rotate it in a staging environment, timing each step. Which consumer broke, and why? Then ask what would have happened if you had done the same rotation in production at 3 a.m., under pressure, because the secret had just leaked.

7.3 Vulnerabilities, Patching, and the Fleet

Severity is not risk

Of the Morris worm’s four doors, the overflow in fingerd is the one that most resembles our daily work: a known class of bug, sitting in a binary that thousands of machines ran, waiting for someone to update it. Its modern descendant is a flaw in a library the whole fleet shares, and the story that best shows how such a flaw becomes a catastrophe is a recent one. In March 2017 the Apache Struts project published a fix for a remote code execution flaw. Roughly two months later, attackers entered Equifax through that same flaw and exposed personal data on about 147 million people. It is tempting to read this as a story about a company that did not patch, but the more useful lesson is subtler and more uncomfortable. The organization had a patching process. The process simply did not reach the system that mattered, because nobody could see clearly which systems ran what. A fix that exists, and a process that applies fixes, are worth nothing to a server the process cannot find.

Knowing where everything runs is the first problem, and Section 7.4 returns to it. The second problem is deciding which flaws deserve our urgency, because there are far more of them than any team can treat as emergencies. Public advisories, each identified by a Common Vulnerabilities and Exposures (CVE) number, arrive by the tens of thousands a year. If every one of them were urgent, none would be, and so the whole craft of vulnerability management rests on separating the few that matter now from the many that can wait. Several public instruments help with that separation, and it is worth being precise about what each one measures, because each answers a different question.

The Common Vulnerability Scoring System (CVSS) scores each flaw from 0 to 10 for how bad exploitation would be in general. It is a measure of severity: it describes the flaw considered on its own, as it might behave in a typical deployment. What it cannot say is whether anyone is actually exploiting the flaw, or whether our code ever calls the vulnerable function. A flaw scored near the top of the scale in a parsing routine we never invoke is, for us, less dangerous than a lower-scored one that sits on our checkout path and is being attacked this week.

The Exploit Prediction Scoring System (EPSS), maintained by the Forum of Incident Response and Security Teams (FIRST), answers a second question: how likely is it that this flaw will be exploited somewhere in the next 30 days? It is a probability, and like any forecast it should be read with care. EPSS knows nothing of our assets, so it cannot tell us whether the flaw is reachable in our systems. And a low score lowers the odds without ruling exploitation out, in the way that a forecast of a small chance of rain still leaves someone wet.

The Known Exploited Vulnerabilities (KEV) catalog, published by the US Cybersecurity and Infrastructure Security Agency (CISA) since November 2021, answers a third question, and the bluntest one: is this flaw already being exploited? A flaw on the KEV list is no longer a forecast. Someone, somewhere, has already walked through that door.

None of these three instruments can know two facts that only Encore can supply. The first is exposure: does the vulnerable component face the internet, or sit on the purchase path where an attacker most wants to be? The second is reachability: does Encore’s code actually reach the vulnerable function? Reachability tools, which trace call paths from our code into a library, inform that second judgment, but they do not settle it, because a path that looks dead to a static analysis can come alive through configuration, reflection or a plugin loaded at runtime.

An emergency physician would recognize this problem at once. Triage, as hospitals practice it, is the sorting of patients by how soon they will be harmed without attention and how much attention can change the outcome, rather than by how serious their condition is in general. A broken wrist waits behind a quiet patient with chest pain, although nobody would call a broken wrist trivial. Vulnerability triage works the same way: CVSS describes the disease, while exploitation evidence and exposure describe the patient in front of us. The analogy breaks at one point that matters. A patient’s condition is largely visible to the nurse who examines them, whereas a flaw’s reachability in our own code is often uncertain, so our triage must sometimes act on a judgment that may be wrong and record the reasoning, so that someone can revisit it when the evidence changes.

Encore combines these signals into a single policy, ordered first by evidence of exploitation and then by exposure:

Figure 7.3 — Encore’s triage policy, ordered by evidence of exploitation and by exposure so that urgency is spent where an attacker is most likely to arrive. The shape follows CISA’s BOD 26-04; the deadlines are deliberately tighter.

Reading the table row by row shows the reasoning behind it. The top row is the true emergency: a flaw someone is already exploiting, in a component an attacker can reach from outside. Encore’s first move there is a compensating control at the edge, such as a gateway rule that blocks the known exploit pattern, because such a rule can be in place within hours while the real fix is built and tested. It then asks whether the attacker came through before the door was closed. The second row is the same kind of flaw behind Encore’s own walls, where an attacker would first need some other way in; the fix is still expedited, but it travels through the normal pipeline rather than as an emergency change. The third row covers flaws that are severe and likely to be exploited but not yet known to be, and these ride the next scheduled rebuild of the base images. The last row is the honest answer to most advisories: the flaw is real, but our code does not reach it, so we write down why we believe that and move on. That written reasoning is what lets a later reader, or a later version of ourselves, notice when the belief stops being true.

Encore need not defend this shape alone, which helps on the day a product manager asks why a team is pausing feature work for a flaw with a middling score. In June 2026 CISA’s Binding Operational Directive 26-04 replaced the KEV directive of 2021. It now sets federal remediation deadlines by four criteria: exposure, KEV status, whether exploitation can be automated, and how much control an attacker gains. Its tightest tier allows three days and adds forensic triage, a check of whether the door was used before it was closed, and Encore borrowed that check for its own top row. Encore sets a stricter 48 hours. A deadline, however, is a promise about what the fleet can do, and Encore can make this one only because its fleet can keep it. How the fleet keeps it is the subject of the rest of this section.

Patching as a flow

The decisive move is to stop thinking of patching as an event. When patching is an event, it is something an organization does when an advisory frightens it enough: a project with a deadline, a war room and a list of servers to touch. Every such event begins by rediscovering how each service is built and deployed, and that rediscovery is where the hours go. When patching is a flow, the same work happens continuously and without ceremony, so that an emergency is simply the ordinary process run sooner.

Encore therefore never patches a running container. It rebuilds the image from an updated base and redeploys it through the ordinary pipeline, canaries included, so that a security fix travels the same road, past the same safety checks, as any other change. Base images are rebuilt weekly whether or not an advisory demands it. The service scorecard then reports the median age of each service’s running images, and the emphasis on “running” is the whole point of the measure. A fresh build that was never deployed has patched nothing, because what an attacker meets is whatever is actually serving traffic.

A touring crew understands why the weekly rhythm matters. A crew that loads out after every show and loads in at the next venue knows the rig goes together, because it put the rig together last night; a missing bolt or a frayed cable turns up on an ordinary afternoon rather than on the night of the big show. A crew that assembles its stage once a year finds all those problems at once, and under pressure. The comparison breaks in an instructive place. The touring rig is the same every night, whereas each rebuild deliberately changes its ingredients, and that is exactly why rebuilding must be rehearsed often: every change is a fresh chance for something to stop fitting, and small, frequent changes reveal the misfit while it is still small. The consequence is stark. A fleet that rebuilds every week patches in a day when it must; a fleet that rebuilds once a year discovers, on the day it must, which services no longer build.

A continuous flow carries its own danger, though, and one of Encore’s rules did not survive 2025. Dependency updates used to merge automatically whenever tests passed on a patch release, on the reasonable theory that patch releases are small and usually fix things. Then attackers showed how that theory could be turned against the people who held it. On 8 September 2025 an attacker phished the maintainer of chalk, debug and 16 other npm packages, with more than two billion weekly downloads between them, and published versions that rewrote cryptocurrency payment addresses in browsers. Days later the Shai-Hulud worm began stealing npm tokens and using them to publish infected versions of its victims’ own packages, so that each compromise seeded the next; by the end of its second wave in November, more than a thousand packages had been compromised.

The two attacks shared a shape worth naming. Each traveled as an ordinary new release, indistinguishable at first glance from the routine updates that automation exists to absorb. And most of the malicious versions were pulled within hours, once scanners and maintainers across the ecosystem noticed them. Put those two facts together and the danger becomes plain: automation that merges a release minutes after it appears is precisely the conduit such attacks need, because it acts inside the brief window before anyone has looked.

Encore’s rule is now a cooldown: a version update must have been public for three days (for patch releases) or seven (for everything else) before automation proposes and merges it, so that the ecosystem’s scanners look first. The cooldown trades a little freshness for the collective attention of everyone else who reads new releases. Security updates skip the wait, and the reason deserves spelling out. A security release fixes a flaw that is already public, so delaying it protects us from nothing unknown; it only lengthens the time during which attackers know of a weakness we have not closed. It is the one kind of release whose delay helps only the attacker. GitHub made a short cooldown Dependabot’s default in July 2026, and Encore’s configuration states its own intervals explicitly:

version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "daily"
    cooldown:              # version updates only;
      default-days: 7      # security updates are never delayed
      semver-patch-days: 3

A cooldown and a weekly rebuild describe the flow on an ordinary week. Whether the flow survives a real emergency is a separate question, and history offers a test. Log4Shell, disclosed on 9 December 2021 with a CVSS score of 10, is the best-documented trial of patching as a flow. The hard part was not writing the fix. It was finding the library, which was often a dependency of a dependency, or bundled inside a vendor’s archive where no one had thought to look. And the first fixed release proved incomplete, so organizations that had struggled to patch once had to do it again. In 2022 the US Cyber Safety Review Board called it an endemic vulnerability, one that would linger in the world’s systems for years because so many of those systems could not easily be found or rebuilt. The lesson echoes Equifax: the speed of a patch is set less by the fix than by our knowledge of where the vulnerable component lives.

Encore’s own test came this spring: a critical remote code execution flaw in a widely used XML parsing library, added to the KEV list within a day of disclosure. That placed it squarely in the top row of Figure 7.3, with a 48-hour deadline. The first question was the one Log4Shell had made so hard, and Encore’s software bills of materials (Section 7.4) answered it in 20 minutes: 37 affected images across 23 services. A binary scan of third-party images found one more copy, inside a vendor’s monitoring agent whose supplier published no bill of materials, which is exactly the kind of hiding place Log4Shell had taught everyone to fear.

The response then followed the top row step by step. Within 6 hours the gateway’s web application firewall (WAF) blocked the known exploit pattern, buying time for the real fix, and the logs were searched for earlier use of that pattern, without result. Meanwhile the weekly-rebuild habit did its work: by hour 30, 35 images were rebuilt and redeployed, and by hour 48, all of Encore’s own. The vendor’s agent was a different matter. Encore could not rebuild software it did not build, so it confined the agent with an egress rule, limiting where it could send traffic, and the agent waited four more days for its supplier to ship a fix. The lesson Encore drew was contractual rather than technical. It now writes patch timelines into vendor contracts, because a supplier’s speed has become part of Encore’s own.

Figure 7.4 — The spring parser flaw: affected images redeployed against hours since disclosure, from the counts Encore recorded. The curve is steep where Encore owns the build and flat where it waits on a supplier.

I have come to believe that a curve of this kind is the most honest picture of a security posture we can draw. Its steep part measures what an organization has practiced; its flat part measures what it has delegated to others and must take on trust. Neither part can be changed once the flaw arrives, because both were set by the habits and contracts of the months before.

Recap. CVSS measures how bad a flaw could be in general; exploitation evidence, exposure and reachability decide how fast we must move, which is also how CISA’s directive now sorts its deadlines. Patching works best as a continuous flow of rebuilds, measured by the age of the images actually running, so that an emergency is the routine done sooner. Because attackers now hide inside ordinary releases, routine updates wait out a cooldown, while security fixes, which help us only when they arrive quickly, are never delayed.

Exercise 7.3. Take the last 20 vulnerability findings for a service you own and reclassify them with Figure 7.3. How many were treated as urgent, and how many deserved to be? Then find out how old a new release must be before your automation merges it, and decide whether that age gives the ecosystem’s scanners time to look first.

7.4 The Software Supply Chain — Provenance, SBOMs, and Signing

When the build is the target

The Architecture book told two stories that frame this section, Thompson’s warning and the xz backdoor of 2024, and there is no need to tell them again. Between them in time sits a third case, and it is the one that changed operational practice most. In December 2020 it became known that attackers had compromised the build system of SolarWinds and had inserted a backdoor into updates of its Orion software. Those updates were signed with the company’s own key and shipped to as many as 18,000 customers. Consider what each of the usual safeguards would have reported. A review of the source code would have found it clean, because the backdoor was added during the build, not in the repository. A check of the signature would have found it genuine, because the company’s real build system had applied it. And yet the product that customers installed carried the backdoor. The attack lived in the one place nobody was looking: the machinery that turns source into something shippable.

That lesson gives us the subject of this section. The software supply chain is everything that touches an artifact between a developer’s keyboard and production: the libraries we pull in, the build system that compiles and packages them, the registry that stores the result and the deployment path that runs it. SolarWinds showed that any of these links can be the one an attacker chooses, so the frameworks that grew up afterward share a single aim. Each link should produce evidence about what it did, and the next link should check that evidence before it trusts what it has been handed.

Shipping offers a faithful picture of this, and it is worth taking a little further than usual. A container leaving a port is closed with a numbered seal, and the seal number is written on the manifest. At every handover, someone compares the seal on the door with the number on the paper. If they match and the seal is intact, nobody opened the container in transit; if they do not, the container is held. The checking is cheap, it happens at every link and it needs no trust in the people who carried the box. Our build evidence works the same way. The analogy also tells us exactly where the method stops. An intact seal says nothing about whether the shipper loaded the container honestly in the first place. Signatures and provenance tell us that an artifact came, unaltered, from the build we expected; they cannot tell us that the build’s inputs were sound. SolarWinds was, in these terms, a dishonest loading behind a genuine seal, and that is why the evidence must begin as early in the chain as we can make it begin.

The pipeline that does the building is itself a chain of dependencies, as March 2025 showed. Most continuous integration today runs on platforms such as GitHub Actions, where a workflow is assembled from small reusable steps, called actions, that other people publish. Attackers compromised one of them, tj-actions/changed-files, which some 23,000 repositories used, and moved its version tags to a commit of their own. That commit printed the pipeline’s secrets into the build logs, and many of those logs were public. A tag is a movable label, so every workflow that referred to the action by tag (by a version name rather than by a fixed commit) simply ran the attacker’s code at its next build, without any change on its own side.

Encore’s golden path applies GitHub’s own hardening guidance, and each of its three rules answers one part of that attack. First, third-party actions are pinned to a full commit hash, because a hash names one exact piece of code and, unlike a tag, cannot be moved to point at another. Second, the token each workflow receives carries only the permissions its job needs, so a hijacked step can do only what that job could do. Third, the build reaches the cloud through short-lived OpenID Connect (OIDC) federation: rather than storing a cloud password in the pipeline, the build platform vouches for the running job’s identity, and the cloud exchanges that identity for credentials that expire soon. A log that leaks such credentials leaks nothing that lives long, which is the lesson of Section 7.2 applied to the build. The workflow header that carries the first and third rules is short:

permissions:
  contents: read
  id-token: write        # OIDC: cloud federation and keyless signing
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<full-commit-sha>   # pinned, never a tag

Protecting the pipeline is half the work; the other half is making the build leave evidence that later links can check. Supply-chain Levels for Software Artifacts (SLSA) is the framework that grades this evidence for the build link. Its build track, unchanged from version 1.0 to the current 1.2, climbs in three steps, and each step closes a gap that the one below it leaves open. Level 1 asks that the build produce provenance: a record of what was built, from which source, by which builder and with which inputs. Provenance at this level is useful for finding things, but anyone could have written it. Level 2 therefore asks that a hosted build platform sign the provenance, so that it can be traced to a real builder and is no longer the word of whoever ran the build. Even signed provenance is only as trustworthy as the platform that signs it, so Level 3 asks that the platform be hardened: builds must be isolated so that they cannot tamper with one another, and the signing must happen out of reach of the build’s own steps, so that a build cannot forge its own provenance. We should be honest about the standing of all this. SLSA is a consensus framework, written by practitioners and without published data on the outcomes it produces. Its practical appeal is that the lower levels are cheap to reach on hosted platforms, which already do much of the work.

Provenance is worth little unless it is signed, and for years signing was held back by the burden of long-lived keys. A key that signs releases must be generated, stored, protected, rotated and, one day, revoked, and most projects quietly decided that the burden was not worth bearing. Sigstore (Newman, Meyers and Torres-Arias, 2022), now used by npm, PyPI and Kubernetes, removed most of that burden by applying the idea of short-lived credentials to signing itself. When the build is ready to sign, it presents its OIDC identity, the same one it uses to reach the cloud, and receives a certificate valid for ten minutes that binds that identity to a freshly generated key. It signs with that ephemeral key, records the signature in a public transparency log, an append-only record that anyone can audit and nobody can quietly edit, and then the key is discarded. There is no long-lived key to steal, because none exists.

This shifts the security question from “who holds the key?” to “whose identity was in the certificate?”, and verification must ask that question precisely. The question Encore wants answered is whether this image was signed by our release workflow running from our main branch. A verifier therefore has to name both the identity issuer and the workflow. If it checks only that the signature is valid, it will accept a signature from any workflow anywhere, since anyone can obtain a ten-minute certificate for their own identity. A signature, in other words, proves who signed; only the verifier’s policy decides whether that signer was the one we trust.

The bill of materials as an index

Provenance tells us where an artifact came from. The next question, and the one Section 7.3 kept asking, is what is inside it. A software bill of materials (SBOM) is a machine-readable inventory of the components inside an artifact, together with their versions, written in a standard format such as SPDX or CycloneDX. The term comes from manufacturing, where a bill of materials lists every part that goes into a product, and the borrowing is apt: when a supplier recalls a faulty part, the manufacturer with a good bill of materials knows within the hour which products contain it.

That comparison points to where an SBOM’s value lies. It lies entirely in being queried. An SBOM filed away beside each image answers nothing on the day it is needed. Encore indexes every SBOM and joins it to the record of which image runs where, so that “where do we run this library?” becomes a query rather than a week of messages between teams. This is how the spring parser flaw of Section 7.3 was traced to 37 images across 23 services in 20 minutes.

The index, however, is only as complete as the documents in it, and an SBOM is only as complete as the tool that wrote it. Academic studies in 2023 and 2024 found that SBOM generators disagree widely with one another and miss vendored or bundled components, meaning code copied into a project or packed inside another archive rather than declared as a dependency. These are the very places where Log4j hid. That is why Encore does not rely on SBOMs alone, and why the binary scan of third-party images in Section 7.3 found the copy inside the vendor’s monitoring agent that no bill of materials had listed.

An inventory also raises the opposite problem: it will report components that are present but harmless. A vulnerable library may be included without the vulnerable function ever being called, and a team that cannot record this will rediscover it, and argue it again, with every rescan. A Vulnerability Exploitability eXchange (VEX) statement is the record of that reasoning: a machine-readable note that a given product is not affected by a given vulnerability, and why. VEX is young and its formats still compete, so Encore chose the one its tools already understand and writes OpenVEX, which its scanner reads. The last row of Figure 7.3, “record as not affected, with the reasoning”, is written in this form.

With provenance, signatures and SBOMs defined, we can see how Encore assembles them into one path. Each stage produces an artifact, a later stage checks it, and each check has a decided behavior when it fails:

Figure 7.5 — Supply-chain controls on Encore’s golden path. Every stage leaves evidence that a later stage checks automatically.

Read down the table and the container seals reappear. The lockfile records the exact hash of every dependency, so the build can check that it received what was chosen. The build signs its image and provenance, and the cluster checks that signature before anything runs. The SBOM travels with the image, so the nightly rescan and the runtime inventory can both ask what is inside. No stage takes the previous one’s word for anything.

Encore applies this path in proportion to what each service protects. Every service reaches SLSA Build Level 2. The purchase path and the identity issuer, the two places where a forged build would do most harm, are built at Level 3 on isolated builders. At the cluster’s door, the admission check rejects any image that was not signed by Encore’s release workflow. Vendor images, which Encore cannot build itself, enter through an exception path that is priced and logged, so that every exception is visible and has a cost someone agreed to pay.

Here a familiar trade-off between security and availability appears. Encore’s admission check fails closed: if the policy engine that verifies signatures cannot be reached, deployments are refused rather than allowed through unchecked. That is the right posture for security, but it means that an outage of the policy engine stops every deployment, which makes the engine as critical as the cluster’s own control plane. Encore therefore runs it replicated across zones. One further exception keeps the arrangement from trapping itself. System namespaces, where the cluster’s own components run, are exempt from the check, so that an unavailable engine cannot block the very pods that would restore it. Section 7.5 will show what happened when one of these settings had been chosen differently, long ago, and forgotten.

Dependencies, human and machine

Everything so far has been about proving that we received what we chose. It cannot tell us whether we chose well, and that is a matter of judgment about people. Each dependency means trusting its maintainers with every release they will ever publish, including the ones we accept automatically. I spent a decade maintaining a widely used open-source library, and the view from the other end is sobering: much of the world’s software rests on a few volunteers’ evenings, and on whether those volunteers stay interested, stay well and keep their accounts secure. Encore therefore asks of each new dependency whether it is maintained, and by how many people. It reads part of the answer from OpenSSF Scorecard, which checks a project’s practices automatically, and supplies the rest by looking.

Coding assistants have added a new way to choose badly. Spracklen and colleagues, at USENIX Security in 2025, found that commercial models suggested packages that do not exist at least 5% of the time, and open models about a fifth of the time. A name that a model invents can be registered by anyone. An attacker who registers such a name and fills it with malicious code, a practice now called slopsquatting, has only to wait for a developer to accept the assistant’s suggestion and install it. The risk is real: when a researcher published a harmless package under one hallucinated name, it drew tens of thousands of downloads. The defense, fortunately, is cheap. Encore’s builds resolve packages only through a curated internal proxy, as the first row of Figure 7.5 shows, and adding a package to that proxy is a reviewed change. A hallucinated name therefore fails at the build, before it can run anywhere that matters.

The assistants are dependencies themselves, and they run with more privilege than most libraries. In July 2025 an attacker used an over-scoped token in the build configuration of Amazon’s Q Developer extension for Visual Studio Code to commit a prompt to the extension’s code. The prompt told the coding agent to wipe local files and cloud resources, and the extension’s release pipeline shipped it as version 1.84.0. Only a syntax error kept it from running. The case joins the two halves of this section: it was a supply-chain attack on the build of a tool, and the tool was one that acts on a developer’s behalf. Such an extension runs with that developer’s credentials, so whatever it is told to do, it can do with the developer’s reach. Encore treats these tools as it treats any other dependency on the critical path: it pins their versions, admits them only from an allowlist and gives agents scoped credentials of their own rather than borrowing a developer’s, as Chapter 8 describes.

Recap. SolarWinds and tj-actions showed that the build itself can be the target, so each link in the chain must produce evidence that the next can check. SLSA grades how trustworthy the build’s evidence is, Sigstore makes signing cheap enough to be universal, and SBOMs pay only when they are queried and only as far as they are complete. None of this tells us whether we chose our dependencies well. Whether they are chosen by people or suggested by their assistants, they still need judgment and a curated gate.

Exercise 7.4. Pick one image running in production today. Can you show, with evidence rather than memory, which commit and which builder produced it, and which version of its largest dependency it contains? List each step you could not answer.

7.5 Detection, Response, and Security Chaos Engineering

Detection that knows what normal looks like

Everything in the sections before this one lowers the chance that an intruder gets in: credentials that expire, patches that land in hours, images that carry proof of where they came from. None of it lowers that chance to zero, and an honest program admits as much. Once we accept that some attacker will eventually succeed, the question that matters most changes. It is no longer only “how do we keep them out?” but also “how quickly do we notice that they are in?”, because every hour between entry and discovery is an hour in which the intruder works undisturbed.

Recall how the Rowmate key was caught. Nobody saw the commit happen, and nothing about the requests themselves was malformed; the key was valid and its calls were the kind Rowmate was allowed to make. What gave it away was behavior. The key was being used at 40 times Rowmate’s usual rate, from a cloud region Rowmate had never used. In other words, the detection worked because something knew what normal looked like for that particular credential, and the stolen key did not look normal. That is the general shape of good detection, and it explains why Encore writes its detections mostly against identity. Every request at Encore already carries a workload identity or a partner credential, so each identity has a history of what it usually does, and a departure from that history is a signal. Three of Encore’s detections show the pattern. One fires when a workload calls a service it has never called before, which is what an intruder exploring from a compromised pod tends to do. One fires when a break-glass session opens without a matching ticket, because the emergency path described in Section 7.2 is legitimate only when someone has declared an emergency. And one fires when an image is running that the admission controller never saw, which means something reached the cluster by a route that skipped the supply-chain checks of Section 7.4.

A stage manager calling a show offers a useful comparison. After a few weeks of performances she knows how the evening normally sounds: when the fly cues land, how long the scene change takes, which door the crew uses. She notices a stranger in the wings not because she has a list of every possible stranger, but because she knows who belongs there. Identity-based detection works the same way, by learning who belongs where and noticing the exception. The analogy breaks in one instructive place. The stage manager learned what normal looks like by being present, while a detection learns it only from the logs it can read, and if those logs reach back only a few days, the detection has no memory of normal to compare against.

That is why Encore treats its detections as engineering artifacts with the same discipline that Chapter 1 applied to burn-rate alerts. Each detection is kept as code, so it is reviewed, versioned and changed deliberately. Each is tested with synthetic events, so that we know it fires when the condition occurs and not merely that it was written. Each has an owner and a runbook, so that whoever is paged at 3 a.m. knows what the alert means and what to do next. And the logs the detections read are kept for weeks rather than days, because a baseline, the record of what an identity normally does against which unusual behavior is judged, needs history before it can tell the ordinary from the strange. A gateway that keeps three days of logs can neither establish a baseline nor answer, a week later, what a stolen credential actually did.

Detection, however, only starts the clock. Once something has been noticed, someone has to respond.

Security incidents follow the same command structure that the Operations book’s chapter on incident response describes for any incident, with an incident commander, clear roles and a single place where the response is coordinated. They differ from an ordinary outage in three ways, and each difference comes from the fact that this time there is someone on the other side.

The first difference is that the adversary may still be watching. In an outage the failing component does not read our chat; in a security incident the intruder may have access to the very mailboxes, chat channels or tickets we would normally use to coordinate. So the incident channel is restricted to the people who need to be in it, and some actions are deliberately held back until containment is ready. If we revoke one stolen credential while the intruder still holds three others, we have only told them we are coming, and given them time to use the rest before we can act on all of it at once.

The second difference concerns evidence. In an outage the natural instinct is to restart the broken thing, and usually that instinct is right. In a security incident, restarting a compromised host or wiping a container destroys exactly the evidence we need to learn how the intruder got in, what they touched and whether they are still present. NIST’s revised guidance of 2025 insists on preserving evidence before remediation for this reason. The engineer’s reflex to restore service has to wait, briefly, while memory, disks and logs are captured, and that wait feels wrong to anyone trained on outages, which is precisely why it needs to be written into the runbook rather than left to judgment in the moment.

The third difference is a legal clock. Under the General Data Protection Regulation (GDPR), a personal data breach must normally be reported to the supervisory authority within 72 hours of the organization becoming aware of it. Seventy-two hours sounds generous until we remember that it includes the time spent working out whether personal data was involved at all, what data, and how many people are affected. That is why a privacy lead belongs in the room from the first hour, not summoned on the second day once the engineers have finished their investigation.

When it comes to containment, the work of stopping an intruder from doing further harm while the investigation continues, the nature of the intrusion decides what has to be done. Where the intruder holds credentials, as in the Rowmate case, containment is mostly revocation: invalidate what was stolen and issue replacements. Where a host itself has been compromised, revocation is not enough, because we can no longer trust anything running on that machine; it must be isolated from the network and rebuilt from a known-good image. The Morris worm showed the cost of having only the second, blunter tool: in 1988 many sites contained the worm by cutting themselves off from the network entirely, which stopped the infection and also stopped the fixes from arriving. Fine-grained revocation is the modern alternative to pulling the cable.

This is also where the earlier sections of the chapter pay off. An organization that can rotate any credential within an hour can contain an incident within the hour. An organization that needs a week to rotate leaves the intruder that week to use what was stolen. The speed of containment in a crisis is decided long before the crisis, on the ordinary days when rotation was either rehearsed or put off.

Security chaos engineering

All of the controls we have described, from detections to admission policies to break-glass alerts, share an uncomfortable property. Most of the time they do nothing visible. A detection that has never fired might be working perfectly in a quiet environment, or it might be broken, and from the outside the two look exactly alike. The same is true of a policy that has never rejected anything. So how do we know our controls work?

Kelly Shortridge and Aaron Rinehart, in Security Chaos Engineering (2023), answer by borrowing the experimental method of Chapter 5’s game days and pointing it at security controls. Their central idea is that a control we have never seen trigger is not a fact about our system but a hypothesis about it. Like any hypothesis, it can be tested: we introduce the condition the control is meant to catch, deliberately and within agreed limits, and we watch whether the control does what we believe it does. The fire alarm is the familiar version of this. A building does not wait for a real fire to find out whether its alarms sound; it presses the test button and runs a fire drill. The analogy is imperfect in a way that matters, because a fire does not change its behavior after watching our drills, whereas an adversary adapts, so security experiments have to be repeated as attackers’ methods change rather than performed once and filed.

It is worth being precise about how much evidence stands behind this practice. The name “security chaos engineering” is recent, and its published evidence consists mostly of practitioner case studies, Capital One’s among them, rather than controlled research. But the practices it gathers under that name are older and well established. Canary tokens are fake credentials, files or records planted where only an intruder would find or use them, so that any use at all is a reliable alarm. Control validation is the routine of triggering a control on purpose to confirm it still responds. And purple-team exercises bring together the red team, who play the attacker, and the blue team, who defend, working side by side rather than in secret, so that every attack attempted is also a lesson in whether and how it was seen. What the new name adds is a frame: treating these older practices as experiments with stated hypotheses and limits, run on a schedule like any other game day.

Encore runs its security experiments as part of its quarterly game day, and the most recent one shows what such experiments find. It ran three. In the first, a canary key was pushed to a test repository, standing in for a leaked credential, and it was detected in four minutes. In the second, an unsigned image was deployed to a staging cell, and admission rejected it, as the supply-chain policy of Section 7.4 requires. Both controls confirmed their hypotheses. Confirmations like these are not wasted effort; they turn “we believe this works” into “we watched it work this quarter.”

The third experiment found a real gap. Its purpose was to see what happens to admission when the policy engine itself is unavailable. The admission webhook is the hook through which Kubernetes asks the policy engine whether a new pod may run, and every such hook has to decide what to do when the engine does not answer. It can fail closed, refusing anything it cannot check (the posture Section 7.4 described), which protects security at the cost of availability, or it can fail open, allowing anything it cannot check, which protects availability at the cost of security. With the engine taken down for the experiment, Encore’s webhook failed open, and an unsigned image deployed without complaint. The setting had been chosen long before, during an upgrade, to keep deployments flowing while the engine was briefly unavailable, and it had simply never been changed back.

What makes this finding instructive is how the decision was made, or rather how it was not. Nobody at Encore had ever decided that security should yield to availability on the purchase path. A setting had decided it, years earlier, for a reason that made sense for one afternoon, and some widely used policy engines still ship with that default today. No review would have caught it, because nobody reviews a value they do not know they chose. Only provoking the failure revealed it.

The fix was to make the choice deliberately. The webhook now fails closed for the purchase path, where an unverified image is the greater risk. Because failing closed makes the policy engine as critical as the control plane, the engine is replicated across zones, as Section 7.4 described, and there is a break-glass procedure for the day it fails anyway, so that a broken engine cannot freeze the purchase path with no way out. This is the trade-off the Kata’s rubric asks about: every control fails one way or the other when its dependencies fail, and the only real question is whether someone chose which way. I have come to believe that this is the most valuable thing security experiments find. A control that has never been seen to fail has only been seen to be configured.

Recap. Detection works best when it knows what normal looks like, which is why Encore writes its detections against identity and behavior, keeps them as code, tests and owns them like any alert, and keeps enough log history to form a baseline. Security incidents use the ordinary command structure with three additions: a restricted channel because the adversary may be watching, preserved evidence before remediation, and a privacy lead watching the GDPR’s 72-hour clock. Containment is mostly revocation, and its speed is decided by how well rotation was rehearsed beforehand. Security chaos engineering, a young name for older practices such as canary tokens, control validation and purple-team exercises, tests controls by provoking them, and its most useful findings are defaults that nobody remembers setting.

Exercise 7.5. Choose one security control you believe in and design an experiment that could prove you wrong about it. Write down four things: the hypothesis (what you expect the control to do), the condition you will inject to trigger it, the limit on blast radius that keeps the experiment safe, and the signal that would show the control did not work. Run the experiment outside production and record what surprised you, including any default you discovered you had never chosen.

Kata: The Spare Key

Your brief: “Hearthstay.” Hearthstay is a vacation-rental marketplace. It lists 85,000 properties from 30,000 hosts across southern Europe and takes 2.5 million guest bookings a year. Its payment flow charges guests when they book and pays hosts after the guest has checked in, so for weeks at a time Hearthstay holds money that belongs to someone else. Around the core marketplace sits a partner API, used by 40 channel managers and resellers to keep hosts’ calendars and prices in sync across the sites where a property is advertised. Each partner authenticates with a static API key, and the oldest of those keys dates from 2018. At check-in the platform also sends guests the codes that open their doors, which means a Hearthstay credential can, at the end of a long chain, unlock a real apartment.

Twenty-six engineers run about 70 services on two Kubernetes clusters. A recent audit found base images up to 500 days old, no software bills of materials (SBOMs), a single service account shared by 11 services, and a payout-admin console that anyone on the office network could reach without multi-factor authentication.

Last month a channel manager’s key turned up in a public code repository. That same week, three hosts found that the bank details for their payouts had been changed. Nobody can say whether the leaked key was the cause, because the gateway keeps its logs for only three days, so the record that could have answered the question no longer exists.

Hearthstay’s story begins where Encore’s Rowmate incident began, with a partner’s key committed to a public repository, and the comparison is the most useful way into the kata. Encore’s key was found within minutes, revoked in 38, and its scopes kept it away from payouts; its audit log then told the team exactly what the key had done. Hearthstay has none of those protections in place. Its keys are older, nothing we know of limits what they can reach, and the one record that could settle the question of the bank details has expired. The kata asks you to build, for a smaller and less prepared company, the routines this chapter built for Encore, and to decide which of them to build first. Twenty-six engineers cannot do everything at once, so the order of the work is part of the answer.

Deliverables:

1. Credential inventory and plan — list every class of credential, human and machine, with its age, its scope and its blast radius, meaning what an attacker holding it could reach. Then sequence the migration to strong sign-in for people, workload identity for services and short-lived tokens for partners, as Section 7.2 did for Encore.
2. Triage policy — Hearthstay’s own version of Figure 7.3. Include the host mobile apps, which a fix reaches only when each host chooses to update, so the deadline for closing a door there cannot be met by Hearthstay’s rebuild alone.
3. Supply-chain path — which stages of Figure 7.5 Hearthstay adopts first, the SLSA level it targets six months from now, and how its SBOMs will be queried when the next widely used library turns out to be vulnerable.
4. Detection and response pack — five detections written against identity, the log retention each one needs to have a baseline worth comparing against, and a runbook for the event that has already happened once: “a channel manager’s key is leaked.”
5. Three security chaos experiments — each with a hypothesis and the limits that keep it safe, and one of them aimed at a change to a host’s payout details, the very event Hearthstay could not explain last month.
6. Break-glass design — who may use it, how the access expires, and how it still works on the day the identity provider is the thing that has failed.

Rubric:

Three of these questions deserve a moment’s thought before you start. Rotation realism is where good plans meet partners’ release calendars: Encore had to run Rowmate’s old and new keys side by side for an hour because Rowmate needed a deploy to switch, and a channel manager that deploys once a quarter turns that hour into months unless your plan gives it a gentler path. Failure posture asks you to do deliberately what Encore discovered it had done by accident, when its admission webhook was found to fail open under a setting nobody remembered choosing; for each control, write down which way it fails and why. And proportion is a reminder that the audit’s findings are not equally urgent. The payout console without multi-factor authentication and the partner keys that never expire are the doors an attacker would try first, and a plan that starts with them will protect Hearthstay sooner than a plan that starts with whatever is easiest to fix.

Where you now stand

You can now treat security as routine work. You can run credentials that expire on their own and rotations that have been rehearsed, so that a leaked key is an inconvenience with a known procedure. You can triage vulnerabilities by evidence of exploitation and by exposure, and patch by rebuilding rather than by tinkering with what is running. You can make every link of the supply chain leave evidence that the next link checks, and you can test your controls by provoking them, because a control you have never watched fail is one you know only as configured. Every control in this chapter, though, guards a system whose parts we wrote, or at least can read, and whose behavior repeats when we ask the same question twice. The newest components Encore runs offer neither comfort. Chapter 8 turns to the models and agents Encore now runs, which are fluent, usually right and never quite the same twice, and to what it takes to operate them with the same care. The other skill this chapter leaned on, acting quickly and in concert when something does go wrong, is the subject of the Operations book.

References

• Heather Adkins, Betsy Beyer, Paul Blankinship, Piotr Lewandowski, Ana Oprea, Adam Stubblefield — Building Secure and Reliable Systems. O’Reilly/Google, 2020 — free online.
• CISA — BOD 26-04: Prioritizing Security Updates Based on Risk. CISA, 2026 — free online.
• CISA — Known Exploited Vulnerabilities Catalog. CISA, 2021 — free online.
• FIRST — Exploit Prediction Scoring System (EPSS). FIRST — free online.
• GitHub — Dependabot version updates introduce default package cooldown. GitHub Changelog, 2026 — free online.
• Kennedy Toomey — The case for dependency cooldowns in a post-axios world. Datadog Security Labs, 2026 — free online.
• CISA — Supply Chain Compromise of Third-Party GitHub Action, tj-actions/changed-files (CVE-2025-30066). CISA, 2025 — free online.
• GitHub — Security hardening for GitHub Actions. GitHub Docs — free online.
• OpenSSF — SLSA: Supply-chain Levels for Software Artifacts, v1.2. OpenSSF, 2025 — free online.
• Zachary Newman, John Speed Meyers, Santiago Torres-Arias — Sigstore: Software Signing for Everybody. ACM CCS, 2022 — free online.
• Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala — We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs. USENIX Security, 2025 — free online.
• AWS — Security bulletin AWS-2025-015 (Amazon Q Developer for VS Code). AWS, 2025 — free online.
• Kelly Shortridge, Aaron Rinehart — Security Chaos Engineering: Sustaining Resilience in Software and Systems. O’Reilly, 2023.
• CISA — Implementing Phishing-Resistant MFA. CISA, 2022 — free online.
• Eugene H. Spafford — The Internet Worm Program: An Analysis. Purdue Technical Report CSD-TR-823, 1988 — free online.

Chapter 8: Operating AI Workloads — Models, Evals, and Agents

A fluent answer is not yet a correct one, and production is where we learn the difference.

Between 1964 and 1966, at MIT, Joseph Weizenbaum wrote a program he named ELIZA, after Shaw’s flower girl, who is taught to speak like a duchess. The name was a small joke with a serious point inside it, because the program, like the flower girl, had learned to sound like something it was not. Its best-known script, DOCTOR, imitated a Rogerian psychotherapist, the kind of therapist who answers a statement by turning it back into a question. Tell such a therapist that you are unhappy at work, and you are likely to be asked why you feel unhappy at work. The choice of role was shrewd, since a therapist who mostly reflects can say very little of substance and still seem attentive. There was no understanding anywhere in the program. It scanned what the patient typed for keywords and applied rules that rearranged the patient’s own words into a reply, and Weizenbaum explained exactly this, plainly, in 1966.

People mistook it for more anyway. In Computer Power and Human Reason (1976) he recalled that his secretary, who had watched him build the program over many months and so knew better than almost anyone what was inside it, began a conversation with it and after a few exchanges asked him to leave the room. “What I had not realized,” he wrote, “is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.” The lesson concerned us more than the program. In ordinary life, fluency and understanding travel together: a person who speaks easily and sensibly about a subject usually knows something about it, and so we have learned to treat the first as evidence of the second. ELIZA pulled the two apart, and the habit went on working on the wrong evidence. We lend trust to fluency, and we lend it quickly, before anything has earned it.

Today’s models are vastly more capable than ELIZA and are often right, and that sharpens Weizenbaum’s warning instead of retiring it. ELIZA was wrong so often that a little skepticism exposed it. A component that is usually right teaches the people around it to stop checking, and when it is wrong, the wrong answer arrives in the same confident, well-formed voice as all the right ones. This is Bainbridge’s irony of automation in a new costume: the better the machine performs, the less practiced its human overseers become at catching the moment it does not.

Encore now depends on such components. The Architecture book designed three of them. The first is a support assistant built on retrieval-augmented generation (RAG), which means that before the model answers a fan’s question, the system retrieves relevant passages from Encore’s own material and asks the model to answer from them, so that its replies rest on Encore’s policies instead of on whatever the model happens to remember. The second is a refund agent that settles refund and rebooking requests by calling Encore’s own services. The third is the AI gateway, through which every model call passes, so that routing, quotas and costs are managed in one place. (The map of Encore in Chapter 1 shows where each piece sits.) Designing these features was the Architecture book’s work; this chapter is about running them, day after day, under real traffic and real adversaries. We will follow them through two episodes. In the first, the model behind the assistant changed although nobody at Encore had asked for a change, and the assistant began explaining the refund rule subtly wrong while every dashboard stayed green. In the second, somebody tried to talk the refund agent into paying them by hiding instructions inside a support ticket. Both episodes make sense only once we see clearly what makes a model different from every other dependency this book has taught us to operate, and so that is where we begin.

8.1 What Is Different About AI in Production

A dependency with opinions

Almost everything in this book so far has rested on one quiet assumption: a component, given the same input, does the same thing. We have leaned on that assumption so often that it has stopped looking like an assumption at all. Monitoring works by comparing today with yesterday, and a change in the numbers means something changed in the system. A canary works by comparing the new version with the old one on the same traffic, and a difference between them is blamed on the release. A test works by comparing an output with an expectation written down in advance, and a mismatch means a bug. Each of these instruments is a way of detecting difference, and each one is only trustworthy when the thing being measured would otherwise hold still.

A large language model (LLM) reached through an API looks, from the outside, like any other managed service of the kind Section 3.3 taught us to operate: someone else runs the hardware, we call an endpoint, we pay for what we use, and we watch its errors and latency from our side of the wire. The shape is familiar. What it no longer does is honor the assumption. Figure 8.1 lays the two kinds of dependency side by side, property by property.

Figure 8.1 — The same dependency, seen twice. Each row on the right breaks an assumption our operations quietly rely on.

It helps to have an image for the difference, and Encore’s own business offers one. A conventional dependency is like a recording: press play twice and we hear the same performance twice, note for note, which is exactly why a sound engineer can compare two pressings and hear a defect. A hosted model is more like a live band. Ask for the same song and we get the same song, recognizably, but the phrasing, the tempo and the ending differ from night to night, and no two performances can be laid on top of each other and compared sample by sample. The analogy is useful up to a point, and the point where it breaks is the important part. When a live band plays a wrong note, the audience hears it. When a model gives a wrong answer, it delivers it with the same polish, the same confident tone and the same tidy formatting as a right one. The wrongness makes no sound.

Take the rows one at a time, beginning with determinism. Temperature is the setting that controls how much randomness a model uses when it chooses each next token; at temperature zero it should, in principle, always choose the single most likely one, and so give the same answer every time. In practice even temperature zero is not reliably repeatable, and the reason has nothing to do with the model’s intentions. A hosted provider serves many customers at once and groups their requests into shared batches on the same hardware. The other customers’ requests in our batch change the order in which the provider’s arithmetic is carried out, and floating-point arithmetic done in a different order can come out very slightly different. Most of the time the difference vanishes. Occasionally it is enough to tip which token is most likely, and once one token differs, every token after it is generated from a different starting point. Our output depends, in a small way, on strangers’ traffic.

The second row changes the daily work of operations more than any other. Machine learning systems fail silently, as Chip Huyen put it in Designing Machine Learning Systems: they return confident, well-formed and wrong answers through an endpoint that is, by every measure we usually take, perfectly healthy. The HTTP status is 200. The latency is good. The response parses. Consider what this means at Encore. If the support assistant starts telling fans the wrong refund rule, every dashboard we built in the earlier chapters stays green: the assistant has 100% availability and excellent latency throughout. The failure lives entirely in the meaning of the words, and only an instrument that actually reads what the assistant said will notice it. That is a new kind of instrument for most operations teams, and building it is the work of Section 8.3.

The third row deserves the same attention, because it removes something we had come to take for granted: control over when our dependencies change. A database we run changes when we change it, through a migration we wrote, reviewed and scheduled. A hosted model changes when its provider retrains it, re-tunes it or retires it, and the provider’s calendar is not ours. The best-known demonstration came in 2023, when Lingjiao Chen, Matei Zaharia and James Zou put the same questions to GPT-4 in March and again in June, under the same model name. Over those three months its accuracy at identifying prime numbers fell from 84% to 51%, and it had grown less willing to answer some sensitive questions. Nobody at the calling end had deployed anything; the thing behind the name had simply become different.

Providers do announce some changes. They publish deprecation schedules: dated notices after which a particular model version stops answering requests. These belong on an operations calendar as surely as a certificate expiry, with one wrinkle worth knowing in advance. The same model is often sold both through the provider’s own API and through the large cloud platforms that resell it, and the retirement date can differ between them, so the date that matters is the one on the platform we actually call. Other changes are easier to miss, such as a convenient model alias quietly being moved to a newer snapshot; Section 8.2 tells how Encore learned that one.

None of this is entirely new in kind, only in degree. In 2015 D. Sculley and colleagues at Google, writing about the hidden technical debt in machine learning systems, named the governing principle CACE: changing anything changes everything. Their point was that a learned component entangles its inputs, so that a small change in one place shifts behavior everywhere. They were writing about models their own teams trained and owned. With a hosted model the principle still holds, but some of the changes are now made by someone else, and we discover them only if we are measuring the right things.

Budgets in tokens and seconds

The last three rows of Figure 8.1 share a single theme: the token has replaced the request as the unit in which a model’s work is counted. A token is a fragment of text, often a word or part of a word, and a model both reads and writes in them. This one change of unit reaches into cost, latency and capacity at once.

Cost comes first, because it explains the startling figure in the table. A conventional service charges per request or per hour, so one call costs about what any other call costs. A model charges per token, both for the tokens we send it and for the tokens it writes back. A fan asking a one-line question that gets a one-line answer is a cheap call. The same fan asking a question that makes the assistant pull several long policy passages into its context and then write a careful, detailed reply is an expensive one. The two calls look identical to a request counter, yet one can cost 100× the other. This is one reason every model call at Encore passes through the AI gateway, and why the gateway sends most turns to a much cheaper model and saves the expensive one for the turns that need it.

Tokens are also the unit of capacity, which matters most on exactly the days Encore cares about most. Providers cap both requests per minute and tokens per minute for each model family, and some will throttle usage that climbs too sharply even when it is still under quota, because a sudden ramp is hard for them to absorb. Encore’s on-sales produce precisely that shape. On a big on-sale morning, support traffic triples within the hour as fans with questions about the sale all ask for help at once. That is the very ramp such limits are designed to punish. For this reason the token quota now sits in the on-sale readiness review, on the same page as the Sale Gate’s admission rate, and is checked with the same seriousness. For the largest on-sales Encore goes further and buys reserved capacity, the priority tiers or provisioned throughput that providers sell to customers with predictable peaks. A quota increase requested on the morning of the sale arrives too late to help; capacity for a peak we can see coming has to be bought before it comes.

Latency has a new shape as well. In a conventional service, latency is a single distribution measured in milliseconds: the request goes in, the response comes out, and we track the p95 and p99 of the gap. A model writes its answer one token at a time, so its latency splits into two quite different quantities, measured in seconds. Huyen’s AI Engineering separates them. Time to first token (TTFT) is how long the user waits before anything appears on the screen. Time per output token is how quickly the rest of the answer follows once it has started. The user feels the first: a blank screen for four seconds feels broken, while a reply that begins at once and then streams steadily feels responsive, even if it takes longer in total. The bill and the queue feel the second, since every output token is paid for and occupies capacity while it is being written.

These differences come together in the way Encore states what “working” means for the assistant. Its service level objective (SLO) has three parts. The first is gateway availability, the familiar kind of objective from earlier chapters. The second is a TTFT under 1.5 s for 95% of turns, a target that quietly presumes two design choices: that the reply is streamed to the fan as it is written, and that the model answering is not one that deliberates at length before it begins. The third is a quality objective, a measure of whether the assistant’s answers are actually right, held to a target like the other two. The first two objectives would have been at home anywhere in this book. The third is the one Figure 8.1’s second row makes necessary, since an assistant can meet both of the others while telling fans the wrong thing. Quality objectives wired to alerts are young practice, and I have come to believe they are where most of the real operational work on AI features lies; much of this chapter is about how to set one well.

Recap. A hosted model keeps the outward shape of a managed dependency, but it withdraws the determinism our instruments depend on, fails in the content of its answers rather than in its status codes, changes on its provider’s schedule instead of ours, and meters its cost, latency and capacity in tokens. Its SLOs therefore need a quality component alongside availability and latency, and its quotas, ramp limits and retirement dates belong on the on-sale calendar next to everything else that can stop a sale.

Exercise 8.1. Choose one model-backed feature you run and fill in Figure 8.1’s right-hand column with real values for it: the exact model version, its retirement date on the platform you actually call, your token quota and your observed peak, and your p95 TTFT. Mark each value you had to look up rather than already knew, since those marks show where your operational picture of the feature has gaps.

8.2 Model and Prompt Changes as Deployments

What actually ships

Section 8.1 showed that a hosted model can change beneath us. The first practical consequence concerns our own changes, and it begins with a simple question: when an AI feature starts behaving differently, what changed? In a conventional service the answer is almost always the code. In an AI feature the answer is usually something that does not look like code at all: the prompt, the model version, the temperature setting, the descriptions of the tools the model may call, or the settings that decide which documents retrieval puts in front of it. Each of these is a few lines of text or a single number, and each can change what the feature does as much as a rewrite of the code would. A sentence added to a system prompt can alter how every refund is explained; a new model version can alter how every sentence is read. Because these changes carry the weight of code, each of them goes through the pipeline Chapter 4 built for code, with the same review, the same tests and the same staged rollout.

To do that, we first need a name for the thing being shipped, and the theater already has one. The prompt book is the stage manager’s master copy of the script, annotated with every lighting, sound and scenery cue. An actor may try something new in rehearsal, and a designer may move a cue by half a second, but the change counts only once it is written into the book, because the book is what the crew follows on the night. The analogy fits our problem well: an AI feature’s behavior is spread across many small settings, and unless they are gathered into one authoritative record, nobody can say exactly which version of the show is playing. Encore’s prompt bundle is that record: the versioned unit that defines an AI feature’s behavior, and that ships whole or not at all. Here is what it contains:

Figure 8.2 — Encore’s prompt bundle. Changing any row produces a new bundle version, which passes the same gates as a code change.

Each row earns its place for a reason worth spelling out. The system prompt is business logic, only written in English instead of a programming language. The sampling parameters shape how adventurous the model’s answers are. The tool descriptions are easy to mistake for documentation, but the model reads them as instructions, so rewording one can change when and how a tool is called. The retrieval configuration decides which help articles and policies the model sees, and a model cannot follow a policy it was never shown. The analogy with the prompt book breaks in one instructive place. A stage manager’s book is read by people, who notice an odd cue and ask about it; a prompt bundle is read by a model, which follows whatever it is given with equal confidence. The discipline of the book matters even more for us than for the theater, because nobody downstream will question an error.

Gathering everything into one bundle that travels through the pipeline guards against what Sculley and his colleagues called configuration debt: the slow accumulation of settings that shape a machine learning system’s behavior but live outside the discipline applied to its code, scattered across files and consoles, rarely reviewed and seldom tested. The idea comes from their 2015 paper; the bundle’s exact shape is Encore’s own. Two choices within it deserve a closer look.

The first is how the model is named. The bundle names it by a pinned snapshot ID, an identifier for one specific, fixed version of the model, and never by a floating alias, a convenient name such as “the current small model” that the provider can repoint at a new version whenever it likes. Pinning fixes the model’s weights, and so it largely prevents the provider-side drift that Section 8.1 described. “Largely” is the right word. Providers acknowledge that changes to their serving infrastructure, to how they batch and run the model, can still shift a pinned model’s behavior slightly. Pinning narrows the problem without removing it, and we shall see shortly why that small residue matters.

The second is that the eval suite travels with the bundle. An eval suite is the set of test conversations and checks that Section 8.3 develops, and its pass thresholds define what “good” means for this feature. If the evals lived elsewhere, someone could loosen a threshold in one place while changing the prompt in another, and no reviewer would see the two together. Keeping them in the bundle means that a change to what “good” means appears in the same diff as the change it judges, where a reviewer can ask whether the bar moved to fit the new behavior.

With the bundle defined, its journey to production is the one Chapter 4 laid out for any change. The bundle is promoted through GitOps: its desired state lives in a Git repository, and an automated process makes production match that state. Continuous integration (CI) runs the eval suite of Section 8.3 against the new bundle before it goes anywhere. Then a canary in the manner of Section 4.3 sends a small share of real traffic to the new bundle and compares it with the current one, the baseline, on three measures: the escalation rate (how often conversations are handed to a human), the quality of a sample of answers, and cost.

Two details of the canary are specific to conversations. First, each conversation stays in one arm of the experiment from start to finish. A fan whose conversation switched bundles halfway through would hear two different voices, and any difference in outcome could no longer be attributed to either bundle. Second, the canary’s duration is set by two conditions together. It runs at 5% of traffic for at least a day, so that it sees the full daily rhythm of support traffic, and it keeps running until every watched category, refunds included, holds enough conversations to show a regression if one exists. The second condition is the one that matters most. Refund conversations are a small fraction of the total, so 5% of a day’s traffic may contain far too few of them to reveal anything, and a canary that ended on the clock alone would declare the bundle safe for refunds simply because it had not yet met enough of them.

Rollback is a git revert: restore the previous bundle in the repository, and the automation restores it in production. That is as simple as rollback gets, with one condition that code rollbacks do not share. Reverting works only while the previous bundle names a model that still exists, and Section 8.1’s deprecation schedules guarantee that every model eventually stops answering. A rollback plan for an AI feature therefore has an expiry date. Encore records that date in each plan, so that nobody discovers in the middle of an incident that the safe version they are reaching for has been retired.

The upgrade nobody scheduled

Encore learned the pinning rule the way most rules in operations are learned, by breaking it first. The assistant uses a cheap tier, a smaller and less expensive model that answers about 80% of conversational turns. That cheap tier had been configured during a hurried launch, and it referred to the model by the provider’s alias for its current small model. The alias looked like a version, so nobody thought of it as something that could change. One Tuesday the provider moved the alias to a new snapshot. It had announced the move in advance, in a notice that nobody at Encore had read. Nothing failed: no errors, no timeouts, no alerts. Every dashboard stayed green, which is exactly the silent failure Section 8.1 warned about.

What changed was one word. Encore’s refund workflow approves a refund automatically only when two conditions both hold: the ticket costs under €80, and more than 48 h remain before showtime. Everything else goes to human review. In roughly a third of refund conversations, the new model explained the rule with or in place of and, turning two conditions that must both hold into two conditions of which either will do. Consider a fan with a €140 ticket for a concert five days away. The fan satisfies only the timing condition, so under the real rule the refund goes to a person. The assistant told this fan that the refund would be automatic, and the fan then watched it go to human review. The workflow did the right thing; the assistant had promised the wrong thing.

The nightly eval run caught it, and the reason that run exists is worth understanding. Evals in CI test a bundle when someone changes it, but this change involved no commit and no deployment, so CI never saw it. Encore therefore also runs its golden set of 200 support conversations every night against the production configuration, exactly as it is serving fans. The reasoning is the residue from the pinning argument above: configuration can change without a commit, as the alias just had, and even a properly pinned model can shift slightly as its provider changes how it is served. A test that runs only when we change something cannot catch a change that someone else made.

Eighteen of the 200 golden conversations concern refunds. For months they had passed 18 of 18. On Wednesday morning, 11 passed. The Support Experience on-call engineer read the seven failing conversations one by one, saw the same misplaced “or” in each, and recognized one fault where the numbers alone suggested seven. By lunchtime the cheap tier was pinned to the previous snapshot.

Figure 8.3 — The nightly golden-set run across the week the alias moved. The refund cases fell from 18 of 18 to 11 while the overall score barely moved, which is why the team reads each category before the total.

The figure carries a lesson about averages. Seven failures out of 200 is a drop of a few percentage points in the overall score, the kind of dip a team might put down to noise. Seven failures out of 18 refund cases is a loss of more than a third in the one category where a wrong answer carries a financial promise. The same seven conversations look trivial in one view and alarming in the other, and only the per-category view shows what is happening. An average is a summary, and a summary is precisely where a small, serious failure goes to hide. That is why Encore’s engineers read each category before they look at the total.

It would be comfortable to conclude that no harm was done, because the refund policy lived in the workflow and the workflow never changed. That conclusion is half right. It is true of the money: the ledger paid nobody wrongly, because the assistant’s words have no power to move funds. It is untrue of the promises. A fan who was told a refund would be automatic has been told something by Encore, and courts and tribunals have begun to treat such statements seriously. In Moffatt v. Air Canada (2024), a British Columbia tribunal held the airline to a bereavement refund that its chatbot had described wrongly, rejecting the airline’s argument that the chatbot was responsible for its own words. What an assistant tells a customer can create an obligation for the company that deployed it. I have come to believe this is where many teams misjudge AI features: they guard the actions carefully and treat the words as harmless, when the words are themselves a kind of action.

So the harder question was not how to fix the fault, which took a morning, but how many fans had heard the wrong rule in the 30 hours before the rollback. A golden set cannot answer that. It tells us how the assistant handles 200 conversations written in advance; it says nothing about the real conversations that took place while the fault was live. Answering it means looking at production itself, which is where the next section’s evals eventually lead.

Recap. Everything that shapes an AI feature’s behavior, from the prompt and the pinned model to the tool descriptions, retrieval settings and evals, ships as one versioned bundle through the ordinary pipeline, and its canary runs until every watched category has enough traffic to show a regression. Because a model and its serving can change without a commit, evals also run nightly against production, and a rollback works only as long as the old model is still offered. The assistant’s words are part of the company’s obligations, so a fault in what it says is an incident even when no money moves.

Exercise 8.2. List every artifact that shapes the behavior of one AI feature you run: prompts, model identifiers, parameters, tool descriptions, retrieval settings and evals. For each, note where it is stored, whether a change to it passes through CI, and whether it can be rolled back in under 10 minutes. Any “no” marks a deployment path you have not yet built, and any model identifier that is an alias marks a change someone else can make for you.

8.3 Evals — Testing Systems That Don’t Repeat Themselves

Error analysis before metrics

The nightly run that caught the refund fault deserves a closer look, because it was not luck. It was the product of a discipline, and that discipline is the subject of this section. The Architecture book explained why a probabilistic component is evaluated rather than unit-tested: when the same input can yield different outputs, we cannot assert one expected answer, and so we judge whether each answer falls within the range of acceptable ones. Here we take up the operating side of that idea. Shreya Shankar and Hamel Husain’s Evals for AI Engineers (2026) teaches evals as a weekly practice rather than a project finished once, and its method begins at the step most teams skip in their hurry to get a number on a dashboard. Before choosing any metric, we read traces, the recorded conversations themselves.

Error analysis is the practice of reading a sample of real interactions, noting in plain words what went wrong in each, and grouping those notes into a taxonomy of failure modes, continuing until new traces stop producing new categories. Its lineage lies outside engineering, in the open coding of qualitative research, where a researcher reads interview transcripts without a fixed scheme and lets the categories emerge from the material rather than imposing them in advance. The practice sounds slow, and in the first hours it is. It is nonetheless the fastest route to useful evals, because it builds them around the failures we actually have instead of the failures we imagined before launch. There is a further reason, and it is a humbling one: Shankar’s own research found that people’s criteria for a good output shift as they read outputs. We do not fully know what “good” means for our assistant until we have seen enough of what it says, so a metric chosen before that reading is a guess about our own standards.

Encore’s refund fault, seen now through this lens, shows why the order matters. Across the whole golden set, the overall score moved only from 193 to 186 out of 200, a dip of a few points that teams soon learn to shrug at, since scores of this kind wobble from night to night. What told the real story, as Section 8.2 recounted, was the on-call engineer’s reading of the seven failures, which revealed one fault repeated seven times: the model had turned the rule’s and into or. That reading was error analysis in miniature, done under pressure. The team’s first response was the simplest instrument available, a code check that any explanation of the refund rule states both conditions jointly, the €80 price limit and the 48 h before showtime. That choice reflects a general rule of the craft. Wherever a property can be checked deterministically, by code that gives the same verdict every time, we check it that way first, because such a check is cheap, fast and beyond argument.

Judges need calibrating

Many properties resist code, however. Whether an answer is grounded in the retrieved documents, whether it is courteous, whether it actually addresses what the fan asked: these need judgment. For them we use a model as a grader, the LLM-as-judge, a second model prompted to score the first model’s output against a stated criterion. It is tempting to treat the judge’s scores as facts. They are readings from a measuring instrument, and like any instrument the judge must be calibrated before its numbers mean anything, just as a laboratory scale is checked against known weights before anyone trusts what it reports.

The question of how to calibrate a judge is older than computing. In 1960 Jacob Cohen published his kappa statistic for agreement between two raters beyond chance, because two people coding the same interviews agree less than either believes, and raw agreement flatters them. To see why chance matters, suppose that nine conversations in ten are perfectly fine. A lazy judge that marks every conversation “fine” will agree with a careful human nine times in ten, and yet it has learned nothing at all. Kappa corrects for this. It asks how much of the observed agreement exceeds what two raters would reach by guessing in proportion to how often each uses each label, and it scales the result so that zero means no better than chance and one means perfect agreement. Our lazy judge would score a kappa of zero, which is exactly what it deserves.

Shopify has published the clearest production account of what calibration buys. For its Sidekick assistant, the judge’s agreement with human experts rose from a kappa of 0.02, essentially chance, to 0.61, against 0.69 for two humans judging the same cases. That last figure is the important one. Human experts do not agree perfectly with each other either, and so their agreement is the ceiling against which any target for the judge should be set; demanding that a judge agree with people more than people agree with one another asks for something no instrument can deliver.

Encore follows the same method. Its support team labels conversations by hand, tunes the judge’s prompt on one split of those labels and measures agreement on a separate held-out split that the tuning never saw, which keeps the team from fooling itself the way students fool themselves by memorizing the answers to a practice exam. The team measures agreement per category, because a judge can be excellent on easy cases such as order-status questions and poor on refunds, and a single overall kappa would hide the weakness precisely where it matters most. And because the judge is itself a model, subject to everything Section 8.1 said about models, it is re-checked monthly and whenever its snapshot changes. An instrument that can drift must be recalibrated on a schedule.

Non-determinism also changes what passing means. A conventional test that passes once will pass again; a golden case for a model may pass on one run and fail on the next. Encore therefore runs each golden case five times and counts it as passed only if all five runs pass. This is the pass^k measure from Yao and colleagues’ τ-bench (2024): the probability that a case succeeds on every one of k attempts, rather than on at least one. The stricter measure matches what fans experience. A case that succeeds 80% of the time looks healthy on a single run, yet it passes all five runs only about a third of the time (0.8 multiplied by itself five times is roughly 0.33), and that third is a fairer picture, since a fan who meets the one failing run in five simply gets a wrong answer and has no way of knowing it.

Small categories have statistical limits, and honesty requires us to name them. With 18 refund cases, each case is worth more than five percentage points. A fall from 18 passing to 11 is unmistakable, and that is the fall Encore saw. A fall to 16 could be noise, two unlucky runs on a bad night, and no threshold on a set this small can tell the difference reliably. Encore accepts that the golden set will catch large regressions and leaves subtle ones to production, which brings us to the second half of the practice.

From golden set to production sampling

Medicine faced Encore’s question long before software did, and learned the answer at terrible cost. After the thalidomide disaster of the early 1960s, regulators confronted an uncomfortable truth about clinical trials. A trial tests a drug on chosen patients under watchful conditions, but the drug’s real behavior emerges only across everyone who takes it, in combinations and circumstances no trial anticipated. And so regulators built post-market surveillance, the continuing collection of evidence about a drug once it is in use. The parallel to our work is close. A golden set is the trial: chosen cases, run under controlled conditions before release. Production sampling is the surveillance: evaluating a share of real interactions with the same judges and code checks, sending a smaller share to people for review, and watching the results as a time series rather than a single verdict. The analogy breaks at one point, and the break is instructive. An approved drug does not change after approval, while a hosted model can, as Section 8.2’s alias move showed. Our surveillance must therefore watch not only for rare effects in an unchanging product but for a product that may quietly become a different one.

Encore needed exactly this surveillance on the Wednesday of the refund fault. The golden set had shown that the model was wrong; it could not say how many fans had heard the wrong rule in the 30 hours before the rollback. Answering that question took two engineers a day, because the evaluator for refund explanations had to be built first and then run over the past conversations. Of about 520 refund conversations in the 30-hour window, it found 164 faulty explanations. With Moffatt in mind, Encore wrote to each of those fans with the correct rule, expedited their reviews and honored any automatic refund the assistant had promised. Then came the structural fix, which mattered more than the cleanup. The assistant no longer paraphrases the refund rule at all. It fills a template from the workflow’s computed verdict, so the words a fan reads come from the code that moves the money. Where a fact can be computed, it is safer to compute it and let the model carry the result than to ask the model to restate the rule and hope it restates it faithfully.

That day’s work became a standing practice. Encore now evaluates a 3% daily sample of conversations and sends 40 a week to human reviewers. The human sample is not drawn uniformly: it oversamples conversations where the judge was uncertain or the fan escalated, because those are where the automated instruments are least trustworthy and a human reading teaches the most. Transcripts are stripped of payment details before anyone reads them and are kept under the same retention rules as support tickets, since a sample of real conversations is still customer data.

The sampled pass rate on refund explanations has become a quality service level indicator (SLI), a measured signal of how well the service is serving its users, with an objective of 98%. Its burn rate, the speed at which failures are consuming the allowed 2%, pages the Support Experience team during business hours, in the manner of the alerts Chapter 1 built for availability. This is Encore’s own adaptation of Chapter 1, without a public precedent I know of, and it has a known weakness. Judge noise makes the signal jumpy, because a judge with imperfect agreement will mark some good answers bad and some bad answers good, and over a small window those errors alone can look like a burn. The alert therefore fires only on windows holding 150 or more sampled refund conversations, enough that a real regression stands out from the instrument’s own scatter.

The two loops also feed each other. Every failure found in production becomes a new golden case, so over time the golden set grows toward the conversations fans actually have rather than the ones its authors imagined, as Shopify’s does. The trial learns from the surveillance. Figure 8.4 places both loops beside the request path.

Figure 8.4 — The request path through the AI gateway, with the eval loop beside it. The quality SLI reaches the gateway only through the on-call engineer.

Practices that live only in people’s habits erode under deadline pressure, and so Encore gathers these into one file in the prompt bundle. CI refuses to promote a bundle that misses any line of it. The file reads almost as a summary of this section: the golden set and where it came from, the five-run rule, gates set per category rather than on the overall average that hid the refund fault, a minimum kappa for every judge with the human ceiling recorded beside it, and canary conditions large enough to show a regression in each category:

# release-gate.yaml: in the prompt bundle; CI blocks promotion on any miss
golden_set: golden/2026-09        # 200 cases, grown from error analysis and production
runs_per_case: 5                  # a case passes only if all five runs pass (pass^5)
gates:                            # per category; the overall average never gates
  refund_explanation: { min_pass: 1.00, evaluator: code/refund_rule_joint }
  order_status:       { min_pass: 0.95, evaluator: judge/grounded@v7 }
  injection:          { min_pass: 1.00, evaluator: code/no_out_of_policy_tool_call }
judges:                           # no gate may rest on an uncalibrated instrument
  judge/grounded@v7:  { min_kappa: 0.60, human_kappa: 0.70, labels: heldout/2026-09 }
canary: { share: 0.05, sticky: conversation, min_hours: 24, min_per_category: 150 }

Notice that the refund gate demands a pass rate of 1.00 and rests on the code check born from the error analysis, not on a judge. The lesson of the or has been written into the one place every future change must pass through.

Recap. Evals begin with error analysis: reading real traces until the failure categories stop growing, so that our metrics describe the failures we actually have. Code checks come first wherever a property can be computed; LLM judges are instruments, calibrated against human labels per category, with the agreement between humans as their ceiling. The golden set is the trial and production sampling the surveillance, and a release gate in the bundle makes both binding on every change.

Exercise 8.3. Read 50 recent traces from an AI feature you run, write one line per failure in plain words and group the lines into at most eight categories. Which failures have no metric on your dashboard, and which of your metrics match no failure you found?

8.4 Failure Modes, Guardrails, and Graceful Degradation

A field guide to content failures

The previous section built instruments for noticing when the assistant’s words go wrong. Instruments are only half of the work, though, because a signal that nobody knows how to answer is merely an alarm. Medicine learned long ago that a disease has to be named before it can be treated, since the name is what connects a set of symptoms to a test that confirms them and a treatment that helps. Content failures deserve the same discipline. Each kind of failure needs a name, each name needs a detector that tells us it is happening, and each detector needs a planned response that limits the harm. The table below is Encore’s field guide, five failure modes with the way each one shows itself, the instrument that catches it and the mechanism that contains it:

Figure 8.5 — Content failure modes with their detectors and containment. The right-hand column lives outside the model, so it keeps working when the model misbehaves.

The right-hand column deserves a second look, because it holds the table’s most important idea. Every containment mechanism there (a citation requirement, a pinned version, a step cap, a scoped credential, a policy enforced by the workflow) is ordinary software that sits outside the model. None of them depends on the model behaving well, and that is the point: a guard that relies on the very component it guards fails at the moment it is needed. We cannot make a fluent model incapable of error, but we can arrange the system so that its errors land somewhere soft.

Two rows in the table are easier to underestimate than the others. The first is drift. A drift is a slow decline in quality that arrives without any deploy of ours, and a hosted model faces three of them at once: the corpus it retrieves from changes as help articles are edited, the questions fans ask change with the season and the shows on sale, and the provider changes the model itself. Pinning a model snapshot, as Section 8.2 described, largely prevents the third kind. Nothing prevents the first two, and only evals catch them, because the code, the configuration and the model can all be exactly as they were while the world around them moves.

The second is refusal. A refusal is the model declining a request it ought to answer, and refusals are easy to miss because they look safe. Nobody writes an incident report about an assistant that was too careful. Yet Chen and her colleagues, whose study of GPT-4 we met in Section 8.1, watched the model grow more reluctant to answer some questions between its March and June releases under the same name. Imagine that happening to Encore’s assistant: it begins politely declining to discuss refunds, every request returns quickly and without error, and the dashboard stays green. From the fan’s side, the assistant is down. That is why the table measures refusal rate per intent and keeps eval cases that the model must answer, so that a loss of nerve after an upgrade shows up as a failing test rather than as a slow trickle of frustrated fans.

The degradation ladder

Guardrails, the checks the Architecture book layered from schemas to grounding, work one answer at a time: they catch a single bad response and replace or block it. Operations needs something more, because sometimes a whole tier fails at once, and then the question is no longer how to fix one answer but what lesser service to offer in its place. That question should be answered long before anything fails. Graceful degradation is the planned, step-by-step retreat to simpler and more dependable forms of service when the preferred form is unavailable or untrustworthy.

Aviation has practiced this for decades through the minimum equipment list, a document that states in advance which instruments an aircraft may fly without, and under what conditions. Its value lies less in the list itself than in when it was written. Nobody decides at the gate, with passengers boarding and the departure slot approaching, whether the flight can go without its weather radar; that was decided months earlier, calmly, by people who were not running late. The analogy fits Encore well, with one difference worth noting: an aircraft’s instruments either work or do not, whereas a model can be fully available and still be wrong, a case we will return to shortly.

Encore’s ladder has four rungs, and each one is exercised on game days so that it is known to work before it is needed:

1. Frontier tier for hard turns and cheap tier for the rest, the normal state.
2. Fallback provider: a second provider’s model, pinned and run against the golden set weekly; degraded by design, since prompts do not transfer cleanly, with quota sized for failover load.
3. Retrieval only: the assistant shows the most relevant help articles and the fan’s order status, without generating prose.
4. Human handoff: the conversation joins the support queue with a summary, and the fan is told honestly how long the wait is.

Read from top to bottom, the ladder trades capability for dependability. Each rung down gives up something fans value (fluent answers, then generated prose altogether, then an immediate reply) in exchange for something that is harder to break. The second rung is labeled “degraded by design” because a prompt tuned for one model does not carry over cleanly to another, so the fallback answers less well, and Encore accepts that openly rather than pretending the two are interchangeable. Retrieval only removes generation entirely, which removes hallucination with it, since showing a help article and an order status involves no model writing anything. The last rung hands the fan to a person and tells the truth about the wait, which is a form of service even when it is a slow one.

The second rung also carries the most uncertainty, and it is worth being honest about why. Multi-provider gateways run at scale, Uber’s among them, so routing between providers is ordinary engineering. What I know of no published account of is a cross-provider failover during a real outage, when the first provider is down for everyone and other customers crowd onto the same second provider at the same moment. The quota Encore reserved might hold, or the second provider might buckle under everyone’s failover at once. Encore therefore treats that rung as a hypothesis rather than a guarantee, and tests it on game days. The rungs themselves are pulled by the gateway’s circuit breakers, which trip on errors, latency or rate-limit exhaustion, using the patterns of Section 5.2.

The trigger teams forget is quality. A circuit breaker watches for a dependency that is failing in ways a machine can see: errors, timeouts, refused requests. A model that is available but wrong produces none of those signals, so it never trips a breaker, and this is exactly where the minimum-equipment-list analogy breaks down. Encore does watch quality, through the refund-explanation SLI of Section 8.3, but here it deliberately stops short of automation. When that SLI burns its error budget at a sustained rate, it pages a person. That person reads a few of the failing conversations, and if they agree the model is wrong, they flip a prepared switch, already tested on game days, that drops refund intents to the retrieval-only rung.

Why not let the SLI flip the switch itself? Because a judge-based SLI is noisy. Its readings come from a model judging another model on a sample, and some of its failures are the judge’s mistakes rather than the assistant’s. A false trip would push every fan with a refund question down to retrieval only, a real loss of service caused by a measurement error. The trade-off, then, is between the speed of an automatic response and the cost of acting on a signal Encore does not yet fully trust. Until the team has watched that signal through a year of on-sales and learned how it behaves, it keeps a person between the measurement and the switch. I have come to believe this is the right order for any new signal: earn trust in the instrument first, and only then hand it the controls.

Figure 8.6 — The degradation ladder and what pulls it. Breakers see errors, latency and rate limits; only the quality signal sees a model that is up and wrong, and it reaches the ladder through a person.

Putting a person in the loop solves one problem and opens another, and Weizenbaum would recognize it. His secretary, as the chapter’s opening recalled, trusted ELIZA within minutes; Encore’s reviewers face a slower version of the same pull. After months of good proposals from the refund agent, they begin to approve faster and read less. This is the irony of automation the chapter opened with, and here it is made worse because the agent’s proposals are so well written. A clumsy proposal invites scrutiny; a fluent one invites a nod.

Encore answers the trap in two ways. The first separates facts from prose. The review screen shows the policy facts (the ticket price, the hours remaining before showtime, and the refund rule’s verdict) in a panel computed by the workflow, set apart from the agent’s explanation. The reviewer can check the decision against numbers the agent did not write, so a persuasive paragraph cannot quietly substitute for a correct one. The second measures attention directly. A few times a month, Encore seeds the queue with a proposal it knows to be wrong and tracks how often reviewers catch it. The precedent is airport security, where threat image projection inserts fictional weapons into real X-ray images, both to keep screeners alert and to measure what they actually catch. Attention that is never tested cannot be known, and a review step whose catch rate is unknown is a review step in name only.

Recap. Each content failure needs a name, a detector that notices it and a container outside the model that limits its harm. A degradation ladder, decided in advance like a minimum equipment list, steps down from the frontier model to human handoff; circuit breakers pull it automatically on errors and latency, while a burn of the quality SLI pages a person who decides. Human review stays meaningful only if the interface separates computed facts from fluent prose and the team measures the reviewers’ attention instead of assuming it.

Exercise 8.4. Write the degradation ladder for one AI feature you run, naming each rung’s trigger, who or what pulls it, and when it was last exercised. Add one quality trigger, and decide whether a person or a machine acts on it, giving your reason.

8.5 Agents with Permissions — Sandboxing, Injection, and Blast Radius

Credentials shaped like the task

Everything in this chapter so far has concerned a system that talks. The support assistant explains, summarizes and occasionally misleads, and the worst it can do on its own is say something wrong. The refund agent is a different kind of component, because it acts: it reads orders, requests refunds and moves money toward people. An assistant that only talks can mislead a fan; an agent that acts can do damage that no apology reverses. The Architecture book gave Encore’s refund agent delegated authority, meaning the agent acts on behalf of a particular fan rather than in its own right. That was a design decision made on a whiteboard. The operational question, the one this section answers, is whether the design holds when somebody is actively trying to break it.

The problem the design must solve is older than language models by several decades. In 1988 Norm Hardy published a short paper called “The Confused Deputy.” It described a compiler that had been granted permission to write to a billing file, which it needed for its own bookkeeping. A user, running the compiler, named that same billing file as the place to write the compiler’s output. The compiler did as it was asked. It wrote the output using its own authority, not the user’s, and so it overwrote a file the user could never have touched directly. Nothing was malicious about the compiler; it was simply a deputy holding more power than the person it was serving, with no way to tell whose authority a given request should run under. An AI agent is a confused deputy by construction, and the reason is structural. A conventional program keeps its instructions (the code) apart from its data (the input), but a language model receives both in the same stream of text, and nothing in that stream reliably marks which sentences are commands from its operator and which are content it happens to be reading. The UK’s National Cyber Security Centre (NCSC) said as much in late 2025, describing large language models as inherently confusable.

If the deputy cannot be relied on to know whose authority it is using, the remedy is to stop giving it authority of its own and to make the authority travel with each request. Encore’s refund agent holds no standing credentials at all: no service account, no long-lived API key, nothing that would let it act if it were simply switched on with no fan present. Instead, at the start of each conversation it performs an OAuth token exchange, the mechanism standardized as RFC 8693, in which one token is traded for another, narrower one. The token it receives names two parties, the agent and the fan it is serving, so every downstream system can see both who is acting and on whose behalf. The token carries only two scopes, orders:read and refunds:request, and it expires after 15 minutes. If the conversation runs longer, the agent must exchange again, and if the fan has gone, there is nothing to exchange.

I should be candid about how mature this is. Token exchange for agents is young practice: AWS AgentCore, Auth0 and Okta now support it, but I know of no published account of running it in production. Encore adopted it anyway because the principle beneath it, short-lived authority delegated by a named person, is old and proven, and only the plumbing is new. The credential is shaped like the task rather than like the agent: it describes one fan’s refund conversation, not everything a refund agent might conceivably need.

Touring crews use the same idea backstage. Each person wears a laminate, and the laminate determines which doors open: a lighting technician’s pass opens the dimmer room but not the artists’ dressing rooms, however persuasive its holder is and however urgent the story. Security at the door checks the pass, not the explanation. The analogy is useful precisely where it breaks. A human guard at a dressing room door can notice someone behaving strangely and decide to ask questions even when the laminate looks right. An API cannot; it sees only a token and a request. So the agent’s credential must be narrow enough that even a strange request, one nobody anticipated, cannot reach beyond it. The narrowness has to do the work that the guard’s judgment would have done.

The credential limits what the agent may touch. Three further controls limit what the agent can do with the environment around it, and they matter because the model inside cannot be trusted to refuse a bad instruction. First, the agent runs in a sandbox with no network egress except to two destinations: the internal Model Context Protocol (MCP) server, through which it calls Encore’s tools, and the AI gateway, through which it reaches the model. This mirrors the way cloud coding agents now run with internet access off by default. No file system outlives the conversation, so nothing an attacker plants in one session waits for the next. Second, budgets for steps, tokens and spend are enforced by the workflow engine, outside the prompt, so a looping or manipulated agent hits a hard wall that no clever text can talk its way past. Third, every tool call, with its arguments and the token’s subject, is written to an append-only audit log, so that after anything unusual we can reconstruct exactly what the agent attempted and for whom.

The ticket that tried to issue orders

The design met its first real attacker one Thursday in spring. A fan opened a support ticket complaining about a cancelled show. Below the complaint, formatted to look like an internal note, was a paragraph claiming to come from Encore’s finance team. It said the customer had been pre-approved for compensation and instructed the agent to refund three listed orders and to skip review, since the refunds were “already authorized.” Two of the three orders belonged to other people’s accounts. This is indirect prompt injection: an attack in which the malicious instruction does not come from the user typing to the model, but arrives inside content the system reads on the user’s behalf, such as a ticket, an email or a web page. Kai Greshake and colleagues first studied it systematically in 2023, and it is the confused deputy in its modern form, since the ticket body is data that the model may treat as a command.

The defenses meant to stop it at the door did not. The input classifier, which screens incoming text for known attack patterns, did not flag the ticket, and the model was partly persuaded: its trajectory, the recorded sequence of steps and tool calls in the session, shows refunds:request called for all three orders. Had the agent held a standing credential with broad refund rights, that would have been the end of the story. It did not. The orders API refused the two calls for other fans’ orders, and the credit for that belongs to the API rather than to the model or the classifier. The division of labor is worth stating carefully. The token says who is asking: this agent, on behalf of this fan. The API’s object-level check, which compares the token’s subject with the owner of each order requested, decides what that person may touch. Neither is sufficient alone. A perfect token is useless if the API never checks ownership, and OWASP ranks exactly that omission, a missing object-level authorization check, first among API security risks.

The third order genuinely belonged to the ticket’s author, so the ownership check passed. But Encore’s refund rule auto-approves only refunds under €80 when more than 48 h remain before showtime, and this order was €95, so the workflow routed it to a human reviewer. The reviewer saw the policy panel, computed by the workflow from the order’s price and the time to showtime, alongside the quoted paragraph from the ticket, and declined. The phrase “already authorized” had no effect on any of this, because the workflow does not read prose. Its decision rests on structured facts (the amount, the hours remaining, the rule’s verdict), and no sentence in a ticket can change those.

Detection came from plain monitoring rather than anything specific to AI. Denials issued to an agent’s identity are rare, provided the tools are designed so that a legitimate agent, serving the fan in front of it, seldom asks for something it may not have. That rarity is what makes them worth alerting on: a denial is a strong sign that the agent is being steered somewhere it should not go. Two denials in one trajectory paged the Support Experience team within minutes.

The blameless review that followed, of the kind the Operations book describes in its chapter on learning from incidents, made three changes and was honest about how much each was worth. First, ticket bodies now reach the model wrapped in delimiters that mark them as untrusted content. The review recorded this as a weak layer, and for good reason: Microsoft’s spotlighting experiments found delimiters the least effective of the marking techniques they tested, and in 2025 Milad Nasr, Nicholas Carlini and colleagues broke 12 published defenses against prompt injection with adaptive attacks, succeeding more than 90% of the time. A delimiter raises the cost of a lazy attack, but it is a request to the model, and the model is exactly the component we have already seen being persuaded. Second, the attack ticket and five variants of it joined the golden set, so every future prompt or model change is tested against them. Fixed test cases, though, say little about an attacker who adapts after seeing what fails, so the review added a human counterpart: before each major change to the prompt bundle, a security engineer spends a day attacking the agent. Third, the denial alert threshold dropped to one, since a single denial had turned out to be signal rather than noise.

What actually protects Encore is the structure the review left alone. There are three parts to it. The first is least privilege by construction: the agent can never hold more authority than one fan’s 15-minute token grants. The second is a person in front of any money that falls outside the narrow automatic rule. The third is that no session holds all three legs of Willison’s lethal trifecta, the combination of access to private data, exposure to untrusted content and a way to send information out. The refund agent reads untrusted text (the ticket) and private data (the fan’s orders), but its sandbox gives it no channel to send anything outward, so even a fully persuaded agent has no means of leaking what it has read. Meta’s “Agents Rule of Two” (2025) turns this reasoning into a rule a design review can apply: without human approval, a session may combine at most two of three properties, namely untrusted input, access to sensitive data or systems, and the power to change state or communicate externally. A design that needs all three must put a person in the loop. I have come to believe that injection must be contained by design, on the assumption that it will eventually arrive, rather than prevented by filters we hope will catch it; the NCSC says the same.

None of this makes the residual risk zero, and it is worth seeing exactly where the remaining risk lives: inside the policy itself. Suppose the attacker had asked instead for a €60 refund on their own order a week before the show. That request is under €80 with far more than 48 h to go, so the workflow would approve it automatically, injection or not, because it is a request the policy permits anyone to make. This is where the idea of blast radius becomes precise. For an agent, the blast radius is the worst outcome reachable within its credentials and its policy in a single session, multiplied by the number of sessions an attacker can open. The two factors call for different controls. Narrow sessions, with scoped tokens and the ownership check, keep the first factor small. A per-account limit on agent-initiated refunds keeps the second small, so that one attacker cannot open a hundred conversations and collect a hundred small refunds. That limit is nothing new: it is the velocity rule every payments fraud team already writes, applied to a new kind of caller.

Recap. An agent is a confused deputy by construction, because instructions and data reach it in the same stream of text. Its authority should therefore travel with the request, in short-lived tokens delegated by a named person, while the resource’s own ownership check decides what that person may touch. Sandboxing, budgets enforced outside the prompt, human approval for money and an append-only audit log contain what the model cannot be trusted to refuse. Delimiters and classifiers are weak layers worth having; the strong control is to deny every session the full lethal combination, and then to bound what remains by limiting both the harm of one session and the number of sessions.

Exercise 8.5. Choose one agent that acts on your systems and write down its credential: who issues it, which parties it names, what it can reach, how long it lives and which actions wait for a person. Check the design against the Rule of Two, noting which two of the three properties each session combines. Then estimate the worst outcome a single session could reach within policy, and how many sessions an outsider could start in an hour.

Kata: Grounded Flights

Your brief: “Skylark.” Skylark is a European regional airline that carries 4 million passengers a year on 90 aircraft, with a team of 60 engineers. It runs two AI features: a chat assistant that answers passengers’ questions, and a rebooking agent that can act, moving passengers onto other flights and issuing vouchers when a disruption strands them. Compensation under EU Regulation 261/2004 is fixed by flight distance at €250, €400 or €600, and people at Skylark, not the model, decide who receives it. The assistant, however, explains those rules to passengers every day, and Skylark’s lawyers have read Moffatt, so they know that what the assistant says can become a promise the airline must keep. Load is the second trouble. On a bad-weather day chat volume rises 30× within two hours, and last winter the token quota ran out at 07:40 on a morning like that. Change is the third: the model behind the cheap tier retires in 90 days, so a model change is coming whether or not anyone schedules it. Security is the fourth. Last month a passenger typed instructions into a booking’s “special assistance” field, the free-text box meant for requests such as a wheelchair at the gate, and nobody can yet say whether the agent read them, still less whether it obeyed.

Deliverables:

1. Release unit — the prompt bundle and its release gate, as Section 8.2 describes them: the prompts, sampling parameters, tool descriptions, retrieval settings and eval suite that ship together as one versioned unit, with the model named by a pinned snapshot ID rather than an alias. Include a rollback plan that still works after the model’s retirement date, when the snapshot you would roll back to no longer answers.
2. Eval plan — begin, as Section 8.3 does, with error analysis: invent 50 traces of the assistant at work, read them, and name the failures you find. From those failures build three evaluators. Make one code-based, a check a program can run without judgment, and one an LLM judge with an agreement target, set relative to how well two human reviewers agree with each other, since that is the ceiling any judge can reach. Finish with a plan for sampling live conversations and a quality SLI for compensation explanations, so you can answer the question Encore had to answer after its refund rule went wrong: how many people heard the wrong rule, and for how long?
3. Capacity and degradation — reserve capacity for disruption days, sized to the 30× surge rather than to an ordinary Tuesday, and write a four-rung degradation ladder in the manner of Section 8.4, from normal service down to human handoff. Give each rung its trigger, including one for quota exhaustion and one for falling quality, and say which rungs a machine pulls and which a person pulls.
4. Agent credential spec — describe the rebooking agent’s token as Section 8.5 describes Encore’s: what it names, which scopes it carries, how long it lives and how large a voucher it may issue unaided. Add the ownership check in the booking API that confines the agent to one passenger’s bookings, and the audit record written for every tool call.
5. Injection drill — turn the special-assistance attack into a game-day scenario. Write down the trajectory you expect the agent to follow, the structural control that stops the attack even if the model is persuaded, the alert that should fire and who it pages, and a plan for adaptive red-teaming, in which a person keeps varying the attack after each defense, because a fixed list of known attacks says little about an attacker who learns.

Rubric:

Where you now stand

You can now operate a component that is fluent and usually right, which is a harder thing to run than one that is merely unreliable. You can ship its prompts and models as releases behind a gate, so that a change to its behavior goes through the same pipeline as a change to code, and a change you did not make is caught by the next morning’s eval run. You can evaluate it with instruments whose own accuracy you have measured, and sample its live conversations as a form of surveillance, in the way medicine keeps watching a drug after its trial, once real patients take it. You can let it step down a degradation ladder chosen in advance, calmly, by people who were not in the middle of an outage. And you can give its agents only the authority that a bad day can afford, so that when one of them is persuaded, the structure around it still says no. Weizenbaum’s secretary trusted a simple program within a few minutes of conversation, and the programs we run today are far more persuasive than ELIZA ever was. Our task is to make trust in them slower to form and better earned. Each chapter so far has taken one part of production engineering and followed it through Encore on its own. Real work does not arrive in chapters. Chapter 9, the capstone, gathers them into a single piece of work, in which the objectives, capacity, runtime, delivery, failure design, data, security and AI practices of this book must hold together on one system at once.

References

• Shreya Shankar, Hamel Husain — Evals for AI Engineers. O’Reilly, 2026.
• Chip Huyen — AI Engineering. O’Reilly, 2025.
• Lingjiao Chen, Matei Zaharia, James Zou — How Is ChatGPT’s Behavior Changing over Time?. arXiv, 2023 — free online.
• Civil Resolution Tribunal of British Columbia — Moffatt v. Air Canada, 2024 BCCRT 149. 2024 — free online.
• Andrew McNamara — Building production-ready agentic systems: Lessons from Shopify Sidekick. Shopify Engineering, 2025 — free online.
• Anthropic — Model IDs and versioning. Claude Platform documentation, 2026 — free online.
• Kai Greshake et al. — Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv, 2023 — free online.
• Milad Nasr, Nicholas Carlini et al. — The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses against LLM Jailbreaks and Prompt Injections. arXiv, 2025 — free online.
• National Cyber Security Centre — Prompt injection is not SQL injection (it may be worse). NCSC, 2025 — free online.
• Meta AI — Agents Rule of Two: A Practical Approach to AI Agent Security. Meta, 2025 — free online.
• Norm Hardy — The Confused Deputy (or why capabilities might have been invented). ACM SIGOPS Operating Systems Review, 1988.
• Michael B. Jones et al. — RFC 8693: OAuth 2.0 Token Exchange. IETF, 2020 — free online.
• Shunyu Yao, Noah Shinn, Pedram Razavi, Karthik Narasimhan — τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv, 2024 — free online.
• D. Sculley et al. — Hidden Technical Debt in Machine Learning Systems. NeurIPS, 2015 — free online.
• Joseph Weizenbaum — Computer Power and Human Reason: From Judgment to Calculation. W. H. Freeman, 1976.

Chapter 9: Production Readiness — The Capstone

“Plans are worthless, but planning is everything.” — Dwight D. Eisenhower

On 30 October 1935, at Wright Field in Dayton, Ohio, the United States Army Air Corps was flying the entrants in a competition for a new long-range bomber. Boeing’s entry, the Model 299, was the favorite. It was larger, faster and longer-ranged than its rivals, and it was also more complicated than any bomber the Army had flown: four engines, each with its own settings, retractable landing gear, wing flaps and trim tabs, all of which had to be managed in the right order. That morning it took off with Major Ployer P. Hill at the controls, one of the Army’s most experienced test pilots, climbed steeply, stalled and crashed. Hill died of his injuries, and so did Boeing’s own test pilot, Leslie Tower, who was aboard. The investigators found nothing wrong with the aircraft. The elevator gust lock, a clamp that holds the tail’s control surfaces still while the aircraft is parked so that the wind cannot batter them, had never been released. A crew of great skill had forgotten one routine step among dozens, and that step happened to be fatal.

Boeing lost the competition, and what happened next is the part that matters to us. One conclusion lay close to hand, that the airplane was simply too much for a crew to manage. The Army’s test pilots drew a different one. They could hardly have found a pilot more skilled than Hill, so they did not ask for more skill. They wrote a checklist instead, short enough to fit on a card, setting out the steps for each phase of flight in order, to be read aloud and confirmed. The Army bought a small number of the aircraft for further testing, flew them with the cards, and the Model 299 went on to become the B-17. The card added no skill to anyone. It moved one kind of memory out of the pilot’s head and onto paper, so that the steps a skilled person forgets precisely because they are routine were read rather than recalled.

The card has a limit, and the limit is the subject of this chapter. The gust lock went onto it because a gust lock had killed two men. Every line on such a card is the residue of some earlier failure, written down so that nobody has to have it twice. A checklist is a memory of failures someone has already had. It says nothing about the failure nobody has had yet, and it says nothing either about the things its authors decided, for good reasons, need not be on it. Every readiness review in software descends from that card. Google’s launch coordination engineers work from a launch checklist, part of which the Site Reliability Engineering book prints; Amazon offers its customers a structured review before a planned peak; and Encore has the card we met in Section 2.3, which its Reliability team, the platform group’s ten site reliability engineers (SREs), works through on the Wednesday before every big on-sale.

This chapter attends one of those Wednesdays. The on-sale is the largest of Encore’s year: Vesper Hale’s 12-date European tour, opening on Friday at 10:00 CET, with about 1.1 million fans expected in the waiting room. We will follow the review through the chapters of this book in their order. Section 9.1 takes the objectives and the error budgets of Chapter 1; Section 9.2 the capacity and cost arithmetic of Chapter 2; Section 9.3 the control plane of Chapter 3 and the change discipline of Chapter 4; Section 9.4 the failure design of Chapter 5 and the data of Chapter 6; and Section 9.5 the security of Chapter 7 and the AI workloads of Chapter 8, before it follows Friday morning to the minute before the doors open. (Chapter 1 of the Operations book’s map of Encore shows where each piece sits.)

I should say now how the review ends, because this chapter is not a mystery. It passes. It is run carefully by people who have, in effect, read the chapters before this one, and nearly every line of it is answered with evidence. It is a good review, and if you work through it as an exercise you will find little to correct. And yet a careful reader, following it line by line, will notice a handful of places where it rests on a belief rather than on a measurement: a test whose setup was chosen sensibly, an exemption granted long ago for good reasons, a list drawn up before the system it describes had changed, a rule that nobody had cause to revisit. I will point to some of them as we pass and gather them at the end, without saying what Friday made of them. That story belongs to the Operations book. The purpose of this chapter is to show what engineering for reliability can establish before a peak, which is a great deal, and to show as honestly as I can the shape of what it cannot.

9.1 The Readiness Review — Objectives and the Budget in Hand

A meeting with a card

The review meets at 14:00 on Wednesday, in the largest room Encore has, with a video link for those who cannot be in it. It is chaired by the staff SRE who leads the Reliability team, and its attendance list reads like the dependency graph of an on-sale written out as people: the On-Sale team, which owns the waiting room and the admitted fan’s journey; the Inventory team, which owns Seat Inventory; Orders & Payments; Bot Screening; Security Engineering; Support Experience, which runs the support assistant; someone from the Finance platform team; and the engineers who will hold the pagers on Friday. Some teams send numbers instead of people. The Runtime team, which runs the clusters, sends its node capacity in writing, as it does for every big on-sale, because nothing it owns is a line on the card.

The meeting has a fixed shape, which Encore took from the launch reviews the Site Reliability Engineering book describes. It walks a card from top to bottom, each line is answered by the team that owns it, and each answer comes with its evidence attached the day before, so that the hour goes on questions. The card has grown since Chapter 2 showed its first five lines. This is how it stood at the end of Wednesday afternoon:

On-sale readiness — Vesper Hale, 12 dates, Fri 10:00 CET (T−2 days)
[x] Objectives: on-sale-window SLO armed; 28-day budgets above policy lines
[x] Admission: 1,500 fans/s per event; arithmetic and load test attached
[x] Load test: open model, bursty, skewed seats, cache state stated (warm)
[x] Writes per shard ≤ 80% at planned rate; failover drain < 5 min
[x] Pre-scale: curtain-up 07:30; Gate concurrency, floors, cache warm
[x] Quotas: function concurrency, LB, PSP rates ≥ 1.3 × forecast peak
[x] Freeze: purchase path from 08:00; standing exemptions unchanged
[x] Dependencies: purchase-path registers current; ladder rehearsed Thu
[x] Data: ledger quorum healthy; backups fresh; restore approvers on rota
[x] Bots: new rules in shadow mode only; false-block rate in objective
[x] AI: token quota and reserved capacity; golden set green two nights
[x] People: on-call, incident commander rotation, comms lead named

Read the card as the Army pilots would have read theirs and every line turns out to be a piece of Encore’s history. The quota line is there because a Sales Analytics backfill once drained the Gate’s share of the cloud account’s function concurrency during a rehearsal (Section 3.3). “Shadow mode only” is there because of the spring dispute between On-Sale and Bot Screening (Section 1.3). “Restore approvers on rota” is there because the first Finance ledger drill lost an hour and forty minutes just reaching its backups, partly because one of the two approvers was on leave (Section 6.5). The line about the payment service providers (PSPs) goes back furthest of all, to Encore’s first outage, when a slow PSP took the purchase path down with it. The card is Encore’s history of failure, compressed into a dozen questions, and the history it records is real and was paid for. What it cannot record is a failure Encore has not yet had.

The hour that matters, priced

The first line concerns the objectives, and the review begins with the one written for days like Friday. Chapter 1 gave Encore a family of service level objectives (SLOs): the seat map within 800 ms for 99.9% of requests, checkout within 3 s for 99.9% of attempts, tickets in the wallet within five minutes for 99.5% of orders, all over a rolling 28 days. It also gave every major on-sale an on-sale-window SLO: 99% of admitted fans finish their journey without an error page or a false block in the first 60 minutes of the sale, counted per fan rather than per request (Section 1.2).

It is worth doing the arithmetic for Friday, because it turns a percentage into something a person can picture. If the forecast holds, and the Gate admits fans at the planned rate, even the longest queue for the most popular date drains within minutes, so most of the 1.1 million fans in the waiting room will be admitted inside the hour. One percent of them is about 11,000 people. That is the window’s error budget: the number of fans whose morning Encore has agreed, in advance and on paper, it can afford to spoil. It is a strange quantity to plan around, and the strangeness is instructive. Nobody in the room wants to spoil 11,000 mornings, and nobody expects to. The number exists so that a bad minute can be weighed against something, and so that on the following Monday nobody has to argue from feeling about whether the sale went well.

Chapter 1 also showed why no burn-rate rule can protect so short a window. A total failure spends 1% of an hour in 36 seconds, and a rule that waits for an hour of evidence fires after the sale is over. So Encore protects the window with people, and the review’s job here is to make sure the people exist. On Friday the On-Sale team’s primary on-call, Rui, will watch the one-minute per-fan error ratio from the moment the Gate opens, together with the on-call engineers of the teams on the purchase path: Jonas for Seat Inventory, Noor for Bot Screening, and the Orders & Payments on-call. Behind them the rescaled page stays armed, firing when both the one-minute and the five-minute per-fan error ratios pass 10%, a burn rate at which the window’s whole budget would be gone in six minutes. The incident commander rotation has Marta, a staff engineer from Orders & Payments, on it that week, reachable within five minutes. Elif, from the Reliability team, will act as comms lead if one is needed, with a status-page draft already written.

A theater has a name for this moment. Thirty-five minutes before curtain, the stage manager calls “the half”, and from that call every member of the company must be in the building and accounted for; an actor who is not there by the half is treated as missing, whatever the reason. The review’s last line is Encore’s half: it names who will be in the building. The comparison holds well for its purpose, which is to make presence a fact. It breaks at the desk. A stage manager can see the whole stage from the prompt corner. On Friday nobody at Encore will see the whole of the system at once. Each of the people the review named will watch the dashboard of one team, and the picture of the whole will exist only in what they say to one another.

Two more numbers complete the objectives. The checkout SLO’s good outcomes include a seat held pending payment, so a morning on which both PSPs falter can still be a good morning for fans if the degradation ladder of Section 5.2 works. And Bot Screening’s false-block rate, the share of admitted sessions screened that its rules turned away wrongly, has an objective of no more than one in a thousand. Applied to a crowd of this size, even that objective allows about a thousand fans to be challenged by mistake. It is worth remembering how the false-block rate recognizes a mistake: by the challenge the blocked fan goes on to pass, which is the operational test the spring dispute produced (Section 1.3).

What the budget said

The second half of the objectives line asks about the error budgets, the 28-day allowances that the error budget policy acts on. Figure 9.1 shows them as the review saw them on Wednesday.

Figure 9.1 — The error budgets on Wednesday afternoon. Every journey sat above the policy’s 25% line, so no team was in the review tier and every team was free to ship until the freeze.

The table settles several questions at once, and it is worth seeing which. No journey had less than a quarter of its budget left, so under the policy that the heads of engineering and product had signed (Section 1.3), no team’s changes needed a second reviewer from the Reliability team, and nothing argued for widening the freeze beyond its usual scope. The last row shows that Encore’s recent major on-sales had all met their windows, the weakest by a margin of three-tenths of a point. And the false-block rate was comfortably inside its objective, which suggested that the arrangement made after the spring dispute (a challenge with every block, shadow mode for every new rule) was doing its work.

What the table cannot do is just as important. An error budget, in the language of Section 1.4, is a gauge rather than a flow: it reports how much remains after the last 28 days, and it says nothing about what the next hour will cost. The review knew this and used the numbers accordingly, as permission to proceed and never as a forecast. One member of the room did turn a remaining budget into a warning. Sixty-four percent of the shop-once-admitted budget is about 5,500 bad journeys, and the spring dispute had shown that 6,000 blocked fans could take 70% of that budget in a quarter of an hour. A bad Friday could put whichever team’s change caused it into the review tier for weeks. Everyone agreed that this was true, and that the way to avoid it was the rest of the card.

Recap. A readiness review is a checklist in the lineage of the Model 299’s card, and each of its lines records a failure the organization has already had. Its first work is to price the hour that matters, here an on-sale window whose 1% budget comes to about 11,000 fans, and to name the people who will watch it, because no burn-rate rule can. The 28-day budgets tell the review what Encore can afford to spend and whether any team must slow down; they report the past faithfully and make no claim about Friday.

Exercise 9.1. For the next predictable peak your system faces, write the event-window objective and turn its error budget into a count of people, as this section did with 11,000 fans. Then name, for each team on the critical path, the person who will watch the indicator and the page that stands behind them. Which team on the path has nobody named, and why?

9.2 The Number at the Gate — Capacity, Headroom, and Cost

A windfall to spend

The second line of the card is the one the review spent longest on, and rightly, because it changed a number that every later line depends on. The On-Sale team proposed to raise the Gate’s admission rate from 1,200 to 1,500 fans per second per event.

The reasoning deserves to be heard in full, because it was good. Chapter 2 built the 1,200 from a per-fan demand of about four ordered seat writes and 26 availability reads over a session of about three minutes, which at 1,200 fans a second loaded each event’s four shards to 80% of their rated write capacity (Figure 2.5). Since then the new seat-map renderer of Section 4.3 had finished its careful rollout and now served the stadium on-sales. It draws the availability of a whole section with one read, and it refreshes only the sections a fan is actually looking at. And because its picture of which seats are free is fresher, fewer fans reach for a seat that someone faster has just taken, so fewer of them need a second hold. The On-Sale team had measured the effect in Soundcheck, its load-test harness, and found about 3.2 writes and 20 reads per fan. Each fan now asked less of the shards, and so more fans could be admitted for the same load.

The engineer who brought the proposal framed it as a question of fairness to the back of the queue. For the most popular dates, the fans at the back of a long queue reach the seat map after the best seats have gone, and at 1,200 fans a second some of them reach it after everything has gone. Admitting them a fifth sooner gives more of them a real chance at a seat. The review’s rule for such a proposal, set down in Section 2.3, is that it must arrive with the arithmetic redone at the new rate and a load test run at the new point. It did, and Figure 9.2 shows the arithmetic.

Figure 9.2 — The admission proposal, with the arithmetic redone at the new rate as the card requires. Every row passes. Every per-fan number in it was measured on Tuesday afternoon.

Each row answers a line of the card. Writes still bind, and they bind at exactly the 80% that Chapter 2 defended: 1,500 fans a second, each making about 3.2 writes, ask for the same 4,800 writes a second that 1,200 fans making four writes did. Reads fall a little, to 75%. The hot-shard row reflects a change the Inventory team had made since Chapter 2 worried about skew: each date’s front-floor sections are now spread across all four shards, and Soundcheck measured the busiest shard at 27% of the writes. The failover line of the card can then be checked directly. A failover rejects writes on its shard for about 40 s, so the busiest shard would come back owing about 51,800 writes, which is 1,296 a second multiplied by 40. It has 204 writes a second to spare, since 1,500 minus 1,296 is 204, so it would drain that backlog in about 254 seconds, a little over four minutes. The card asks for under five. It passes, without much room. And Little’s law, applied to the last row, says that about 270,000 fans will be in seat selection or checkout at once for each date while admission runs at full rate, which the holds store and the session services had been sized to carry.

The review was, in effect, deciding how to spend a windfall. Nobody proposed to touch the 20% of headroom that Section 2.3 had argued for, the margin that absorbs skew and the backlog of a failover. What had changed was how many fans it took to fill the other 80%. Someone from the Finance platform team asked whether the saving could come off the bill instead, and the answer showed how the costs of Section 2.5 are shaped. Shards are step-fixed costs, allocated whole per event, and an event cannot run on three and a fifth shards. The saving could be spent only as speed or as margin, and the On-Sale team proposed to spend it as speed and to keep the margin where it was. The room approved, and the new rate went out that evening through the configuration rings.

Tuesday afternoon in Soundcheck

The load test behind Figure 9.2 was a good one by every standard Chapter 2 set. Soundcheck replayed recorded on-sales, scrubbed of personal data, against the production-sized copy of the purchase path. It generated load from an open model, with each simulated fan arriving on its own schedule, and it failed any run in which the generator had dropped a scheduled arrival, so coordinated omission could not creep back in. Its arrivals were bursty, its seat choices were skewed toward the front-floor sections, and its stub of the payment providers answered with a latency distribution taken from production. After holding 1,500 fans a second, the run pushed on to 1,900, past the point where goodput stopped rising, and then fell back to 1,500 to confirm that the system recovered on its own, as Section 5.3 recommends. It did. The p99 of POST /holds stayed well inside its share of the checkout budget.

The team ran two tests that afternoon. The first, at 1,200 fans a second, re-measured the per-fan numbers with the new renderer in place. The second, at 1,500, began about an hour later. The test plan stated the cache state as the card requires, in a single line: seat-map geometry cache: warm; matches on-sale conditions, since curtain-up warms the cache before the doors open. That statement was honest, and it was true in the sense intended. On the morning of an on-sale, the curtain-up workflow does warm the geometry cache, the layout of every hall that Catalog serves from a cache held in its own pods, and its cache step reports the hit ratio it achieved. A test against a cold cache would have measured a morning Encore did not intend to have.

Ships offer the oldest picture I know of a margin and the conditions it depends on. In 1876, after a long campaign by Samuel Plimsoll against overloaded and unseaworthy vessels, Parliament required British merchant ships to carry a load line, a mark on the hull showing how deep the ship may safely sit in the water. What makes the load line instructive is that it is not one line. The same ship floats higher in dense, cold salt water than in warm fresh water, and the sea is rougher in a North Atlantic winter than in a tropical summer, so the mark carries a ladder of lines, one for each kind of water and season, and a master loads to the line for the water the ship will actually meet. Encore’s 80% is a load line in just this sense: a margin that keeps the shards afloat with room for weather. The analogy breaks in an instructive place. A ship’s hull carries all its lines at once, painted where everyone can see them. Encore’s capacity figure carries one line, measured in one condition, and the condition appears only in a sentence of the test plan.

Two facts about that condition are worth holding in mind, without yet drawing anything from them. The first is that the per-fan numbers belong to the whole morning as much as to the shards. Fans ask for fewer reads and fewer second holds when their seat maps draw quickly and show the hall as it is, which is the effect the new renderer was built to produce; the numbers were measured in a hall whose maps drew quickly. The second is that the afternoon’s second run met a system its first run had already warmed, the very effect Section 2.3 warned about, though here the warmth was the intended condition. The last production-scale run against the hidden test event, months earlier, had been at 1,200 fans a second and before the renderer changed. The 1,500 point had been measured on the production-sized copy, which the card permits. A load test answers the question its setup asked, and this one had asked a fair question carefully.

Limits that belong to others, and the bill

The card’s quota line covers the limits that other people set, and here the review was brisk because the nightly quota inventory of Section 3.3 had done most of the work. The Gate runs in its own cloud account, with reserved function concurrency sized to Friday’s forecast and provisioned concurrency to be raised toward its 400 instances by curtain-up; the account’s limit stood at 1.6 times the forecast peak. The load balancers stood at more than twice their forecast load. Both PSPs had been sent the morning’s forecast, as Amazon’s Infrastructure Event Management practice suggests, and both had confirmed that their agreed request rates stood at 1.4 times Encore’s peak. The autoscaler ceilings of Section 2.4, derived from what each event’s shards can absorb, were recomputed against the new per-fan demand and came out unchanged, since the shards would see the same writes a second as before.

The bill took five minutes. Pre-scaling is paid for from the moment capacity is warm, so the Gate’s provisioned instances and the raised floors elsewhere would cost a few thousand euros for the two and a half hours between curtain-up and the doors, doing nothing a cost report would recognize as work. Encore’s variable cost per ticket, €0.12 at last measurement (Figure 2.6), put the morning’s marginal spend on the order of €24,000 if the 12 dates sold their roughly 200,000 tickets. Reserved model capacity for the support assistant, which Section 9.5 returns to, appeared as its own line. Cost anomalies at Encore notify the owning team urgently in business hours, and a Friday morning is business hours, so a runaway line would be seen the same day. Nobody proposed a saving. On a morning like this the cheapest configuration is not a goal, and Section 2.5 had already shown why headroom that looks idle on a bill is the price of a short queue paid in advance.

Recap. The review treated a change to the admission rate as Chapter 2 asked: with the arithmetic redone at the new rate, a load test run at the new point, and the test’s cache state written down. The windfall from lower per-fan demand was spent on speed while the 20% of headroom was kept, and the quotas, ceilings and bill were checked against the forecast. Every per-fan number came from a careful test on one afternoon, in one condition, and a margin is only as good as the water it was measured in.

Exercise 9.2. Take the capacity figure your system relies on for its next peak and write beside it, as a load line would, the condition in which it was measured: cache state, data volume, client versions, time of day. Then name one condition the peak itself might bring that differs from it, and estimate which per-user number would change first if it did.

9.3 Loops and Changes — The Control Plane and the Freeze

Curtain-up and the governors

The pre-scale line of the card brings the review to the control plane, and to the crowd of governors that Chapter 3 taught us to see in every runtime. Most of Friday’s preparation is itself one of those governors. At 07:30 the curtain-up workflow will propose a plan for each of the 12 events, Rui will review it, and on his word it will run. The plan raises the Gate’s provisioned concurrency toward its 400 instances, in EU-1 where the tour’s writes are homed and, since the October region game day of Section 6.5 found the gap, in EU-2 as well. It raises the floors of the autoscalers in front of Orders and Catalog through commits to Git, which is the lesson of Catalog’s fight with Argo CD in Section 3.1: intent goes to the loop that owns the field, and it leaves a record. It sets Seat Inventory’s minimum replica counts. And it warms the seat-map geometry cache for all 12 venues, reporting the hit ratio it achieves. Each step checks its result, and the workflow finishes before the freeze begins at 08:00 and posts its report to the on-sale channel.

The review checked the plan the way Section 3.1 recommends checking any set of loops, by asking who owns each field. The autoscalers own replica counts, and no manifest beside an autoscaler sets replicas, which Encore’s policy rules now enforce. Argo CD reconciles everything else from the environment repository, with self-heal on. The cluster autoscaler adds nodes when pods cannot be placed, and the Runtime team’s written note confirmed that the core clusters had node headroom for the floors curtain-up would set. Argo CD’s sync windows, which Section 3.4 introduced, were available but still not wired to the on-sale calendar; the freeze is enforced instead in the delivery pipeline, which refuses deploys to frozen services between the freeze’s start and the sale’s end, and in the on-sale channel. The review found nothing on this line to change.

The tenants’ cells made a short appearance. Three Professional tenants had small sales of their own on Friday morning, each in its own cell, and the review noted them only to confirm that nothing in Vesper Hale’s preparation touched those cells. That is what cells are for (Section 5.4): a cell’s trouble stays inside it, and so does its good fortune, which means that a tenant’s sale in cell P7 needs no line on Encore’s card at all.

Cue, Encore’s operations agent, has a place on this line too, because its permissions change with the calendar. At 08:00 the control plane withdraws its write scopes on the purchase path, so that nothing Cue proposes during the freeze can produce a working token. If an incident is declared, its mitigation actions return, each still requiring a named person’s approval in the channel. Two small actions may run without approval, granted on Cue’s record: raising Catalog’s autoscaler floor through Git, up to a cap of 60 replicas, and holding a node in the platform’s upgrade reconciler. The review confirmed that the withdrawal was scheduled and moved on.

One governor did not appear in the review at all, and it was absent for a reason the room would have found perfectly sound. The Kubernetes upgrade wave of Section 3.2, which moves each new version from the canary cell through the Professional cells to the core clusters last, had reached its final stage that week. The node-upgrade reconciler was working through the core clusters’ node pools one node at a time, as it had for every wave since it replaced the old upgrade weekends, draining each node through the disruption budgets that allow Seat Inventory to lose one pod per shard and no more. The wave had been announced in the platform’s weekly change digest, where waves are always announced. It did not come to the readiness review because the review covers what the freeze covers, and the reconciler, as low-risk platform automation, had been exempt from the freeze for more than a year (Section 4.4). As far as anyone could recall, it had never troubled an on-sale.

Touring productions know a version of this arrangement. A show that arrives at a venue brings its own crew, who run the show’s lights, sound and rigging, while the house crew runs the building: the fire curtain, the house lights, the air handling, the maintenance of the building’s own systems. The production manager advances the venue in detail, but the building’s maintenance schedule is the building’s business, and nobody on the tour thinks to ask whether the house plans to test its fire alarm during the second act, because no venue would. The comparison is exact up to that last clause, and it breaks there. A house engineer about to start an alarm test would look through the door, see an audience and wait. The reconciler knows disruption budgets and nothing of calendars, as Section 3.2 put it; it looks at nodes, and it never looks through the door.

An exemption is a decision not to look. It is made once, carefully, by people who weighed the alternatives, and pausing the reconciler before each of Encore’s 160 on-sales a year had genuinely created more work than it seemed to save. After that the exemption stops being a question, which is its purpose, and the thing exempted stops being examined, which is its cost. Nobody at Encore had decided that the reconciler should drain nodes during a major on-sale. They had decided, long before, that it did not need to be asked.

What shipped before eight

The card’s freeze line has two halves: what changes before the freeze and what may change during it. The first half is the subject of Chapter 4, and the review took it change by change, because about two-thirds of Encore’s incidents in the past year had begun within an hour of a change (Section 4.1).

Four changes reached the purchase path between the review and the freeze. The new admission rate, a single setting in the tour’s event configuration, went out on Wednesday evening through the same configuration rings as code: first to the staff ring and the synthetic buyers that run against the hidden test event, then outward, with a bake at each ring. The flag state that put seatmap-renderer-v2 in front of the 12 dates went out on Thursday morning by the same road, recorded in the deploy history like any other production change. Bot Screening’s rule bundle carried two new rules, both in shadow mode, so that they would log what they would have blocked without blocking anything; the bundle traveled the rings on Thursday, and the Gate’s loader validated its shape, its size limit and its signature, keeping the last good bundle if anything failed (Section 4.3). And Orders & Payments shipped a small patch to its PSP client library on Thursday morning. The patch touched money, so it sat in the first review tier of Section 4.5, read by two reviewers, one of them the domain owner, before it passed the full canary schedule of Figure 4.3 in about 45 minutes.

The last purchase-path deploy finished its canary at 16:10 on Thursday, which left the new code a full evening and night of ordinary traffic before the freeze. That margin was deliberate. Canaries do not run during on-sales, because a surge of 500× leaves no stable baseline to compare against and no time to bake (Section 4.4), so whatever was going to be on the purchase path at 10:00 had to have been watched before then, on traffic that could still tell a bad build from a busy minute.

Elsewhere at Encore, change continued at its usual pace. About 180 production deploys land on a working day, across 40 teams, and Friday would be a working day for every team whose service was not on the frozen list. That is as it should be. A freeze is an expensive instrument, Section 4.4 argued, and the evidence favors a narrow one, aimed at the changes most likely to do harm, over a broad one that stops everything and then releases the backlog in a flood.

The freeze as written

The second half of the freeze line is the freeze itself, and Figure 9.3 shows it as the review read it.

Figure 9.3 — The freeze for Friday, as the review read it. Each row has a sound reason behind it, and the list of frozen services was written some years before this review.

Every row of the table has a sensible argument behind it, and the people who made those arguments were not careless. Certificate rotation must run whatever the calendar says, because credentials that expire on their own are only safe if they are renewed on their own (Section 7.2). The reconciler’s exemption we have just discussed. Catalog’s absence follows from what Catalog was when the list was drawn up, the service behind search and listings, and a fan can indeed buy a seat without searching for one. The list had been amended when a new service joined the purchase path by name, and not otherwise. Since it was written, the seat map had come to draw its geometry from a cache held in Catalog’s pods, and nothing in the way the list was kept would have prompted anyone to ask whether that changed Catalog’s place.

The last row is the one Section 4.4 counted. In the twelve months before this review, the on-sale channel had granted 41 exceptions, rising quarter by quarter from 3 to 11, 13 and 14, every one of them judged on its merits by whoever was senior and awake, and none of them followed by an incident. No one had added them up. The review did not add them up either, and it had no reason to: the review concerns one sale, and the exceptions belonged to other sales, each one closed and forgotten when its sale ended. Section 4.4 also noted the subtler weakness in a freeze drawn this way. Cloudflare’s outage of November 2025 began with a change to a database that the broken proxy depended on, and a freeze written as a list of protected services, with automation exempt, would have let that change through. As Section 4.4 concluded, a freeze drawn around services leaves open whatever those services depend on.

None of this was raised in the room, and I do not think it should surprise us that it was not. A readiness review is good at examining the things in front of it. The freeze’s boundary is not in front of it; the boundary is the frame within which everything else is examined, and a frame is the hardest part of a picture to see. The review asked whether the freeze would be in force from 08:00, confirmed that the pipeline would enforce it, confirmed that the standing exemptions were unchanged, and ticked the line.

Recap. Friday’s preparation is itself a governor, curtain-up, and the review checked its plan by asking who owns each field and confirming that intent reaches the owning loop through Git. Changes to the purchase path, configuration and flags included, traveled the rings and finished their canaries a night before the freeze. The freeze is narrow, as the evidence advises, and every row of it has a sound reason; but an exemption removes a loop from examination, a list of frozen services drifts away from the dependencies it was meant to describe, and exceptions granted one at a time are never seen as a total.

Exercise 9.3. List every loop and every route that can change your production system during your next freeze, including platform automation, certificate and secret rotation, flag changes and data pushes. For each, write down which calendar it reads, if any, and when its exemption, if it has one, was last examined. Then draw the frozen list from the dependency graph of your critical path rather than from memory, and compare the two.

9.4 Failure, Rehearsed — Stability Patterns and the Ledger

The registers and the ladder

The dependencies line of the card is Chapter 5’s, and it begins with the integration-point registers of Section 5.1, the table each service keeps of its dependencies, with the timeout, retry policy and breaker that guard each call, what the service does when the dependency is slow or down, and the date on which anyone last saw that behavior happen. The review read the registers of the frozen services. Orders & Payments’ 11 rows were all dated within the quarter, led by the two PSPs, whose slow behavior had been seen at last quarter’s tech rehearsal and whose secondary path is exercised every day by the small share of ordinary payments sent to it. Seat Inventory’s three rows were current. The card asks about the registers of the purchase-path services, and those were in good order.

The degradation ladder for payments came next, and the review walked it rung by rung, as Section 5.2 built it. The breaker on the primary PSP opens at a 40% failure rate, counting as slow any call over 600 ms, a threshold set between the PSP’s normal p99 of 450 ms and the 800 ms timeout so that a slow provider trips it as surely as a failing one. When it opens, new payment attempts move to the secondary PSP, and attempts already in flight finish where they began, because idempotency keys belong to one provider. If both providers are degraded, verified fans’ seats are held as a pending payment for up to 20 minutes instead of the usual five, capped per fan, and each fan receives a link to complete payment with strong customer authentication when a provider recovers; unpaid holds lapse and their seats return to sale. On the last rung, checkout fails fast with an honest message and the Gate slows admission. No rung issues a ticket without an authorized payment. Behind the Gate, services shed background work first and checkout last, Envoy’s adaptive concurrency filter bounds what each server holds in flight, and Seat Inventory has no breaker, relying on deadlines and the retry budget so that one sick shard cannot cut off the healthy ones.

Then the review did something Chapter 5 of the Operations book would approve of, and listed the generic mitigations, the moves that reduce harm without anyone first understanding its cause. Figure 9.4 is that list.

Figure 9.4 — The generic mitigations on the review’s page, with who may pull each one and when it was last pulled. Like every list on the card, it holds the moves Encore had needed before.

The last column is the one Section 5.1 taught us to value, and almost every row of it has a date. Two rows deserve a word. The shard failover’s time comes from last quarter’s tech rehearsal, at which Jonas, who will be Seat Inventory’s on-call on Friday, performed the failover by hand for the first time and found that the runbook named a flag removed two releases earlier; the step took 11 minutes instead of three. The runbook has since been corrected, and Jonas read through it again on Monday. And the admission dial at the top of the table is the plainest move Encore has: one setting, reversible in seconds, whose cost to fans is a slightly longer wait in a queue they already expect. It is on the list, it has an owner, and it was rehearsed on Thursday.

Thursday’s tech rehearsal

Encore’s quarterly game days are called tech rehearsals, after the theater’s practice of running every cue in sequence before opening night (Section 5.5). For its largest on-sales the Reliability team also holds a short one in the days before, aimed at the moves the sale is most likely to need. Thursday’s ran at 09:00 against the hidden test event, which was given four seat-ledger shards of its own for the occasion, with synthetic buyers arriving at 1,500 a second. It was written down in advance:

experiment: vesper-hale-ladder-and-dial
hypothesis: >
  Primary PSP breaker opens within 15 s; new payments move to the secondary;
  with both PSPs slow, holds become pending payments and links are issued;
  admission falls to 900 and returns to 1,500, each within a minute of approval;
  synthetic checkout success stays at or above 97%; no seat is sold twice.
target: test event only (tenant: encore-internal), home region EU-1
injections:
  - at: T+0     psp_latency: { psp: primary, added: 2s, share: 60% }
  - at: T+8m    psp_latency: { psp: secondary, added: 2s, share: 60% }   # both slow
  - at: T+15m   gate_admission: { event: test, rate: 900 }   # IC approves in channel
  - at: T+20m   gate_admission: { event: test, rate: 1500 }
abort_if:
  - real-event checkout SLO burn rate > 2x for 5m
  - any duplicate SeatSold for the test event
duration: 30m

Marta, on the incident commander rotation that week, played the commander in the channel, and Cue was switched off, as it is for half of all drills, so that the people would practice without waiting for an agent’s suggestion. The machines did what the hypothesis predicted. The breaker opened in 8 s and new payments moved to the secondary. When the secondary slowed too, 312 holds became pending payments and their fans’ links were issued; when the latency was removed, 290 of those payments were completed with authentication and the rest lapsed on schedule, their seats returning to sale. Marta approved the drop to 900 in the channel, the dial moved within 40 s, and synthetic errors fell within two minutes; the return to 1,500 was as quick. No seat was sold twice. Synthetic checkout success held at 98.6%.

The observers’ notes, as usual, were worth more than the confirmation. The most useful note was small. When Elif drafted the status-page message for the injected slowdown, the template told fans that some were seeing errors and when the next update would come, but said nothing about the question every fan in a queue is actually asking, which is whether to stay. She added a line saying that places in the queue were safe. A second note recorded a property of the setup. Since last quarter’s rehearsal, when Bot Screening’s velocity rule had quarantined several synthetic buyers that were retrying their rejected holds, the synthetic buyers carry a marker that the rules honor, so that rehearsals are no longer interrupted by them. The rule therefore took no part in Thursday’s rehearsal, by design. The rehearsal included no shard failover; last quarter’s had covered it, and the runbook it exposed had been fixed.

There is an old saying in the theater that a bad dress rehearsal means a good opening night. It is a superstition, and like most superstitions it carries a grain of truth inside a comforting falsehood: a rehearsal that goes badly has found things while they are cheap to fix, and a show that has found them does tend to open better. The falsehood is the implied converse, that a smooth dress promises nothing in particular, or promises trouble. A smooth rehearsal promises exactly what it tested and no more. Thursday’s showed that the payment ladder and the admission dial worked, under a commander who had now used the dial once, against synthetic fans. And there the analogy breaks in the place that matters most. A dress rehearsal runs the real show with the real cast. Encore’s rehearsal ran a test event with synthetic buyers, whose behavior is written in a script: they retry when the script says to retry, they do not refresh in frustration, they do not tell their friends, and they do not behave differently when they are afraid of missing out.

State that must survive the morning

The data line of the card is Chapter 6’s, and it is the line on which Encore’s preparation is deepest. All 12 dates are homed in EU-1, as every major on-sale before them had been. Each date’s four seat-ledger shards are consensus groups of five voting members: two in EU-1 in different zones, where the leader sits; two in EU-2; and a witness in EU-3 that stores the log and votes but serves no traffic. A seat write is acknowledged once three of the five hold it, about one cross-region round trip of 18 ms, and group commit keeps each shard’s ordered writes at its rated 1,500 a second. The losses this arrangement absorbs are the ones Section 6.4 worked through. A lost node or zone moves a shard’s leader to the other replica in EU-1, with about 40 seconds of rejected writes while the new leader is elected and fenced in, a cost the drain arithmetic of Section 9.2 had already allowed for. An isolated EU-1 loses its majority and stops writing, and the members in EU-2 and EU-3 elect leaders there, with a recovery point objective (RPO) of zero, no split brain, and a recovery time objective (RTO) of minutes. The review checked the witness’s health and the replication lag of every group, and asked Jonas what a lost node would cost on Friday. His answer was the one the design gives: about 40 seconds on one shard, and nothing worse.

The October region game day had left three findings at the edges, and the review checked each. The Gate’s pre-warm now runs in EU-2 as part of curtain-up. The Orders reconciliation job is no longer pinned to EU-1. The third finding, that Bot Screening’s risk data reaches EU-2 only by a daily copy, was still open, its ticket scheduled for the following month; Bot Screening’s lead accepted the risk for Friday by name, and the review recorded the acceptance as one of its two notes. A region failover remains, by Encore’s design, automated in its steps and human in its trigger, and the review named who on Friday would have the authority to pull it.

The rest of the line concerned recovery. The backup freshness alert of Section 6.2 was quiet for every system of record. The Finance ledger, which in its first full restore drill had taken nine hours against a two-hour RTO, now restores its current financial year first, with a warm standby behind it, and its second drill had brought that slice back in 1 hour 50 minutes; the standby approver rota named someone for Friday. The holds store, migrated to a key-value store with per-key expiry in Section 6.3, carries both the five-minute holds and the pending-payment holds of up to 20 minutes.

That left the derived data, which Section 6.1 warned is where teams are most often surprised. The seat-map geometry cache needs no backup, because its contents can always be computed again from Catalog’s data. What it needs, Section 6.1 said, is a rebuild whose duration is known. The only rebuild duration anyone had measured was curtain-up’s warm-up on a quiet morning, about 12 minutes for 12 venues with no fans asking for anything. That figure was accurate, and it was the right figure for the purpose it served, which was planning the warm-up. Whether it said anything about rebuilding the cache while a crowd was asking for it, nobody had had occasion to find out.

Recap. The review checked failure design as Chapter 5 built it: current registers for the purchase-path services, a payment ladder whose rungs were rehearsed on Thursday, and a list of generic mitigations with owners and the dates they were last pulled. The rehearsal confirmed what it tested, with synthetic fans whose behavior is scripted. The data line rests on the deepest engineering Encore has, a quorum ledger with zero RPO and rehearsed restores, while the one derived store on the opening path had a rebuild time measured only on a quiet morning.

Exercise 9.4. Write Figure 9.4 for your own system: every generic mitigation, who may pull it, how long it takes and when it was last pulled. Then design a short pre-peak rehearsal for the two moves you are most likely to need, and write down, beside its hypothesis, every way in which its synthetic users will behave differently from your real ones.

9.5 Adversaries and Assistants — and the Last Hour

Screening a crowd

The bots line of the card belongs to Chapter 7, and to the one adversary in this book that studies its opponent. The people who run bots against Encore rehearse, as Section 7.1 observed. They try their tools on minor on-sales, where little is at stake and much can be learned, and bring what works to the major ones, and an on-sale of 1.1 million fans for an artist of Vesper Hale’s standing is exactly the kind of morning they prepare for. Bot Screening tunes its rules weekly for that reason, and Noor, who will be its on-call on Friday, presented the team’s preparation.

The preparation was the one the spring dispute had produced. Two new rules, written against tools seen probing smaller sales that month, were in the bundle in shadow mode, so that on Friday they would record what they would have blocked and block nothing; if they proved sound, they would enforce at a later sale. The bundle had traveled the rings on Thursday, and the Gate’s loader, which keeps the last good bundle and, at a cold start with nothing good to load, fails open on bot scoring while still metering admission, had been exercised at the last game day. Every block now offers a challenge and a way back into the sale. The false-block rate stood at 0.4 per thousand. The rules already in force were unchanged since the last major on-sale, and the card asks about new rules, so there was nothing further to review.

It is worth pausing on how the instruments were built, because the review relied on them, and rightly. The false-block rate recognizes a person by the challenge they pass after a rule has turned them away. That makes it an excellent instrument for every rule that offers a challenge, and the arrangement after the spring dispute required every new rule to offer one. Whether every rule already in the bundle offered one was a fact about the bundle’s history rather than about its recent changes, and Section 1.3 recorded that history: among the rules in force was a velocity rule, written years earlier, that watches how fast and how often a session acts, which had never caused a dispute and so had never been reviewed or brought under the new arrangement. The review’s question did not reach it, because the question was about what had changed, and the velocity rule had not changed in years.

Two other facts sat nearby, each in a different team’s picture of the system. The fan app now retries a rejected hold for the fan, with jitter and a client-side throttle, the fix that last quarter’s rehearsal produced (Section 5.5); a retry spread out by jitter is kinder to a recovering service, and it is still a retry. And Bot Screening’s rules see sessions, not the purchase path: a rule watching how fast a session acts has no way to know whether the session is acting quickly because it is a bot or because the seats it tried to hold were refused. Neither fact came up on Wednesday, because neither was on anyone’s line.

The partner side of screening took a few minutes. Resale partners call the availability endpoints during on-sales, and since the Rowmate leak of Section 7.2 they do so with 15-minute tokens bound to a client certificate, at rates the gateway limits per partner for the duration of a major sale. The detections that caught Rowmate’s key, written against what each credential normally does, were confirmed active, with their baselines intact.

Credentials on a Friday

The rest of Chapter 7 reached the review through Security Engineering, and its preparation is a good example of an exemption that had been examined. Workload identities are SPIFFE Verifiable Identity Documents (SVIDs), certificates valid for an hour and renewed at their half-life, issued by SPIRE servers in both regions, with an alert when any certificate passes 75% of its life unrenewed. Certificate rotation is exempt from the freeze, and the review asked why, as it should. The answer was the right one: credentials that expire on their own are safe only if they are renewed on their own, and freezing their renewal would turn a security property into an outage at the end of the hour. This exemption had been examined and kept. The reconciler’s had been granted and then left alone. The difference does not lie in the exemptions themselves, both of which are reasonable. It lies in whether anyone had looked at them since.

Nothing else on the security line was due to move. No rotation of the PSP credentials or the wallet signing keys fell within two days of the sale. The break-glass path, two engineers, sixty minutes and an account outside single sign-on, had been walked at the last game day. The admission webhook for the purchase path fails closed, behind a policy engine replicated across zones. And if a critical vulnerability on the Known Exploited Vulnerabilities list arrived during the freeze, the top row of Figure 7.3 would still apply: the freeze bars non-emergency changes, a compensating rule at the edge comes first, and the 48-hour clock starts regardless of the calendar. The review recorded that Security Engineering’s on-call knew the freeze’s emergency path, and ticked the line.

The assistant’s quota

The AI line is Chapter 8’s, and it is short because Chapter 8 put the assistant’s operations into machinery. On a big on-sale morning support traffic triples within the hour, a sharp ramp of exactly the shape that providers’ rate limits punish, so Support Experience had forecast the morning’s tokens per minute, confirmed the quota for each model family with headroom, and bought reserved capacity for the morning, as Encore does for its largest sales. The fallback provider’s quota had been sized for failover load, on the understanding of Section 8.4 that the second rung is degraded by design and untested at the scale of a real provider outage.

The assistant’s behavior had been fixed in place well before the freeze. Its prompt bundle names its models by pinned snapshot, and no new bundle had shipped since Tuesday, because a bundle’s canary needs at least a day and enough conversations in every category to show a regression (Section 8.2). The golden set of 200 conversations had passed in full on Tuesday and Wednesday nights against the production configuration, each case run five times; the refund rule is explained from a template filled by the workflow’s verdict, so the model never paraphrases it; and the quality indicator on refund explanations, with its 98% objective and its 150-conversation minimum, was armed. The refund agent held no standing credentials and would hold, as always, one fan’s 15-minute token at a time, under a per-account limit on agent-initiated refunds. Support had doubled its staff for the morning, because the last rung of the assistant’s ladder ends in people, and because Support, more than any dashboard, hears what fans are actually experiencing.

What the review rested on

At 15:10 the chair read the card back from top to bottom. Every line was ticked. There were two notes: Bot Screening’s acceptance of the daily risk-data copy until the following month, and the Finance standby approver’s name for Friday. The review’s verdict, recorded in the on-sale channel, was ready. That evening the new admission rate went out through the rings.

It was a good review, and it is worth being exact about why. Nearly every line was answered with evidence, gathered in advance and attached. The arithmetic was redone at the new rate and checked by people who understood it. The load test was honest about its conditions. The ladder and the dial were rehearsed. The ledger, the backups and the restores had been proven in drills that were allowed to fail. Each team spoke for what it owned, and what each team said was true. If the purpose of this book has been to make a review like this one possible, then this book has done its job.

Now read the same afternoon a second way. Instead of listing what the review checked, list the beliefs its checks rested on: the things each answer took for granted in order to be an answer at all. Figure 9.5 is that list, as I would draw it from this chapter.

Figure 9.5 — Six beliefs beneath the review’s ticks. Each was reasonable and each had evidence; the last column is where the evidence stopped.

None of these is a mistake in the ordinary sense, and it would be a misreading of the table to treat its last column as a list of things somebody should have done. Each row describes a judgment that competent people made for good reasons, in a context that made it sensible, and each was confirmed, sometimes many times, by mornings on which nothing went wrong. That is the shape of what the Operations book will call drift, and it will give the idea the full treatment it deserves. Here I want to draw a narrower lesson, about what engineering can and cannot settle before a peak.

Everything in this book is a way of turning a belief about a system into evidence. An SLO turns “fans are happy” into a ratio. A capacity model turns “the shards can take it” into arithmetic, and a load test turns the arithmetic into a measurement. A rehearsal turns “the breaker will open” into an observation, a restore drill turns an RTO into a stopwatch reading, and an eval turns “the assistant explains the rule correctly” into a pass rate. A readiness review gathers that evidence in one room. What it cannot do is gather evidence about the beliefs nobody has noticed holding, and those are the beliefs embedded in the frame of the review itself: in what the card asks, in what the freeze lists, in what an exemption excludes and in what a team assumes about another team’s service. Forty teams each see their own part of Encore deeply, which is how a system this size can be built at all, and the price of that depth is that each team’s picture of the others is a sketch. The review is the place where the sketches meet. It is a better place than most, and the sketches are still sketches.

And the system does not hold still while it is being reviewed. Between Wednesday afternoon and Friday morning, Encore would ship hundreds of changes outside the frozen list, its loops would act on their own schedules, its adversaries would adjust their tools, and its fans would behave as frightened, hopeful people behave, which no script reproduces. Readiness is a statement about what we have examined. The system is under no obligation to stay within it.

The decision that a review produces has an older and far graver precedent, which I mention only for its shape, since the stakes bear no comparison. In the early hours of 5 June 1944, Eisenhower, who had already postponed the Normandy landings by a day because of a storm, heard his chief meteorologist, James Stagg, forecast a short break in the weather, and gave the order to go on the 6th. He had the best forecast anyone could give him, and he still could not know how the morning would go. He also wrote a short statement, to be released if the landings failed, that took the responsibility on himself. A go decision of that kind is a commitment to operate whatever the morning brings, made by people who have prepared as well as they know how and who understand that preparing is not the same as knowing. This is what Eisenhower meant, I think, in the line that opens this chapter. The plan for Friday would meet a morning it did not describe. What the planning had left behind were people who knew the system, a dial they had turned the day before, a ledger that would not sell a seat twice whatever happened above it, a comms lead who knew to tell fans that their place was safe, and an engineer who had failed over a shard by hand.

Ten to ten

At 07:30 on Friday, Rui opened curtain-up’s plan for the 12 events. He had been woken twice in the small hours that week by pre-warm alerts, the kind that fire around the preparations for a scheduled sale, and he read the plan through twice before he pressed go. By 07:55 the report was in the on-sale channel: the Gate’s instances provisioned and confirmed in both regions; the floors set through Git; the geometry cache warm for all 12 venues, with a hit ratio of 99%. At 08:00 the freeze began. The pipeline began refusing deploys to the four frozen services, and the control plane withdrew Cue’s write scopes on the purchase path.

Through the next two hours the waiting room filled. Fans arrived from every country on the tour, each given a place and a page that cost almost nothing to serve. The synthetic buyers walked the whole purchase path against the hidden test event once a minute, and every one of them bought its ticket. Bot Screening’s dashboards showed the expected probing by tools testing the edges of the Gate, and the two shadow rules began recording what they would have blocked. Support’s doubled queue was quiet. Marta, on the incident commander rotation, kept her laptop open at her desk in Orders & Payments; Elif had the status-page draft, with its new line about places in the queue, ready to publish if anyone asked her to.

Elsewhere, Encore’s Friday went on as Fridays do. Deploys continued to every service that was not on the frozen list, each through its own pipeline. In the core cluster in EU-1, the upgrade reconciler finished with one node, waited for its pods to settle elsewhere, and moved on to the next in its pool, as it had all week. In Bot Screening’s rule bundle, which had last changed on Thursday, the velocity rule sat where it had sat for years.

At 09:59 the waiting room held about 1.1 million people. The admission rate for every date read 1,500 fans per second. Jonas had the shard dashboards open, and on Rui’s screen the per-fan error ratio read zero, as it always does in the last minute before anyone has been let in.

Recap. The last lines of the card covered the adversary and the assistant: new bot rules in shadow mode, credentials whose exemption from the freeze had been examined and kept, a token quota with reserved capacity, and evals green against the production configuration. The review passed because nearly every answer was evidence. Beneath its ticks lay beliefs that the evidence did not reach, each reasonable and each confirmed by ordinary mornings, and the system kept moving after the review ended. Engineering turns beliefs into evidence; what remains after it has done so is the work of operating.

Exercise 9.5. Take the readiness review or launch checklist your organization last used and build Figure 9.5 from it: for each answer, write the belief it rested on, why that belief was reasonable, the evidence behind it and what the evidence did not reach. Choose the row whose last column worries you most, and decide who in your organization would be the first to notice, on the day, if that belief turned out false.

Kata: Midnight Doors

Your brief: “Tessera.” Tessera is a marketplace that sells to 38 million customers in 14 EU countries on behalf of 120,000 independent sellers. It is built and run by 1,100 engineers in 110 teams across three cloud regions. Once a year it holds Tessera Days, a sale that opens at midnight CET and runs for 36 hours, and its first 15 minutes are the hardest of the year: traffic rises 60× within ten minutes of midnight, and 5,000 “doorbuster” lots of limited stock, mostly game consoles, phones and limited-edition trainers, sell out in under a minute each. Every doorbuster unit must be sold exactly once, and reseller bots arrive before the customers do. Checkout depends on two payment service providers and a buy-now-pay-later provider, each with an agreed rate. A shopping assistant answers customers’ questions, and a pricing agent adjusts sellers’ prices within limits each seller sets.

Last year’s Tessera Days met its own event objective at 97.1% against 99%. Three things went wrong in the first half hour. A third-party recommendation widget’s configuration, pushed by its vendor at 00:03, doubled the weight of the product pages. The buy-now-pay-later provider rate-limited Tessera at 00:06, because nobody had sent it a forecast. And the pricing agent, misreading a supplier feed, cut the price of 300 lots to a tenth of their value before a person noticed at 00:40. Tessera runs a code freeze from 22:00, which recorded 63 exceptions over the last year’s events, all granted in chat; its platform automation is exempt as low-risk. Its readiness review is a spreadsheet of 140 lines that every team marks green the week before.

Tessera’s brief is Encore’s Wednesday at three times the size, and the comparison is the way in. Its doorbusters are seat-like: a scarce thing sold under a crowd’s pressure that must never be sold twice. Its widget configuration traveled the road that configuration and data travel in Section 4.3, outside the rings that its code obeys. Its pricing agent is an agent with permissions, whose blast radius is the subject of Section 8.5. And its spreadsheet of 140 green lines is a card that has grown long enough to be read without being thought about. The deliverables ask you to prepare Tessera Days the way this chapter prepared Vesper Hale, and then to do the one thing Encore’s review did not: write down what your preparation rests on.

Deliverables:

1. Event objectives and budgets — Write the event-window objective for the first hour after midnight, counted per customer, and turn its budget into a number of people. State the 28-day budgets of the journeys it touches, what Tessera’s error budget policy allows each team to ship in the week before, and who will watch which indicator at 00:00, with the page that stands behind them.
2. Capacity, test and cost model — Choose the unit of capacity that binds for doorbusters, derive the admission or rate limit that keeps it at a stated margin, and design the load test that measures the per-customer demand behind your arithmetic. State the test’s conditions, including cache states and third-party page weight, and confirm the result at production scale. Send forecasts to every provider with an agreed rate, and price the night.
3. Change and control-plane census — List every route and every loop that can change production between 22:00 and the end of the first day: code, configuration, vendor pushes, flags, data feeds, platform automation and agents. For each, say which calendar it reads, whether the freeze covers it, and when any exemption was last examined. Draw the frozen scope from the dependency graph of the doorbuster path, and replace the chat with a register of exceptions that shows its running total.
4. Failure and recovery plan — Write the integration-point registers for the doorbuster path with the date each failure behavior was last observed, the degradation ladder for the three payment routes, and the list of generic mitigations with who may pull each one. Design a rehearsal for the night before, and state how its synthetic customers differ from real ones. Show what keeps a doorbuster unit from being sold twice through a failover.
5. Adversary and agent plan — Review every bot rule in force, old ones included, against your challenge and appeal policy, and say what each rule can see of the checkout path’s errors. Bound the pricing agent: its credential, the largest change one session can make, the number of sessions it can run in an hour, and the person who approves anything outside those limits. Size the shopping assistant’s quota and reserved capacity for the ramp after midnight, and state its evals’ state on the night before.
6. The beliefs beneath the green — Build Figure 9.5 for Tessera from your own deliverables: each belief your plan rests on, why it is reasonable, its evidence and what the evidence does not reach. Then write the go decision as it would be recorded, with the names of the people who will operate the first hour and what each of them will watch.

Rubric:

Where you now stand

You have come to the end of this book, and it is worth taking stock of what you can now do. You can treat reliability as a product decision, set objectives that people would recognize as describing their own experience, and turn an error budget into an agreement teams will honor. You can reason about capacity in the unit that binds, test it honestly and read its cost beside it. You can see a runtime as a crowd of governors, give each field one owner and keep loops from fighting. You can ship change in small steps through rings and canaries, configuration and data included, and scope a freeze by what it protects. You can design for failure with budgets, breakers, shedding and bulkheads, and rehearse it until the rehearsal teaches you something. You can keep state recoverable and prove it with a stopwatch, operate security as routine work, and run models and agents whose words and actions you have bounded. And you can gather all of it into a readiness review that answers with evidence. That is the engineering of a resilient system, and Encore’s Wednesday shows how much of a morning it can secure in advance. It also shows what it cannot secure: the beliefs beneath the evidence, the changes that arrive after the review, and the hour in which a crowd meets the system for real. That hour belongs to the people who notice, coordinate, decide and learn while the system is running, and their work is the subject of the Operations book, Operating Production Systems — When Production Fails. Its fifth chapter opens Encore’s Gate at 10:00.

References

• Atul Gawande — The Checklist Manifesto: How to Get Things Right. Metropolitan Books, 2009.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering, ch. 27, “Reliable Product Launches at Scale”. O’Reilly/Google, 2016 — free online.
• Betsy Beyer, Niall Richard Murphy, David K. Rensin, Kent Kawahara, Stephen Thorne (eds.) — The Site Reliability Workbook. O’Reilly/Google, 2018 — free online.
• Alex Hidalgo — Implementing Service Level Objectives. O’Reilly, 2020.
• Arun Kejariwal, John Allspaw — The Art of Capacity Planning, 2nd ed.. O’Reilly, 2017.
• Amazon Web Services — AWS Infrastructure Event Management. AWS — free online.
• J.R. Storment, Mike Fuller — Cloud FinOps, 2nd ed.. O’Reilly, 2023.
• Argo CD — Sync Windows. Documentation — free online.
• Jez Humble, David Farley — Continuous Delivery. Addison-Wesley, 2010.
• Michael T. Nygard — Release It!, 2nd ed.. Pragmatic Bookshelf, 2018.
• Martin Kleppmann, Chris Riccomini — Designing Data-Intensive Applications, 2nd ed.. O’Reilly, 2026.
• Heather Adkins, Betsy Beyer, Paul Blankinship, Piotr Lewandowski, Ana Oprea, Adam Stubblefield — Building Secure and Reliable Systems. O’Reilly/Google, 2020 — free online.
• Chip Huyen — AI Engineering. O’Reilly, 2025.
• Shreya Shankar, Hamel Husain — Evals for AI Engineers. O’Reilly, 2026.

Appendix: Encore at a Glance

Throughout this book, Encore has been introduced the way a touring crew learns a new venue: one part at a time, as each part becomes relevant. Seat Inventory appears when a chapter needs a core domain to protect, the Sale Gate when a chapter needs a door to control, and the payment providers when a chapter needs something outside our walls to fail. That arrangement serves a reader who goes from front to back, because every part arrives with its reason for existing. It serves much less well the reader who opens the book at Chapter 5, or who returns months later to look up one practice and finds that the example assumes things they never read or have since forgotten.

This appendix is for that reader. It gathers the whole company onto one page, so that a single glance can answer the question “what was that again?” Each row names one part of Encore, says in a sentence or two what it is, and in nearly every case points in two directions. The first pointer leads to the Architecture book, where the part was designed and where the reasoning behind its shape is laid out. The second leads to the chapter of this book that shows how the part is engineered to withstand production, and, where the part’s story is one of incidents and the people who respond to them, to the Operations book as well. The books divide the labor deliberately: the Architecture book explains why Encore looks the way it does, this one explains what it takes, on-sale after on-sale, to keep it working that way, and the Operations book follows what happens on the days it does not.

A glossary like this is a small exercise in the abstraction the rest of the book keeps recommending. Every row compresses a chapter’s worth of reasoning into a line, and the compression is useful only because the full explanation sits one link away. Read the table as a map of the territory, then follow a pointer when you need to walk the ground itself.

Figure A.1 — Encore at a glance. The Architecture book explains why each part was designed as it was; the named chapter shows what it takes to run.