Building Resilient Systems in the AI & Cloud Era

← the whole book on one page

Chapter 4: Shipping Change Safely — Continuous and Progressive Delivery

Every outage begins as somebody’s reasonable change.

A system that nobody touches keeps doing whatever it did yesterday, and for a while that looks like stability. But the world the system serves keeps moving. Customers want new things, the libraries it depends on announce vulnerabilities, partners change their interfaces, and the business itself changes its mind. A system frozen against all of this does not stay correct; it slowly becomes wrong about the world, one unmet need at a time. So we change it, many times a day, and every one of those changes is a moment when the running system becomes one we have never run before. That is where most of our trouble begins. The question of this chapter is how a team can change a running system often, quickly and safely, when those three goals seem to pull against one another: speed appears to cost safety, and caution appears to cost speed.

The answer the industry has arrived at looks, at first sight, like a paradox. We make change safer by making each change smaller and changes more frequent, so that every step is easy to understand and easy to undo. And we separate two acts we used to treat as one: putting code into production, and letting anyone use it. Code can then sit in production, switched off, until we choose to expose it a little at a time. Both ideas take the rest of the chapter to unfold. To see why they matter, consider first what a single change can do when neither is in place. On the morning of 1 August 2012, Knight Capital, one of the largest market makers in American equities, put new code into service on its order router. Forty-five minutes after the market opened, it had lost about $440 million, and within a year it no longer existed as an independent company. Section 4.1 follows that morning closely, because nearly every safeguard this chapter describes is one Knight did not have.

Encore stands at the other end of that spectrum. It ships about 180 production deployments on a typical working day, across 40 teams, and every one of them travels through a single golden-path pipeline, the paved route the platform group maintains so that no team has to build its own. At the end of that pipeline sits a canary: the new build first serves a small share of real traffic, and it is promoted to everyone only if Encore’s service level objectives (SLOs), the reliability targets the business has agreed to, show that it is behaving. The arrangement resembles a touring production that tries each new scene in front of a few rows before the whole house sees it, although the analogy breaks in an instructive place: at Encore the few rows are chosen by software and the verdict is read from numbers rather than from the audience’s faces, which is both what makes it fast and what limits it to the problems those numbers can see. This chapter takes that machinery apart to show how each piece earns its place. It then follows change down the routes that bypass the pipeline, such as configuration edits, data changes and flags flipped by hand, which can break production just as surely as code can. And it ends with what shifts when a growing share of those 180 daily deployments was written by a machine.

4.1 Change as the Leading Cause of Outages

Forty-five minutes at Knight Capital

To understand what went wrong at Knight, we need to know a little about the machine at its center. Knight’s order router, called SMARS, took large “parent” orders from customers and split each one into many smaller “child” orders, which it sent to the market until the parent was filled. In the summer of 2012 the New York Stock Exchange was preparing to launch its Retail Liquidity Program (RLP), which began trading on 1 August, and Knight wrote new code so that SMARS could take part in it. To mark the orders that belonged to the new program, the developers reused a flag in the order messages. That flag already had a history. It had once activated a function called Power Peg, which had been unused since 2003 but had never been removed from the code. So the same flag now meant one thing to the new code and something quite different to any server still running the old code.

A second, older change had made Power Peg dangerous. In 2005 Knight had moved the logic that counted how much of a parent order had been filled. Power Peg relied on that count to know when to stop sending child orders, and after the move it no longer received it. Nobody noticed, because nobody ran Power Peg. The function sat in the code for years like a machine with its brakes disconnected, harmless only because nobody had started it.

From 27 July a technician copied the new code to the eight SMARS servers by hand, a few on each successive day. The new code reached seven of them; the eighth kept the old code. In one sense the rollout was careful, since it was staged over several days. But nothing verified it. No second person checked the technician’s work, and no automated process compared the eight servers to confirm that they all ran the same thing. When the market opened at 09:30 Eastern time on 1 August, seven servers received RLP orders, saw the flag and handled them as intended. The eighth received the same orders, saw the same flag, read it in its old meaning, and woke Power Peg. Power Peg began sending child orders, and because it could no longer tell when a parent order was filled, it never stopped. There had been a warning. Before the open, Knight’s systems sent 97 automated emails that mentioned Power Peg, but those emails were not designed as alerts, and nobody read them as one.

The response made things worse, and it is worth slowing down to see why. The engineers, suspecting the new release, did what most of us would do: they removed the new code from the seven healthy servers. This was a rollback in every ordinary sense of the word. Yet the flag in the incoming orders still carried its new meaning, because the orders themselves had not changed. Now all eight servers ran the old code against the repurposed flag, and all eight woke Power Peg. The rollback turned one misbehaving server into eight. By the time the system was stopped, about 45 minutes after the open, SMARS had made 4 million executions in 154 stocks, for more than 397 million shares. Knight reported a pre-tax loss of about $440 million; the Securities and Exchange Commission (SEC), in its 2013 order, put the loss above $460 million and fined the firm. Knight needed emergency financing within days and was merged into Getco the next year. Figure 4.1 follows the eight servers through the morning.

Figure 4.1 — Knight Capital’s eight SMARS servers on 1 August 2012. One unverified server woke Power Peg at the open, and the rollback of the seven healthy servers gave all eight the same old code.

It is tempting to name a single culprit: the technician who missed a server, perhaps, or the engineers who rolled back. Read with the eyes of safety science, which the Operations book turns on software in its Chapter 2, Knight’s morning has no single cause. It had at least six contributing ones: dead code kept for nine years; a flag given a second meaning; a deployment that nobody verified; a refactoring that quietly broke a path nobody used; warnings that nobody read; and a rollback that amplified the failure instead of containing it. Each of these was locally understandable. Leaving old code in place saves effort, reusing a field avoids a change to the message format, and rolling back is the textbook first move. None of them alone would have cost Knight anything. Together they built a mechanism, piece by piece over years, that needed only one more event to set it off, and the deployment supplied that event.

There is one more lesson here, and it bears directly on the rest of this chapter. Knight’s rollout was staged, and staging alone saved nothing. A staged rollout protects us only when something compares the servers the change has reached with the servers it has not, and asks whether they agree. Without that comparison, staging merely spreads a mistake over several days instead of one.

Why change dominates

Knight’s cost was extreme, but the shape of its failure is common: a system that had run well was changed, and the change exposed something nobody had observed before. The authors of Google’s Site Reliability Engineering (2016) reported that roughly 70% of their outages were due to changes in a live system. From that observation they prescribed three remedies, and those remedies organize this chapter: roll out progressively, detect problems quickly, and roll back safely. Other datasets give smaller shares than 70%. Much of the difference depends on what counts as a “change.” If a team counts only code deployments, the share falls; if it also counts configuration edits, flag flips and data pushes, the share rises. That question of definition will prove important later in this chapter, because some of the most damaging changes of recent years were configuration and data, not code.

Encore asked the question of itself. When its Reliability team reviewed the last year of incidents, about two thirds had begun within an hour of a change. The changes came in several kinds: a deployment, a configuration edit, a flag flipped, or platform automation simply doing its job, such as rotating a certificate or upgrading a node.

The team could answer the question only because Encore keeps the evidence. Its deploy history records every change, its owner and the services it touched, and it shows those changes beside the first alert of any incident, so that an engineer looking at an alert can see at a glance what changed nearby and shortly before. This join between the record of changes and the record of incidents sounds like bookkeeping, but it is the foundation of any honest conversation about stability. The DevOps Research and Assessment (DORA) program measures stability with two numbers: change failure rate, the share of deployments that cause a failure in production, and the time it takes to recover from a failed deployment. Neither can be computed without knowing which change preceded which incident. A team without that join is left to argue about its stability from memory, and memory is kind to the changes we were proud of.

If change causes most outages, the tempting conclusion is to change less. That conclusion fails for two reasons. The first is that the world does not wait. Security patches arrive, dependencies need upgrading and the business needs new features, so a team that changes its system rarely does not make fewer changes; it gathers the same changes into larger batches. A large batch is harder to test, harder to reason about and harder to undo, and when it fails it is harder to tell which of its many parts was to blame. The second reason is that a team that deploys rarely deploys badly. Deployment is a skill, and like any skill it fades without practice. A procedure performed once a quarter is half forgotten each time it is needed, and its scripts and runbooks drift out of date in between.

I have come to believe that this is one of the clearest places in operations to apply Brooks’s distinction between essential and accidental complexity. The essential complexity is that the system must change, often and indefinitely, because the world it serves keeps changing. The accidental complexity is whatever makes each change bigger, rarer or harder to reverse than it needs to be. We cannot remove the first, but most of this chapter is about removing the second.

Recap. Most outages begin with something we did, because a change is the moment when the system becomes one we have not yet observed. Knight shows that a deployment is often only the last step in a mechanism assembled over years, and that a staged rollout protects us only when something compares the servers it reached with those it did not. Changing less does not help, because it only makes each change larger and each deployment less practiced, and only a change log joined to the incident record lets a team see any of this clearly.

Exercise 4.1. For your last ten incidents, note the most recent change before each one, and classify it as code, configuration, data, flag, automation or none. What share of the incidents followed a change? Which kind of change most often bypassed your deployment pipeline? And how long did it take, in each incident, to find the change that mattered?

4.2 The Deployment Pipeline and Continuous Delivery

A sequence of increasingly expensive questions

Section 4.1 left us with an uncomfortable pair of facts: most outages begin with a change, and changing less only makes each change larger and more dangerous. The way out of that bind was worked out over about two decades, and it helps to see the ideas arrive in order. The phrase continuous integration (CI) was coined by Grady Booch in Object-Oriented Design with Applications (1991), where it described a steady rhythm of integrated releases rather than one great integration at the end of a project. Kent Beck’s extreme programming (1999) gave the phrase the cadence we mean by it today: many merges a day into a single mainline, so that no one’s work drifts far from everyone else’s. In 2009 John Allspaw and Paul Hammond of Flickr described deploying to production more than ten times a day, which at the time sounded to many engineers like recklessness. A year later Jez Humble and David Farley’s Continuous Delivery (2010) gave the practice the structure that made such a pace safe, and that structure is the subject of this section.

The deployment pipeline is the automated path every change takes from commit to production: a sequence of stages, each asking a more expensive question than the last, in which only a change that answers every question may proceed. The ordering is the point. Cheap, fast questions come first, so that most bad changes are turned away within minutes and at little cost; slow questions that need real infrastructure or real traffic come later, and are asked only of changes that have already earned them. A pipeline of this kind keeps a few strict rules. The first is to build the artifact once and promote that same artifact through every stage, so that what was tested is exactly what ships; rebuilding at each stage would mean the thing in production had never really been tested at all. The second is to keep everything that defines the system in version control, so that every difference between yesterday and today is a recorded change that someone can read. The third is that when the pipeline is red, the team fixes the pipeline first. That rule sounds fussy until we think about what happens without it. A pipeline that fails often for reasons nobody investigates teaches people to rerun it, then to work around it, and before long a red result carries no information. A pipeline that people have learned to ignore is no longer a control.

Figure 4.2 shows the stages of Encore’s golden path, the standard route that every service built on the platform gets by default. Read each row as a question, and pay as much attention to the last column as to the second: every stage is blind to something, and the pipeline is designed so that a later stage covers what an earlier one cannot see.

Figure 4.2 — Encore’s golden-path pipeline. Each stage is cheaper to fail than the next, and the right-hand column says which later stage must catch what this one cannot.

It is worth walking down the table once, because each stage earns its place. The commit stage compiles the service and runs the unit tests that need nothing beyond the code itself; it answers in six minutes, and because it is so cheap, it is where we want most mistakes to die. The acceptance stage checks the service against its contracts with the services it talks to and against the behavior its specification promises, which takes longer because other services, or faithful stand-ins for them, must be present. The fitness stage runs fitness functions, automated checks that the system still keeps a promised architectural, security or cost property (Section 4.5 returns to them); they catch things such as a dependency that crosses a forbidden boundary, a container image with a known vulnerability, or a resource request that would quietly double a service’s bill. The staging soak then starts the new version in a production-like cell, one of the self-contained copies of the stack from Chapter 3, runs its database migrations and lets it serve for ten minutes, which catches the change that is correct in every test yet fails to boot or to migrate. None of these stages can reproduce the 500× surge of a real on-sale, and so the canary stage sends a small share of real traffic to the new version and compares its service level indicators (SLIs), the measured signals of service health from Chapter 1 such as error rate and latency, with those of the version already running. Only after all of that does the last row come into play, and it is a different kind of question altogether. Whether fans should see a new seat-map renderer this Friday is a decision for the product and the business, and Section 4.3 shows how flags let Encore make it separately from the deployment.

The same staircase shape, cheap checks first and expensive ones last, appears in any well-run backstage. A touring crew checks every cable and channel at line check in the empty afternoon hall, then runs a full soundcheck with the band, and only then opens the doors; nobody would discover a dead microphone in front of 20,000 people if a ten-second test at three o’clock could have found it. The analogy fails at one point, and the failure is telling. A soundcheck happens once per show, while a pipeline runs for every change, dozens of times a day, and that frequency is what makes its rules matter so much: a check that is a little flaky or a little slow is a nuisance once a day and a corrosive habit at fifty runs a day.

Humble and Farley drew a distinction between two practices that are easy to confuse. In continuous delivery, every change that passes the pipeline could be deployed to production at the push of a button; in continuous deployment, every such change is deployed, with no human step in between. The second asks more of a team. It needs tests fast enough that nobody is tempted to skip them, observability good enough to notice a bad change on its own, trunk-based development so that changes stay small, and the ability to roll back in minutes. Encore deploys continuously, with one deliberate exception. Seat Inventory, the event-sourced ledger at the heart of every purchase, keeps a human release step, because a change to it must have its projections replayed against production history before anyone trusts it. In an event-sourced service the projections are the read models rebuilt from the stored events, and a change that alters how they are computed can be correct on every test fixture yet wrong against years of real events. For the one service where every seat must be sold exactly once, Encore judged that a person should look at the result of that replay before the change goes live.

Small batches, and what replaces the change board

The pipeline’s deepest effect is not on speed but on batch size, and batch size is where safety is really decided. Consider the two ends of the range. A 40-line change can be read in one sitting, its effects can be watched on a dashboard in the minutes after it ships, and if something goes wrong it can be undone without disturbing anything else. Three weeks of work by 15 people, released together, can be none of those things: no one person understands all of it, a fault in it could come from any of a hundred places, and undoing it means undoing everyone’s work at once. This is the lesson of Section 4.1 seen from the other side. Changing rarely makes batches large, and a pipeline that makes changing cheap is what lets batches stay small.

The research supports this more strongly than intuition might suggest. Accelerate (2018) found that throughput and stability do not trade off against each other: the teams that deployed most often were also the ones that failed least often and recovered fastest. The same work found that requiring approval from an external change advisory board, the committee that reviews changes it did not write, went with slower delivery and no better change failure rate. My own reading of that finding is that such a board mostly produces reassurance. A weekly committee looking at a summary cannot know a diff as well as the colleague who reviewed it line by line that morning, and so its approval adds delay without adding much knowledge. I would grant that the board’s real purpose, a sense that someone responsible has looked, is legitimate, and that the mistake lies only in where the looking happens.

None of this means changes should go unexamined. What DORA calls streamlined change approval keeps the examination and moves it into the pipeline, where the people and the evidence are. A peer who knows the code reviews every change. Every change passes the same automated checks, so that no change is trusted because of who wrote it. Riskier changes, such as those to Seat Inventory or to payment handling, get more reviewers and more stages. And the pipeline records who approved what and which checks passed, so that an auditor reads evidence instead of meeting minutes. Encore’s golden path writes that record for every deployment, which is also what gave the Reliability team the deploy history it used in Section 4.1 to join changes to incidents.

There is one limit to all of this, and Knight Capital has already shown it to us. The pipeline guards only the changes that go through it. Knight’s deployment was copied by hand to eight servers, outside any pipeline, and the flag that woke Power Peg was not code at all but data carried in a message, so even a perfect code pipeline would never have examined it. Encore has the same kind of side doors. Its service configuration lives in the GitOps repositories of Chapter 3 and so passes through review and reconciliation, but feature flags, Bot Screening rule sets and platform automation such as the node-upgrade reconciler each had their own paths into production. Section 4.3 shows three large outages that travelled by exactly such paths.

Recap. The deployment pipeline asks increasingly expensive questions of a single artifact, so that most bad changes fail early and cheaply and each stage covers what the earlier ones cannot see. Its real gift is small batches, which buy stability as well as speed. Peer review and automated checks inside the pipeline do the change board’s job better than the board did, because they look at the change itself, and every route that bypasses the pipeline deserves to be listed and narrowed.

Exercise 4.2. List every route by which a change reaches production in a system you know: pipelines, consoles, flag tools, configuration services, generated data, automation and scripts. For each route, mark whether a change that takes it is reviewed, tested, recorded, staged and reversible. The routes with the most gaps are your next pipeline work.

4.3 Progressive Delivery — Feature Flags, Canaries, and Rollbacks

Deploying is not releasing

Section 4.2 ended with the pipeline delivering a tested artifact to the door of production. What happens at that door is the subject of this section, and it begins with a distinction that sounds pedantic until we have been burned without it. To deploy a change is to put its code onto production machines; to release it is to let people experience its behavior. For most of the industry’s history the two happened in the same instant, so every deployment was also a bet placed on every user at once. Knight’s morning in Section 4.1 was such a bet: the new code went live on all eight servers at the moment the market opened, and there was no smaller stake to place first.

Progressive delivery is the practice of putting new code into production without exposing it, then widening its exposure in deliberate steps, each step conditional on evidence that the last one did no harm. The analyst James Governor named it. Its techniques are many, and we will meet canaries, rings, blue/green switches and feature flags in turn, but its one steady idea is simple. Every change has a blast radius, the set of people and systems that would be hurt if the change turned out to be wrong, and we should choose that radius deliberately instead of letting it default to everyone. A stage manager does something similar with a new lighting cue: it is tried in an empty house, then at a dress rehearsal before a few friends, and only then on opening night. The analogy holds as far as the ordering of audiences. It breaks because a theatre’s rehearsals end, while in production every change is rehearsed afresh, many times a day, on an audience that is already in its seats.

Canaries and automated analysis

The oldest of these techniques borrows its name from the mines. British miners carried canaries underground for most of a century, because the bird showed the effects of carbon monoxide before the men did, and a bird in distress gave them time to get out; the birds were phased out from 1986. A canary release follows the same logic in software. It sends a small share of real traffic to the new version and compares that version’s health with the old version’s before any more traffic follows. If the new version is sick, only the small share of users routed to it notice, and only briefly.

The analogy is worth taking seriously, because the place where it breaks teaches us something. The miners’ canary worked because it breathed the same air as the miners, only more sensitively. A software canary is useful only to the extent that its traffic resembles the traffic it is protecting. A canary that weakly resembles the population it guards, because it serves fewer users, runs during quieter hours or receives cheaper requests, can sing happily while the mine fills. Most of the craft of canary analysis is in making the comparison fair.

Encore’s purchase path shows what fairness takes. At each step of Figure 4.3, an automated analysis compares the canary with a baseline: a group of the same size, running the old version, started at the same moment and receiving the same kind of traffic. Each of those conditions removes a way the comparison could lie. The canary is never compared with yesterday, because yesterday had a different audience and a different mix of events. It is never compared with the long-running fleet either, because processes that have run for days have warm caches and settled connection pools, which flatter them against anything freshly started; a new build would look slower for reasons that have nothing to do with its code. A baseline started alongside the canary suffers the same cold start, so whatever difference remains belongs to the change.

The analysis watches SLIs that fans would recognize as their own experience: the rate at which checkouts succeed, the p99 latency of POST /holds (the call that holds a seat while a fan pays), and the rate of PaymentFailed events caused by Encore itself, as opposed to a declined card, which tells us about the fan’s bank and nothing about our build.

Figure 4.3 — Encore’s canary schedule for purchase-path services. The request minimum matters as much as the minutes, because a canary judged on too little traffic cannot tell a bad build from noise.

Two columns of Figure 4.3 deserve a closer look. The first is the bake, the minimum time a step must run before the analysis may judge it. At step 1 the bake has two conditions, a time and a request count, and both must be met. The reason is statistical. If the canary has served only a few hundred requests, a handful of failures could be bad luck or a bad build, and no honest analysis can tell which; “beyond noise” in the table means precisely that the difference from the baseline is larger than chance alone would plausibly produce. The second column is the error-budget burn at step 3. Burn rate, from Chapter 1, measures how fast the service is spending its error budget, where 1× would use the budget exactly over the SLO window. At 25% of traffic, the canary is now large enough to matter to the budget itself, so the step adds a test the earlier steps did not need: the change must not be spending the budget at more than twice the sustainable pace. Added together, the bakes come to 45 minutes from the first fan on the new version to the last.

Statistics are not the only reason to roll back. An integrity alarm fires when the system violates a rule that must never be broken, such as a seat with two SeatSold events, which would mean two fans each hold a ticket for the same chair. Such an alarm rolls the canary back at once, whatever the statistics say, because no amount of good latency compensates for selling a seat twice; Figure 4.4 draws that path. Canaries also do not run during on-sales, for reasons Section 4.4 gives.

The schedule has a limit we should state plainly. Bakes measured in minutes catch only failures that show themselves in minutes. A slow leak, such as memory that grows a little with every request or a queue that drains slightly slower than it fills, passes every step and is left to the burn-rate alerts once the change is everywhere. That is why the largest operators, Amazon among them, bake for hours between waves of regions: they are buying time for slow failures to surface while most of the world still runs the old version.

Figure 4.4 — Figure 4.3’s schedule drawn over time on a logarithmic scale. A canary that holds reaches every fan in about 45 minutes. One that raises an integrity alarm at 25% goes straight back to zero, without waiting for the statistics.

A canary is one way to shape exposure; two others complete the toolkit, and each trades something different. Blue/green deployment runs a complete new environment (green) beside the old one (blue) and switches all traffic from one to the other at once. It gives up the gradual widening of a canary, and in return it buys an instant switch back, at the price of paying for double capacity while both environments exist. Encore uses it for the serverless Sale Gate, its virtual waiting room, where an all-at-once switch with an immediate way back suits a service that must behave identically for every fan in the queue.

Ring deployment widens exposure by population instead of by percentage. Each ring is a group of users chosen because of who they are, and the rings are ordered so that those best able to absorb harm see the change first. Microsoft has long released Windows and Azure this way. Encore’s rings start with its own staff and synthetic buyers, the scripted clients that exercise real purchase paths, who notice problems and complain to the right people. They then widen through the self-service and Professional cells to Encore’s own fan site, and end with the Enterprise silo for venue-chain-a, whose contract places it last. That final position is worth noticing: here the order of exposure is set partly by engineering judgment and partly by a commercial agreement, and the pipeline has to honor both.

Figure 4.5 — Encore’s delivery rings, widening by population from staff and synthetic buyers outward. The outermost ring is last because a contract places it there.

Configuration and data travel the same road

Everything so far has concerned code, and an engineer could finish the previous subsection believing the problem solved. History says otherwise. The three most instructive outages of recent years were not caused by code at all, and each traveled down a road that progressive delivery had not yet been built on.

On 19 July 2024 CrowdStrike pushed Channel File 291, a content update for its Falcon sensor on Windows. The sensor’s code itself went through staged rollouts. Its “rapid response content,” however, was treated as data rather than code, and data went to every machine at once. The file supplied a 21st input field to an interpreter that provided only 20. The content validator, the one check standing between the file and the fleet, missed the mismatch, and when the sensor read past the end of what it expected, the out-of-bounds read crashed about 8.5 million Windows machines, by Microsoft’s estimate, in the 78 minutes before the file was withdrawn. CrowdStrike’s remediation reads like a summary of this section: staged content deployment through canary rings, customer control over when updates arrive, and bounds checks in the code that reads the content.

On 12 June 2025 a policy change reached Google Cloud’s Service Control, the system that checks authorization and quota on API calls. The change carried blank fields. It replicated globally within seconds, and in every region it reached a code path that had been added two weeks earlier without a feature flag or error handling, and crashed it. The code path had been waiting, unflagged and unguarded, for the data that would wake it, and that data arrived everywhere at once. Google committed to making Service Control fail open, to flag-protecting changes to critical binaries, and to propagating global data “incrementally with sufficient time to validate and detect issues.”

On 18 November 2025 a permissions change in a Cloudflare database doubled the size of a generated feature file for Cloudflare’s bot-management model. That file was pushed network-wide every five minutes. It now overran the proxy’s limit of 200 features, and an unhandled error became a panic that broke much of the web for about three hours. Cloudflare’s first remediation was to harden “ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input.”

Set side by side, the three stories rhyme, and the three operators reached one recipe from different directions. Its first half is to stage configuration and data as we stage code: through rings, with a bake and an automated halt, so that a bad push reaches a canary before it reaches every region. Its second half is to make every consumer treat internal configuration as untrusted input, even though its own colleagues produced it. In practice that means four things. Validate the input on arrival. Enforce the limits the code silently assumes, such as Cloudflare’s 200 features or CrowdStrike’s 20 fields, so that a violation becomes a rejected file instead of a crash. Keep the last version known to be good, so that rejecting a new file leaves the component running on the old one. And decide in advance whether a component with nothing good to load should fail open, carrying on without the protection the configuration provided, or fail closed, refusing to serve until it has something trustworthy. Neither answer is right in general; what matters is that someone chooses it calmly, beforehand, instead of discovering it during an outage.

I have come to believe that this is the most underrated lesson of the decade in operations. As an industry we learned long ago to distrust input from strangers; we are only now learning to distrust input from ourselves.

Encore’s nearest equivalent is the Bot Screening rule bundle that every Sale Gate instance loads, the set of rules that decides which visitors look like bots. Shadow-mode trials (Chapter 1) run a new rule against real traffic without acting on it, which tests whether the rule is wise. They do not test whether the bundle that carries it is well formed, and until recently the bundle reached every instance within a minute, by its own road, outside the rings. The Reliability team has since moved the bundle, the flag service’s state and the edge routing table onto the same rings as code, and rewrote the Gate’s loader along these lines:

MAX_RULES = 5_000   # far above today's bundle, far below what exhausts memory

def load_bundle(raw: bytes, current: Bundle) -> Bundle:
    try:
        bundle = Bundle.parse(raw)          # schema, types, required fields
        if len(bundle.rules) > MAX_RULES:
            raise BundleError(f"{len(bundle.rules)} rules exceeds {MAX_RULES}")
        bundle.verify_signature()           # built by the pipeline, never by hand
    except Exception as err:                # a bad bundle must never crash the Gate
        metrics.increment("rule_bundle_rejected")   # pages Bot Screening on call
        log.error("keeping bundle %s: %s", current.version, err)
        return current                      # last-known-good
    return bundle

Each line of the loader answers one part of the recipe. Parsing checks the shape of the bundle. The MAX_RULES limit turns an assumption into an enforced rule, set far above any real bundle so it never trips by accident and far below the size that would exhaust memory, which is the Cloudflare failure made impossible. The signature check ensures the bundle came out of the pipeline rather than from someone’s laptop. Any failure keeps the current bundle and pages the Bot Screening team, so a bad file degrades into a quiet alert instead of a fleet of crashed waiting rooms.

The last decision in the recipe is the hardest, and the loader cannot make it alone. A Gate instance that cold-starts with no valid bundle has no last-known-good version to fall back on. Encore’s answer is to fail open on bot scoring but keep metering admission. Letting a few bots through for a while is a nuisance; letting the full crowd of an on-sale through at once would overwhelm Seat Inventory, and it is the rate limit, not the bot rules, that protects it.

Feature flags and their debt

Canaries and rings decide which machines and which populations receive a new version. Feature flags go one level finer. A feature flag is a runtime switch that decides which behavior deployed code exhibits, so that code can reach production long before its behavior reaches anyone. With a flag, deploying and releasing become two separate acts performed by different tools, and the release can be widened, narrowed or reversed without deploying anything at all.

Encore’s new seat-map renderer shows what that buys. Behind the flag seatmap-renderer-v2, it reached staff first, then one fan in a hundred on small events, then all events below 5,000 seats, and finally, after a quiet month, the stadium on-sales. When it misdrew curved balconies on one venue’s map, the team turned it off for that venue in seconds and fixed it at leisure. No rollback, no emergency deployment, no other venue affected: the flag shrank the blast radius to exactly the fans who needed protecting.

The price of that power is that flags accumulate, and they age differently by kind. A release flag, which guards a new feature during its rollout, should live for weeks and then be removed along with the old code path. A kill switch, which lets operators turn off an expensive or risky feature under stress, may rightly live for years. An experiment flag should live as long as its experiment and no longer. Trouble begins when flags outlive their purpose, and here Knight’s morning in Section 4.1 is the cautionary tale: a flag given a second meaning, guarding a dead function nobody had removed in nine years.

With Knight in mind, Encore’s flag service enforces a few simple rules, each aimed at one part of that failure. Every flag has an owner, a kind and an expiry date, so that no flag is anyone’s orphan. No name is ever reused, so that an old meaning can never wake under a new one. A flag change is a production change, recorded in the deploy history, so that an incident timeline shows the flip of a switch alongside every deployment. And an expired release flag blocks its owner’s next deployment, which is stricter than most tools’ default of a warning that is easy to ignore. Uber’s Piranha, a tool that writes the clean-up change itself by removing the flag and the dead branch from the code, is the gentler complement: it lowers the cost of doing the right thing instead of raising the cost of not doing it.

The rules came from a sobering count. When the team surveyed the service, a third of Encore’s 2,300 flags had not changed state in a year. Each of those flags still had two branches in the code, and for each one, only the branch in use was being exercised; the other had gone untested for a year and could fail in any way at all if someone flipped the switch. A flag that is never turned off is dead code wearing a switch.

Rollback or roll forward

Every technique in this section assumes that when something goes wrong, we can get back. That assumption deserves examination, because it is false more often than we like to think. To roll back is to return production to the last version known to be good; to roll forward is to repair the problem with a new change instead. Rollback is the right default for stateless code, because a stateless service holds nothing that outlives the process: replace the binary, and the world is as it was.

Knight’s morning shows where that reasoning fails. Its rollback spread the failure from one server to all eight, because the old code met a world the new code had changed: the repurposed flag was already flowing through the order messages, and the old code read it with its old meaning. Rolling back the code did not roll back the world. In most systems that changed world is data: rows written in a new format, events with a new field, a cache filled by the new version. Once the new version has written something the old version cannot read, rolling back means running code that misunderstands its own data.

The defense is to keep each change compatible with its predecessor, so that the previous version can always read whatever the new one wrote. Two patterns do this. The expand-and-contract migrations of Chapter 6 first expand a schema to accept both old and new forms, then migrate, then contract once nothing uses the old form. Amazon’s two-phase pattern applies the same idea to formats on the wire and on disk: one deployment teaches every node to read a new format, and only a later deployment begins writing it. Between the two, rolling back is always safe, because every node already understands everything that could have been written.

Not every change can be made this way, and Encore treats the exceptions as exceptions rather than pretending they do not exist. Its pipeline records whether each deployment is safe to roll back. One that is not gets a longer canary, because its only escape is forward and it must be right before it spreads, and a named engineer awake for its first hour, because rolling forward takes a person who understands the change.

Recap. Progressive delivery separates deploying from releasing so that we can choose each change’s blast radius. Canaries are only as honest as their comparison, so they are measured against a simultaneous baseline on enough traffic, and rings order exposure by who can best absorb harm. Configuration and data must travel the same staged road as code, into consumers that distrust them and keep a last-known-good version. Flags make release a runtime decision but turn into debt unless they are owned and retired. And rollback is safe only while the previous version can still read the data the new one wrote.

Exercise 4.3. Pick the configuration or data file that the most machines in your fleet load. How long does a change to it take to reach all of them? What validates it on arrival, and what limits does the code that reads it silently assume? And what does a consumer do with a malformed one: crash, keep the last good version, or fail open or closed, and did anyone decide which?

4.4 Testing in Production

Rehearsing in the real room

For a long time, “testing in production” was something an engineer said with a guilty smile, as a confession that the real testing had been skipped and the users had been left to find the bugs. The phrase has since changed its meaning, and the change is worth understanding. It is now the name of a discipline, and the reason is simple once stated: no staging environment reproduces production. Staging has a copy of the code, but it does not have production’s data, with its years of odd records and forgotten edge cases; it does not have production’s traffic, with its real mix of devices, networks and impatient people; and it does not have production’s scale. For Encore the last point is decisive. A 500× surge in the minutes after a big artist announces a tour cannot be rented for an afternoon and replayed in a test cluster. It exists only where it happens, in front of real fans, and so some of what we need to learn can only be learned there.

The theater has known this for centuries. A company rehearses for weeks in a rehearsal room, but it still holds a technical rehearsal and a dress rehearsal on the real stage, under the real lights, because the room changes things that no amount of rehearsal elsewhere can predict: the sight lines, the time a costume change really takes in the wings, the way a voice carries to the back of the hall. Testing in production is our dress rehearsal in the real room. The analogy breaks at one important point, and the techniques below exist because of that break: a dress rehearsal has no paying audience, while production always does. Every method of testing in production is therefore a way of learning from the real room while keeping the audience, and everything that would touch the audience, out of the experiment. Three such methods matter at Encore.

The first is the synthetic user: a scripted client that exercises a real path through production continuously, behaving as a customer would. Encore’s synthetic buyers do everything a fan does. They hold seats, pay through the payment provider’s test route and receive tickets, all for a hidden test event that no real fan can see, so they consume no real seats and move no real money. They run every hour, and every minute during on-sales. Their value is that they answer a question real traffic cannot. At three in the morning, an absence of ticket sales is ambiguous: it may mean nobody wants to buy, or it may mean nobody can. A synthetic buyer that fails to complete its purchase removes the ambiguity, because it always wants to buy. During an on-sale the situation is reversed, since real traffic is enormous and noisy, and the minute-by-minute synthetic purchase becomes a steady reference point that walks the whole path, from the Sale Gate through Seat Inventory and Orders to the payment provider, at a moment when everything else is moving.

The second is the dark launch: new code runs against real traffic, but its results are hidden from users, so the organization learns how the new code behaves before anyone depends on it. The technique has a well-known early example. In 2008 Facebook prepared its chat service by having live pages quietly send it simulated traffic, so that the back end met realistic load before a single user could open a chat window. Encore used the same idea when it built a new availability projection for Seat Inventory. (As Section 4.2 explained, a projection is a read model, built here by replaying the stream of seat events into a shape that is fast to query.) The new projection was fed the same events as the old one, and both answered every availability query; fans saw only the old answer, while the two answers were compared and every disagreement was investigated. The team switched over only when a fortnight had passed without a single disagreement. That is a demanding standard, and a fitting one for the service whose whole purpose is to sell each seat exactly once.

The third is shadow traffic: real requests are copied to a new version of a service, and the new version’s responses are thrown away. Where a dark launch usually lives inside the running system, shadowing typically happens in the network path, at the edge or in the mesh, which duplicates requests toward a separate deployment. The method is safe for reads, because answering the same question twice changes nothing. It becomes dangerous the moment a request causes a side effect in the world. Shadowing the Notification service naively would send every fan two emails, and if the shadow also issued wallet passes, every fan would receive the same ticket twice. Encore therefore shadows write traffic only into sandboxes, where the side effects land in test mailboxes and test wallets that no fan will ever open. The trade-off is plain: the closer a shadow comes to production’s real behavior, the more care it takes to keep its effects invisible.

These three methods let us learn from the real room safely on an ordinary day. On the day of a big on-sale, however, Encore does something that looks like the opposite of testing in production: it stops changing the purchase path altogether. That practice, and the slow way its boundary wears down, is the subject of the rest of this section.

Change freezes and their exceptions

A change freeze is a period in which production changes are forbidden except for emergencies, because a failure during that period would cost more than usual. The idea is old and intuitive. Retailers have frozen their systems through the holiday season for decades, on the reasoning that the weeks that bring in the most revenue are the wrong weeks to discover a bad deployment. The reasoning is sound, but the evidence that freezes actually reduce harm is thin. Practitioners report a familiar pattern at the other end: changes pile up during the freeze, and when it lifts they all land together, so failures spike in the days afterward. A broad freeze may only move the risk to a later date, and concentrate it there.

The one serious quantitative study I know of comes from Meta. In its winter 2023 freeze, Meta used risk models to decide which changes needed to be held, and gated only 16% of them; the rest were allowed through. Compared with the previous winter, it landed 42% more changes and saw 52% fewer outages. This is one company’s observational study, and it is not proof; a different winter, a different mix of changes or a different company might tell another story. But the evidence we have points one way: toward a narrow freeze aimed at the changes most likely to do harm, and away from a broad one that stops everything and then releases the backlog in a flood. Google’s freeze after its June 2025 Service Control outage (Section 4.3) had exactly that narrow shape. It covered only Service Control and manual policy pushes, and only until the repairs were in.

Encore’s freeze is scoped in the same spirit, and for a reason specific to its traffic. It covers the purchase path, from two hours before each big on-sale until the on-sale ends. The justification is that during an on-sale Encore’s ordinary safety instruments go blind. The rest of the week, a risky change is made safe by a canary (Section 4.3), which compares the new version with a baseline running the old one and waits for enough traffic and enough minutes to tell a bad build from noise. When load rises 500× in minutes, both conditions fail. There is no stable baseline, because the traffic, the caches and the load on each shard are changing faster than any comparison can settle. And there is no time to bake: a canary that passes every step of Figure 4.3 takes about 45 minutes to reach every fan, longer than the most intense part of the surge. So the instruments that make change safe on a Tuesday afternoon are unusable in precisely the hour when an outage costs the most. A freeze is what remains when those instruments are gone, and in Encore’s case it is a deliberate, narrow choice rather than a seasonal habit.

The hard part of any freeze is its boundary, the precise answer to the question “which changes does this cover?”, and boundaries erode. Over the past year Encore’s freeze has softened, one reasonable step at a time. Exceptions are granted in the on-sale chat by whoever is senior and awake at the moment, and are recorded nowhere else; there were 41 of them in 12 months, rising quarter by quarter from 3 to 11, 13 and 14. Low-risk platform automation, such as certificate rotation and the node-upgrade reconciler that drains and replaces Kubernetes nodes, was excluded from the freeze, because pausing it for every on-sale created more work than it saved. Catalog was judged to sit off the purchase path, on the grounds that a fan can buy a seat without ever searching for one. Each step had a sensible argument behind it, and the people who took them were not careless. Together, though, those steps turned the freeze from a rule into a custom, something the organization believed it followed while its practice quietly moved away. This is what safety science calls drift, in which a system migrates toward the edge of safety through a sequence of locally rational decisions, none of which looks like a decision to accept more risk.

There is a second, subtler weakness in how the boundary is drawn, and the Cloudflare outage of November 2025 (Section 4.3) shows it clearly. The change that triggered it was a routine permissions change in a database, far from the proxy it eventually broke: the database change altered a generated file, and automation carried that file to every machine. Now imagine a freeze written the way most freezes are written, as a list of protected services with automation exempt. It would have let that change through, because the change touched no listed service and no human pushed anything by hand. A freeze drawn around services guards the services themselves, but leaves open everything they depend on. Encore’s boundary is drawn that way, and so are its exemptions. Encore has not yet asked whether its freeze should follow dependencies rather than service names, or where its exceptions ought to be written down so that their accumulation is visible to everyone. The Operations book returns to both questions, first on the morning of the Vesper Hale on-sale and then in the review that followed it.

Recap. Synthetic users, dark launches and shadow traffic let us learn what only the real room can teach, while keeping users and side effects out of the experiment. The evidence for change freezes is thin, and what there is favors narrow, risk-based freezes over broad ones. Encore’s freeze is justified, because its canaries are blind during a 500× surge, but its boundary has drifted through a series of sensible exceptions, and it is drawn around services rather than around what those services depend on. Exceptions belong in writing, where their growth can be seen.

Exercise 4.4. List every change that reached production during your last freeze, including automation runs and configuration or data pushes. Who granted each exception, and where is it recorded? Then pick one frozen service and trace what it depends on: which of those dependencies could have changed during the freeze without anyone treating the change as an exception?

4.5 Machine-Written Change — Review, Fitness Functions, and Provenance

The Tuesday morning, revisited

The Architecture book ended on a Tuesday morning, after a coding agent had rebuilt Encore’s Notification service overnight. The agent had brought in a new framework and new retry semantics, and the result spanned 40 files that nobody had read in full. Every check passed, and the canary, judging the build on its service level indicators, promoted it. One engineer nonetheless spent 20 minutes on that change alone, reading the retry logic with a question in mind that no check was asking, and saw that the new retries could send a fan the same ticket twice. The engineer added a spec-level idempotency test before merging. The point of the story is not that the pipeline failed, because it did everything that was asked of it. The trouble was that nothing in it asked whether the new retries were safe, since until that morning nobody had thought the question needed asking.

We have met unread code before in this chapter. Power Peg sat in Knight’s router for nine years, and nobody read it again before the repurposed flag woke it; unread code is as old as software itself. What is new is the speed at which it now arrives. By autumn a quarter of Encore’s merged changes carried an agent label, and those changes were larger than the ones people wrote. The industry’s evidence points the same way, though it is young and should be read as such. DORA’s 2024 report estimated that each 25% increase in AI adoption went with a 7.2% fall in delivery stability, which is to say that teams adopting these tools more heavily also saw more of their changes go wrong. Its 2025 report found that throughput gains had arrived but that the stability penalty was still in place, and it named small batches and strong version control among the capabilities that help a team keep the gains without paying the penalty.

A second finding suggests why the penalty might exist. ImpossibleBench (2025) gave coding agents tests that they could not honestly pass, and found that the agents often edited the tests instead. This matters more than it first appears. A test is the pipeline’s written record of what we intended, and a pipeline can check a change only against that record. An author who is rewarded for making the checks pass, and who is able to rewrite the checks, can satisfy the pipeline while abandoning the intention behind it. People are capable of the same shortcut, but the pipeline was built on the assumption that the person changing a test knows what the test is for, and that assumption no longer holds for every author.

The best production account so far is Google’s 2025 report on large code migrations. About 80% of the modifications that landed were written by a model, yet every change still passed the existing tests, was reviewed by the owners of the code it touched, and was rolled out gradually. The lesson I draw from it is that a high share of machine authorship can be workable when the old disciplines of testing, ownership and progressive delivery surround it, and that what the machine changes is how much weight those disciplines must carry.

Encore’s policy

After the Notification morning, Encore’s architecture group and the Reliability team wrote a policy together. Its premise has two halves, and the tension between them is deliberate. The pipeline treats every change alike, whoever or whatever wrote it, because a check that relaxes for a trusted author is a check that will one day be walked around. Review, on the other hand, weighs what a change could break, because careful human attention is finite and must be spent where the damage would be greatest. The policy has four parts, and each answers a weakness that the Notification change exposed.

Spec-level tests. The first weakness was that Encore’s tests described the old implementation, so a rewrite could keep them green while changing behavior they had never pinned down. A spec-level test checks a property the business depends on, stated independently of any implementation, so that it stays true when every line beneath it is regenerated. For Notifications, the property is that one TicketIssued event yields exactly one wallet pass and one email under any sequence of retries and restarts. Encore writes it as a property-based test, which generates many such sequences (a timeout here, a crash after sending but before recording, a duplicate delivery there) and checks the property against every one. The test therefore explores orderings of failure that a person writing examples by hand would not think to try. And because ImpossibleBench showed what an agent may do with an inconvenient test, these tests are guarded. People write or approve them, and an agent’s pull request that modifies one is blocked until a human owner approves the test diff separately, through ordinary code-owner rules. The agent may change the implementation as freely as it likes, but it may not quietly change the definition of correct.

Small changes and provenance. The second weakness was size. A 40-file change is hard for any reviewer to hold in mind, and a canary that promotes it can say only that the whole looked healthy, not which part carried the risk. Following DORA’s 2025 advice, Encore now asks agents for changes that a reviewer can hold in mind at once; the Notification rewrite would today arrive as three changes, each passing through its own canary, so that a problem would surface in a third of the code rather than somewhere in all of it. The third weakness was that, after the fact, nobody could easily say what had written what. The golden path already signs build provenance, a verifiable record of how and from what source each artifact was built (Chapter 7 takes it up), and authorship now sits on top of it. Every commit records any agent that wrote it, with its model version, and the human who sponsored it, meaning the person who asked for the change and answers for it. The instinct is the same as the Linux kernel’s, which asks for an Assisted-by tag and lets only a human sign off a patch. I should be candid about the evidence here: nobody yet knows whether such labels shorten incidents. Encore keeps them because they are cheap to record now and cannot be added later, and an incident review that someday wants to ask whether a model version mattered can ask only if someone wrote the version down.

Risk-tiered human review. Human attention is the scarcest resource in the whole pipeline. Compute can be bought by the hour and checks can be added by the dozen, but an engineer who reads carefully can read only so much in a day. The policy therefore spends that attention according to blast radius, the idea Section 4.3 introduced for rollouts: here, the extent of the harm a faulty change could do before anyone stopped it, as Figure 4.6 shows.

Figure 4.6 — Encore’s review tiers. The tier comes from what a change touches; provenance decides only how long an agent’s change bakes and whether a human samples it.

The table rewards a careful reading, because what it leaves out is as deliberate as what it includes. Authorship does not decide the tier. A one-line change to the purchase saga is Tier 1 whether a person or an agent wrote it, and an agent’s rewrite of a dashboard is Tier 3. Provenance enters only at the margins, in the longer bake for agent changes in Tier 2 and in the sampling for Tier 3. Notice also what a Tier 2 reviewer is asked to read. It is not every line of code, which is the part an agent produces most cheaply, but the spec, the invariants and the test diff, which are the places where a wrong decision would be hardest to see later. The reviewer reads the contract closely and the implementation by sampling.

There is a useful analogy in a touring production’s stage manager during a changeover between acts. She cannot inspect every cable the crew lays, and she does not try to. She checks for herself the rigging that hangs over the performers’ heads, keeps an eye on whoever is working the pyrotechnics, and trusts the rest to practiced hands and a checklist, glancing at a corner now and then to keep everyone honest. Where the analogy breaks is revealing. A stage manager knows her crew, and their competence has been visible across years of shows. An agent’s competence is harder to read: it shifts with each model version, and it can be excellent in one file and confidently wrong in the next. That is why Encore samples an agent’s work instead of extending it a reputation.

Tiering review by what a change touches is old practice, and Encore did not invent it. IT service management has long sorted changes into classes by risk, and at Google owners files guard sensitive paths by requiring a named owner’s approval. What Encore added are two numbers, and they deserve honesty about their standing: the doubled bake and the one-in-ten sample are Encore’s own choices, without published precedent. The doubled bake is a cheap hedge in the spirit of DORA’s stability finding. If agent changes are somewhat more likely to fail, a longer bake gives a slow failure more time to show itself in the canary, and the only price is waiting. The sample exists for a subtler reason. Tier 3 relies on automated checks, and checks decay quietly as the code beneath them changes, so the sample is there so that someone notices when Tier 3’s checks begin to miss things, before the users of an internal tool notice first.

Fitness functions. The fourth part addresses the gap that sampling leaves open. If humans read the contract closely and the code only in part, something must still read every line. That something is the set of fitness functions we met at the pipeline’s fitness stage in Section 4.2, which Building Evolutionary Architectures (2nd ed., 2022) defines as automated checks that a system keeps a promised property as it evolves. At Encore they are the only reviewers that read every line at the speed agents write. Three examples show what they look like. Every handler with an external side effect, such as sending an email or issuing a wallet pass, must take an idempotency key, so that a retried request can be recognized as a repeat rather than acted on twice. No service may stack its own retry layer on top of the mesh’s, because two layers of retries multiply each other and turn a burst of failures into a storm of attempts; nor may any service exceed the 10% retry budget that Google’s overload guidance recommends. And no outbound dependency may appear without a registry entry, so that an agent cannot quietly put a new service on the path to a fan. The limit of fitness functions is worth stating plainly. A check can see that an idempotency key is present, but not whether the code uses it correctly, whether, for instance, it records the key before or after the email goes out. That is a question about behavior, and it is why the spec-level tests remain. The two kinds of check divide the work between them: fitness functions enforce structure everywhere, and spec-level tests verify behavior where it matters most.

None of this is free, and the policy does not pretend otherwise. Tier 1 reviews are slower than they were. Agent-heavy teams grumbled that the doubled bake looked like distrust, and in a sense they were right, though it is a distrust of risk that has not yet been measured rather than of the teams themselves. Spec-level tests take real thought, because stating a property independently of any implementation forces a team to decide what the business actually requires, a question that a test written against existing code lets it avoid. Against these costs stands a single observation. The Notification engineer’s 20 minutes were the most valuable that anyone at Encore spent that week, and they were valuable because they went to the right change, with the right question, instead of being spread thinly across everything. The policy exists to spend such minutes where they matter and to keep them from being diluted. A review that reads everything a machine wrote reads nothing carefully.

Recap. Machine-written change is new in two ways: in its volume, and in its tendency to satisfy the checks rather than the intentions behind them, and so far delivery stability pays for it. Encore’s answer combines proven pieces (guarded spec-level tests, small changes, build provenance and fitness functions) with authorship labels and sampling rates that are its own young choices, kept because they are cheap and because their worth can be judged only later. The pipeline checks every change alike; review also weighs who wrote it.

Exercise 4.5. Choose a service you own and write its two most important invariants as spec-level tests, stated so that they would remain true if every line of the current code were replaced. Then decide which review tier the service belongs in, and describe what your review would read in full, and what it would only sample, if an agent rewrote the service overnight.

Kata: Always Open

Your brief: “Kestrel.” Kestrel is a mobile-first retail bank with 1.8 million customers in three EU countries, built and run by 110 engineers in 14 teams. A bank of this kind never closes. Whenever a customer taps a card, at a supermarket at noon or a petrol station at three in the morning, the payment network asks Kestrel to approve or decline it, and every such card authorization must be answered within 2 s, day and night. Above that sits a regulator, which requires a record of every production change: who approved it, what it touched and how to undo it.

Kestrel changes itself often, about 60 deployments a day, but its change records do not keep pace with its changes. They are written by hand after the fact, for a weekly risk committee that reviews them, and its change failure rate, the share of changes that cause a failure in production, is about 15%. Not every change travels through the deployment pipeline, either. Kestrel’s fraud rules, which decide which payments look suspicious, are treated as data rather than code: they are published from a console straight to every authorization node at once.

Kestrel’s flags have their own history. The bank carries 1,900 of them, and one, a flag that had controlled a fraud rule, was reused for a new rule. For a whole morning it declined valid card payments in one country. Kestrel also freezes production for Black Friday, but the exceptions to that freeze are granted in chat. And coding agents now write a third of Kestrel’s pull requests, most of them fraud rules.

Kestrel’s troubles will look familiar, because each of them is an episode of this chapter wearing a bank’s clothes. The reused flag is Knight Capital’s repurposed flag, with a fraud rule in place of Power Peg. Rules pushed to every node at once take the same road that carried CrowdStrike’s Channel File 291 and Cloudflare’s feature file. The weekly committee is the external change board that Accelerate found brought no better change failure rate. The exceptions granted in chat are Encore’s 41 exceptions, recorded nowhere else. And the agents’ pull requests are the Notification morning of Section 4.5, with one sharpening detail: at Kestrel, the changes machines most often write are fraud rules, which sit directly on the path that must answer within 2 s.

The deliverables follow the chapter’s order. Taken together, they should describe one bank in which every change, whether code, configuration or data and whether written by a person or a machine, travels a known route, reaches a small population before a large one, leaves a record the regulator can read without anyone writing it by hand, and can be undone, or else is known in advance to be unsafe to undo.

Deliverables:

1. Change-route map — List every route by which a change reaches Kestrel’s production, treating code, configuration and data alike: the pipeline, the fraud-rule console, the flag tool, scripts, platform automation and anything else you find. For each route, mark whether its changes are reviewed, tested, recorded, staged and reversible, and name the gaps. This is Exercise 4.2 at the scale of a bank, and it should make plain how many of Kestrel’s changes never meet the pipeline at all.
2. Pipeline redesign — Lay out the stages a change passes through, as a sequence of increasingly expensive questions, with peer review by someone who knows the code. Have the pipeline write the change record itself, capturing who approved the change, what it touched and how to undo it, so that the regulator’s record is produced as a side effect of shipping rather than reconstructed afterwards. Then take a position on the weekly risk committee: what, if anything, it should still do once the evidence lives in the pipeline’s record instead of in its minutes.
3. Progressive rollout plan — Define the customer rings through which Kestrel’s changes widen, and the canary SLIs that decide whether a change may move to the next ring; for a bank whose promise is a 2 s answer, the speed and outcome of card authorizations are natural places to look. Show how fraud rules are published the same way: staged through rings, and validated on arrival by every authorization node, so that a malformed rule set is rejected rather than loaded. Finally, say when rollback is unsafe, because the old version would meet data the new one has changed, and what extra care such changes receive.
4. Flag and freeze policy — Write the rules every flag must obey, including how you ensure a flag’s name and meaning are never reused, and a plan for working down the 1,900 flags Kestrel already has. Scope the Black Friday freeze by dependency and risk rather than by a list of services, and replace the chat with an exceptions register that places every exception in the change record.
5. Review tiers for machine-written change — Define review tiers by what a change could break, so that human attention goes where an error costs most. Write spec-level tests for fraud rules, stating properties the bank depends on that must stay true however the rules beneath them are regenerated, and fitness functions that check every change automatically for the properties Kestrel has promised to keep.

Rubric:

Where you now stand

You can now treat change as the leading cause of outages without trying to stop it, because you have seen that changing less only makes each change larger and each deployment clumsier. You can design a deployment pipeline as a sequence of increasingly expensive questions, and let it write the record that a change board once kept by hand. You can choose each change’s blast radius, through canaries compared against a simultaneous baseline and rings ordered by who can best absorb harm. You can send configuration and data down the same staged road as code, into consumers that validate them and keep the last good version when a bad one arrives. You can keep feature flags from hardening into debt, scope a freeze honestly and keep its exceptions in writing, and spend human review where machine-written change can do the most harm. Everything in this chapter has been about the changes we choose to make. Chapter 5 turns to the failures we do not choose, and to how a system can be built to withstand them.

References

• Jez Humble, David Farley — Continuous Delivery. Addison-Wesley, 2010.
• U.S. Securities and Exchange Commission — In the Matter of Knight Capital Americas LLC. SEC, 2013 — free online.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering. O’Reilly/Google, 2016 — free online.
• CrowdStrike — Channel File 291 Incident: Root Cause Analysis. CrowdStrike, 2024 — free online.
• Google Cloud — Incident report, 12 June 2025. Google, 2025 — free online.
• Matthew Prince — Cloudflare outage on November 18, 2025. Cloudflare, 2025 — free online.
• Nicole Forsgren, Jez Humble, Gene Kim — Accelerate. IT Revolution, 2018.
• DORA — Streamlined change approval. dora.dev — free online.
• Clare Liguori — Automating safe, hands-off deployments. Amazon Builders’ Library — free online.
• Audris Mockus, Rui Abreu, Peter C. Rigby, et al. — Leveraging Risk Models to Improve Productivity for Effective Code Un-Freeze at Scale. ACM TOSEM, 2025.
• DORA — Accelerate State of DevOps Report 2024 and State of AI-assisted Software Development. Google Cloud, 2024–2025 — free online.
• Stoyan Nikolov, et al. — How is Google using AI for internal code migrations?. ICSE-SEIP, 2025 — free online.
• Ziqian Zhong, Aditi Raghunathan, Nicholas Carlini — ImpossibleBench. arXiv, 2025 — free online.
• Linux kernel documentation — AI Coding Assistants. kernel.org, 2025 — free online.
• Neal Ford, Rebecca Parsons, Patrick Kua, Pramod Sadalage — Building Evolutionary Architectures, 2nd ed.. O’Reilly, 2022.