Building Resilient Systems in the AI & Cloud Era

← the whole book on one page

Chapter 9: Production Readiness — The Capstone

“Plans are worthless, but planning is everything.” — Dwight D. Eisenhower

On 30 October 1935, at Wright Field in Dayton, Ohio, the United States Army Air Corps was flying the entrants in a competition for a new long-range bomber. Boeing’s entry, the Model 299, was the favorite. It was larger, faster and longer-ranged than its rivals, and it was also more complicated than any bomber the Army had flown: four engines, each with its own settings, retractable landing gear, wing flaps and trim tabs, all of which had to be managed in the right order. That morning it took off with Major Ployer P. Hill at the controls, one of the Army’s most experienced test pilots, climbed steeply, stalled and crashed. Hill died of his injuries, and so did Boeing’s own test pilot, Leslie Tower, who was aboard. The investigators found nothing wrong with the aircraft. The elevator gust lock, a clamp that holds the tail’s control surfaces still while the aircraft is parked so that the wind cannot batter them, had never been released. A crew of great skill had forgotten one routine step among dozens, and that step happened to be fatal.

Boeing lost the competition, and what happened next is the part that matters to us. One conclusion lay close to hand, that the airplane was simply too much for a crew to manage. The Army’s test pilots drew a different one. They could hardly have found a pilot more skilled than Hill, so they did not ask for more skill. They wrote a checklist instead, short enough to fit on a card, setting out the steps for each phase of flight in order, to be read aloud and confirmed. The Army bought a small number of the aircraft for further testing, flew them with the cards, and the Model 299 went on to become the B-17. The card added no skill to anyone. It moved one kind of memory out of the pilot’s head and onto paper, so that the steps a skilled person forgets precisely because they are routine were read rather than recalled.

The card has a limit, and the limit is the subject of this chapter. The gust lock went onto it because a gust lock had killed two men. Every line on such a card is the residue of some earlier failure, written down so that nobody has to have it twice. A checklist is a memory of failures someone has already had. It says nothing about the failure nobody has had yet, and it says nothing either about the things its authors decided, for good reasons, need not be on it. Every readiness review in software descends from that card. Google’s launch coordination engineers work from a launch checklist, part of which the Site Reliability Engineering book prints; Amazon offers its customers a structured review before a planned peak; and Encore has the card we met in Section 2.3, which its Reliability team, the platform group’s ten site reliability engineers (SREs), works through on the Wednesday before every big on-sale.

This chapter attends one of those Wednesdays. The on-sale is the largest of Encore’s year: Vesper Hale’s 12-date European tour, opening on Friday at 10:00 CET, with about 1.1 million fans expected in the waiting room. We will follow the review through the chapters of this book in their order. Section 9.1 takes the objectives and the error budgets of Chapter 1; Section 9.2 the capacity and cost arithmetic of Chapter 2; Section 9.3 the control plane of Chapter 3 and the change discipline of Chapter 4; Section 9.4 the failure design of Chapter 5 and the data of Chapter 6; and Section 9.5 the security of Chapter 7 and the AI workloads of Chapter 8, before it follows Friday morning to the minute before the doors open. (Chapter 1 of the Operations book’s map of Encore shows where each piece sits.)

I should say now how the review ends, because this chapter is not a mystery. It passes. It is run carefully by people who have, in effect, read the chapters before this one, and nearly every line of it is answered with evidence. It is a good review, and if you work through it as an exercise you will find little to correct. And yet a careful reader, following it line by line, will notice a handful of places where it rests on a belief rather than on a measurement: a test whose setup was chosen sensibly, an exemption granted long ago for good reasons, a list drawn up before the system it describes had changed, a rule that nobody had cause to revisit. I will point to some of them as we pass and gather them at the end, without saying what Friday made of them. That story belongs to the Operations book. The purpose of this chapter is to show what engineering for reliability can establish before a peak, which is a great deal, and to show as honestly as I can the shape of what it cannot.

9.1 The Readiness Review — Objectives and the Budget in Hand

A meeting with a card

The review meets at 14:00 on Wednesday, in the largest room Encore has, with a video link for those who cannot be in it. It is chaired by the staff SRE who leads the Reliability team, and its attendance list reads like the dependency graph of an on-sale written out as people: the On-Sale team, which owns the waiting room and the admitted fan’s journey; the Inventory team, which owns Seat Inventory; Orders & Payments; Bot Screening; Security Engineering; Support Experience, which runs the support assistant; someone from the Finance platform team; and the engineers who will hold the pagers on Friday. Some teams send numbers instead of people. The Runtime team, which runs the clusters, sends its node capacity in writing, as it does for every big on-sale, because nothing it owns is a line on the card.

The meeting has a fixed shape, which Encore took from the launch reviews the Site Reliability Engineering book describes. It walks a card from top to bottom, each line is answered by the team that owns it, and each answer comes with its evidence attached the day before, so that the hour goes on questions. The card has grown since Chapter 2 showed its first five lines. This is how it stood at the end of Wednesday afternoon:

On-sale readiness — Vesper Hale, 12 dates, Fri 10:00 CET (T−2 days)
[x] Objectives: on-sale-window SLO armed; 28-day budgets above policy lines
[x] Admission: 1,500 fans/s per event; arithmetic and load test attached
[x] Load test: open model, bursty, skewed seats, cache state stated (warm)
[x] Writes per shard ≤ 80% at planned rate; failover drain < 5 min
[x] Pre-scale: curtain-up 07:30; Gate concurrency, floors, cache warm
[x] Quotas: function concurrency, LB, PSP rates ≥ 1.3 × forecast peak
[x] Freeze: purchase path from 08:00; standing exemptions unchanged
[x] Dependencies: purchase-path registers current; ladder rehearsed Thu
[x] Data: ledger quorum healthy; backups fresh; restore approvers on rota
[x] Bots: new rules in shadow mode only; false-block rate in objective
[x] AI: token quota and reserved capacity; golden set green two nights
[x] People: on-call, incident commander rotation, comms lead named

Read the card as the Army pilots would have read theirs and every line turns out to be a piece of Encore’s history. The quota line is there because a Sales Analytics backfill once drained the Gate’s share of the cloud account’s function concurrency during a rehearsal (Section 3.3). “Shadow mode only” is there because of the spring dispute between On-Sale and Bot Screening (Section 1.3). “Restore approvers on rota” is there because the first Finance ledger drill lost an hour and forty minutes just reaching its backups, partly because one of the two approvers was on leave (Section 6.5). The line about the payment service providers (PSPs) goes back furthest of all, to Encore’s first outage, when a slow PSP took the purchase path down with it. The card is Encore’s history of failure, compressed into a dozen questions, and the history it records is real and was paid for. What it cannot record is a failure Encore has not yet had.

The hour that matters, priced

The first line concerns the objectives, and the review begins with the one written for days like Friday. Chapter 1 gave Encore a family of service level objectives (SLOs): the seat map within 800 ms for 99.9% of requests, checkout within 3 s for 99.9% of attempts, tickets in the wallet within five minutes for 99.5% of orders, all over a rolling 28 days. It also gave every major on-sale an on-sale-window SLO: 99% of admitted fans finish their journey without an error page or a false block in the first 60 minutes of the sale, counted per fan rather than per request (Section 1.2).

It is worth doing the arithmetic for Friday, because it turns a percentage into something a person can picture. If the forecast holds, and the Gate admits fans at the planned rate, even the longest queue for the most popular date drains within minutes, so most of the 1.1 million fans in the waiting room will be admitted inside the hour. One percent of them is about 11,000 people. That is the window’s error budget: the number of fans whose morning Encore has agreed, in advance and on paper, it can afford to spoil. It is a strange quantity to plan around, and the strangeness is instructive. Nobody in the room wants to spoil 11,000 mornings, and nobody expects to. The number exists so that a bad minute can be weighed against something, and so that on the following Monday nobody has to argue from feeling about whether the sale went well.

Chapter 1 also showed why no burn-rate rule can protect so short a window. A total failure spends 1% of an hour in 36 seconds, and a rule that waits for an hour of evidence fires after the sale is over. So Encore protects the window with people, and the review’s job here is to make sure the people exist. On Friday the On-Sale team’s primary on-call, Rui, will watch the one-minute per-fan error ratio from the moment the Gate opens, together with the on-call engineers of the teams on the purchase path: Jonas for Seat Inventory, Noor for Bot Screening, and the Orders & Payments on-call. Behind them the rescaled page stays armed, firing when both the one-minute and the five-minute per-fan error ratios pass 10%, a burn rate at which the window’s whole budget would be gone in six minutes. The incident commander rotation has Marta, a staff engineer from Orders & Payments, on it that week, reachable within five minutes. Elif, from the Reliability team, will act as comms lead if one is needed, with a status-page draft already written.

A theater has a name for this moment. Thirty-five minutes before curtain, the stage manager calls “the half”, and from that call every member of the company must be in the building and accounted for; an actor who is not there by the half is treated as missing, whatever the reason. The review’s last line is Encore’s half: it names who will be in the building. The comparison holds well for its purpose, which is to make presence a fact. It breaks at the desk. A stage manager can see the whole stage from the prompt corner. On Friday nobody at Encore will see the whole of the system at once. Each of the people the review named will watch the dashboard of one team, and the picture of the whole will exist only in what they say to one another.

Two more numbers complete the objectives. The checkout SLO’s good outcomes include a seat held pending payment, so a morning on which both PSPs falter can still be a good morning for fans if the degradation ladder of Section 5.2 works. And Bot Screening’s false-block rate, the share of admitted sessions screened that its rules turned away wrongly, has an objective of no more than one in a thousand. Applied to a crowd of this size, even that objective allows about a thousand fans to be challenged by mistake. It is worth remembering how the false-block rate recognizes a mistake: by the challenge the blocked fan goes on to pass, which is the operational test the spring dispute produced (Section 1.3).

What the budget said

The second half of the objectives line asks about the error budgets, the 28-day allowances that the error budget policy acts on. Figure 9.1 shows them as the review saw them on Wednesday.

Figure 9.1 — The error budgets on Wednesday afternoon. Every journey sat above the policy’s 25% line, so no team was in the review tier and every team was free to ship until the freeze.

The table settles several questions at once, and it is worth seeing which. No journey had less than a quarter of its budget left, so under the policy that the heads of engineering and product had signed (Section 1.3), no team’s changes needed a second reviewer from the Reliability team, and nothing argued for widening the freeze beyond its usual scope. The last row shows that Encore’s recent major on-sales had all met their windows, the weakest by a margin of three-tenths of a point. And the false-block rate was comfortably inside its objective, which suggested that the arrangement made after the spring dispute (a challenge with every block, shadow mode for every new rule) was doing its work.

What the table cannot do is just as important. An error budget, in the language of Section 1.4, is a gauge rather than a flow: it reports how much remains after the last 28 days, and it says nothing about what the next hour will cost. The review knew this and used the numbers accordingly, as permission to proceed and never as a forecast. One member of the room did turn a remaining budget into a warning. Sixty-four percent of the shop-once-admitted budget is about 5,500 bad journeys, and the spring dispute had shown that 6,000 blocked fans could take 70% of that budget in a quarter of an hour. A bad Friday could put whichever team’s change caused it into the review tier for weeks. Everyone agreed that this was true, and that the way to avoid it was the rest of the card.

Recap. A readiness review is a checklist in the lineage of the Model 299’s card, and each of its lines records a failure the organization has already had. Its first work is to price the hour that matters, here an on-sale window whose 1% budget comes to about 11,000 fans, and to name the people who will watch it, because no burn-rate rule can. The 28-day budgets tell the review what Encore can afford to spend and whether any team must slow down; they report the past faithfully and make no claim about Friday.

Exercise 9.1. For the next predictable peak your system faces, write the event-window objective and turn its error budget into a count of people, as this section did with 11,000 fans. Then name, for each team on the critical path, the person who will watch the indicator and the page that stands behind them. Which team on the path has nobody named, and why?

9.2 The Number at the Gate — Capacity, Headroom, and Cost

A windfall to spend

The second line of the card is the one the review spent longest on, and rightly, because it changed a number that every later line depends on. The On-Sale team proposed to raise the Gate’s admission rate from 1,200 to 1,500 fans per second per event.

The reasoning deserves to be heard in full, because it was good. Chapter 2 built the 1,200 from a per-fan demand of about four ordered seat writes and 26 availability reads over a session of about three minutes, which at 1,200 fans a second loaded each event’s four shards to 80% of their rated write capacity (Figure 2.5). Since then the new seat-map renderer of Section 4.3 had finished its careful rollout and now served the stadium on-sales. It draws the availability of a whole section with one read, and it refreshes only the sections a fan is actually looking at. And because its picture of which seats are free is fresher, fewer fans reach for a seat that someone faster has just taken, so fewer of them need a second hold. The On-Sale team had measured the effect in Soundcheck, its load-test harness, and found about 3.2 writes and 20 reads per fan. Each fan now asked less of the shards, and so more fans could be admitted for the same load.

The engineer who brought the proposal framed it as a question of fairness to the back of the queue. For the most popular dates, the fans at the back of a long queue reach the seat map after the best seats have gone, and at 1,200 fans a second some of them reach it after everything has gone. Admitting them a fifth sooner gives more of them a real chance at a seat. The review’s rule for such a proposal, set down in Section 2.3, is that it must arrive with the arithmetic redone at the new rate and a load test run at the new point. It did, and Figure 9.2 shows the arithmetic.

Figure 9.2 — The admission proposal, with the arithmetic redone at the new rate as the card requires. Every row passes. Every per-fan number in it was measured on Tuesday afternoon.

Each row answers a line of the card. Writes still bind, and they bind at exactly the 80% that Chapter 2 defended: 1,500 fans a second, each making about 3.2 writes, ask for the same 4,800 writes a second that 1,200 fans making four writes did. Reads fall a little, to 75%. The hot-shard row reflects a change the Inventory team had made since Chapter 2 worried about skew: each date’s front-floor sections are now spread across all four shards, and Soundcheck measured the busiest shard at 27% of the writes. The failover line of the card can then be checked directly. A failover rejects writes on its shard for about 40 s, so the busiest shard would come back owing about 51,800 writes, which is 1,296 a second multiplied by 40. It has 204 writes a second to spare, since 1,500 minus 1,296 is 204, so it would drain that backlog in about 254 seconds, a little over four minutes. The card asks for under five. It passes, without much room. And Little’s law, applied to the last row, says that about 270,000 fans will be in seat selection or checkout at once for each date while admission runs at full rate, which the holds store and the session services had been sized to carry.

The review was, in effect, deciding how to spend a windfall. Nobody proposed to touch the 20% of headroom that Section 2.3 had argued for, the margin that absorbs skew and the backlog of a failover. What had changed was how many fans it took to fill the other 80%. Someone from the Finance platform team asked whether the saving could come off the bill instead, and the answer showed how the costs of Section 2.5 are shaped. Shards are step-fixed costs, allocated whole per event, and an event cannot run on three and a fifth shards. The saving could be spent only as speed or as margin, and the On-Sale team proposed to spend it as speed and to keep the margin where it was. The room approved, and the new rate went out that evening through the configuration rings.

Tuesday afternoon in Soundcheck

The load test behind Figure 9.2 was a good one by every standard Chapter 2 set. Soundcheck replayed recorded on-sales, scrubbed of personal data, against the production-sized copy of the purchase path. It generated load from an open model, with each simulated fan arriving on its own schedule, and it failed any run in which the generator had dropped a scheduled arrival, so coordinated omission could not creep back in. Its arrivals were bursty, its seat choices were skewed toward the front-floor sections, and its stub of the payment providers answered with a latency distribution taken from production. After holding 1,500 fans a second, the run pushed on to 1,900, past the point where goodput stopped rising, and then fell back to 1,500 to confirm that the system recovered on its own, as Section 5.3 recommends. It did. The p99 of POST /holds stayed well inside its share of the checkout budget.

The team ran two tests that afternoon. The first, at 1,200 fans a second, re-measured the per-fan numbers with the new renderer in place. The second, at 1,500, began about an hour later. The test plan stated the cache state as the card requires, in a single line: seat-map geometry cache: warm; matches on-sale conditions, since curtain-up warms the cache before the doors open. That statement was honest, and it was true in the sense intended. On the morning of an on-sale, the curtain-up workflow does warm the geometry cache, the layout of every hall that Catalog serves from a cache held in its own pods, and its cache step reports the hit ratio it achieved. A test against a cold cache would have measured a morning Encore did not intend to have.

Ships offer the oldest picture I know of a margin and the conditions it depends on. In 1876, after a long campaign by Samuel Plimsoll against overloaded and unseaworthy vessels, Parliament required British merchant ships to carry a load line, a mark on the hull showing how deep the ship may safely sit in the water. What makes the load line instructive is that it is not one line. The same ship floats higher in dense, cold salt water than in warm fresh water, and the sea is rougher in a North Atlantic winter than in a tropical summer, so the mark carries a ladder of lines, one for each kind of water and season, and a master loads to the line for the water the ship will actually meet. Encore’s 80% is a load line in just this sense: a margin that keeps the shards afloat with room for weather. The analogy breaks in an instructive place. A ship’s hull carries all its lines at once, painted where everyone can see them. Encore’s capacity figure carries one line, measured in one condition, and the condition appears only in a sentence of the test plan.

Two facts about that condition are worth holding in mind, without yet drawing anything from them. The first is that the per-fan numbers belong to the whole morning as much as to the shards. Fans ask for fewer reads and fewer second holds when their seat maps draw quickly and show the hall as it is, which is the effect the new renderer was built to produce; the numbers were measured in a hall whose maps drew quickly. The second is that the afternoon’s second run met a system its first run had already warmed, the very effect Section 2.3 warned about, though here the warmth was the intended condition. The last production-scale run against the hidden test event, months earlier, had been at 1,200 fans a second and before the renderer changed. The 1,500 point had been measured on the production-sized copy, which the card permits. A load test answers the question its setup asked, and this one had asked a fair question carefully.

Limits that belong to others, and the bill

The card’s quota line covers the limits that other people set, and here the review was brisk because the nightly quota inventory of Section 3.3 had done most of the work. The Gate runs in its own cloud account, with reserved function concurrency sized to Friday’s forecast and provisioned concurrency to be raised toward its 400 instances by curtain-up; the account’s limit stood at 1.6 times the forecast peak. The load balancers stood at more than twice their forecast load. Both PSPs had been sent the morning’s forecast, as Amazon’s Infrastructure Event Management practice suggests, and both had confirmed that their agreed request rates stood at 1.4 times Encore’s peak. The autoscaler ceilings of Section 2.4, derived from what each event’s shards can absorb, were recomputed against the new per-fan demand and came out unchanged, since the shards would see the same writes a second as before.

The bill took five minutes. Pre-scaling is paid for from the moment capacity is warm, so the Gate’s provisioned instances and the raised floors elsewhere would cost a few thousand euros for the two and a half hours between curtain-up and the doors, doing nothing a cost report would recognize as work. Encore’s variable cost per ticket, €0.12 at last measurement (Figure 2.6), put the morning’s marginal spend on the order of €24,000 if the 12 dates sold their roughly 200,000 tickets. Reserved model capacity for the support assistant, which Section 9.5 returns to, appeared as its own line. Cost anomalies at Encore notify the owning team urgently in business hours, and a Friday morning is business hours, so a runaway line would be seen the same day. Nobody proposed a saving. On a morning like this the cheapest configuration is not a goal, and Section 2.5 had already shown why headroom that looks idle on a bill is the price of a short queue paid in advance.

Recap. The review treated a change to the admission rate as Chapter 2 asked: with the arithmetic redone at the new rate, a load test run at the new point, and the test’s cache state written down. The windfall from lower per-fan demand was spent on speed while the 20% of headroom was kept, and the quotas, ceilings and bill were checked against the forecast. Every per-fan number came from a careful test on one afternoon, in one condition, and a margin is only as good as the water it was measured in.

Exercise 9.2. Take the capacity figure your system relies on for its next peak and write beside it, as a load line would, the condition in which it was measured: cache state, data volume, client versions, time of day. Then name one condition the peak itself might bring that differs from it, and estimate which per-user number would change first if it did.

9.3 Loops and Changes — The Control Plane and the Freeze

Curtain-up and the governors

The pre-scale line of the card brings the review to the control plane, and to the crowd of governors that Chapter 3 taught us to see in every runtime. Most of Friday’s preparation is itself one of those governors. At 07:30 the curtain-up workflow will propose a plan for each of the 12 events, Rui will review it, and on his word it will run. The plan raises the Gate’s provisioned concurrency toward its 400 instances, in EU-1 where the tour’s writes are homed and, since the October region game day of Section 6.5 found the gap, in EU-2 as well. It raises the floors of the autoscalers in front of Orders and Catalog through commits to Git, which is the lesson of Catalog’s fight with Argo CD in Section 3.1: intent goes to the loop that owns the field, and it leaves a record. It sets Seat Inventory’s minimum replica counts. And it warms the seat-map geometry cache for all 12 venues, reporting the hit ratio it achieves. Each step checks its result, and the workflow finishes before the freeze begins at 08:00 and posts its report to the on-sale channel.

The review checked the plan the way Section 3.1 recommends checking any set of loops, by asking who owns each field. The autoscalers own replica counts, and no manifest beside an autoscaler sets replicas, which Encore’s policy rules now enforce. Argo CD reconciles everything else from the environment repository, with self-heal on. The cluster autoscaler adds nodes when pods cannot be placed, and the Runtime team’s written note confirmed that the core clusters had node headroom for the floors curtain-up would set. Argo CD’s sync windows, which Section 3.4 introduced, were available but still not wired to the on-sale calendar; the freeze is enforced instead in the delivery pipeline, which refuses deploys to frozen services between the freeze’s start and the sale’s end, and in the on-sale channel. The review found nothing on this line to change.

The tenants’ cells made a short appearance. Three Professional tenants had small sales of their own on Friday morning, each in its own cell, and the review noted them only to confirm that nothing in Vesper Hale’s preparation touched those cells. That is what cells are for (Section 5.4): a cell’s trouble stays inside it, and so does its good fortune, which means that a tenant’s sale in cell P7 needs no line on Encore’s card at all.

Cue, Encore’s operations agent, has a place on this line too, because its permissions change with the calendar. At 08:00 the control plane withdraws its write scopes on the purchase path, so that nothing Cue proposes during the freeze can produce a working token. If an incident is declared, its mitigation actions return, each still requiring a named person’s approval in the channel. Two small actions may run without approval, granted on Cue’s record: raising Catalog’s autoscaler floor through Git, up to a cap of 60 replicas, and holding a node in the platform’s upgrade reconciler. The review confirmed that the withdrawal was scheduled and moved on.

One governor did not appear in the review at all, and it was absent for a reason the room would have found perfectly sound. The Kubernetes upgrade wave of Section 3.2, which moves each new version from the canary cell through the Professional cells to the core clusters last, had reached its final stage that week. The node-upgrade reconciler was working through the core clusters’ node pools one node at a time, as it had for every wave since it replaced the old upgrade weekends, draining each node through the disruption budgets that allow Seat Inventory to lose one pod per shard and no more. The wave had been announced in the platform’s weekly change digest, where waves are always announced. It did not come to the readiness review because the review covers what the freeze covers, and the reconciler, as low-risk platform automation, had been exempt from the freeze for more than a year (Section 4.4). As far as anyone could recall, it had never troubled an on-sale.

Touring productions know a version of this arrangement. A show that arrives at a venue brings its own crew, who run the show’s lights, sound and rigging, while the house crew runs the building: the fire curtain, the house lights, the air handling, the maintenance of the building’s own systems. The production manager advances the venue in detail, but the building’s maintenance schedule is the building’s business, and nobody on the tour thinks to ask whether the house plans to test its fire alarm during the second act, because no venue would. The comparison is exact up to that last clause, and it breaks there. A house engineer about to start an alarm test would look through the door, see an audience and wait. The reconciler knows disruption budgets and nothing of calendars, as Section 3.2 put it; it looks at nodes, and it never looks through the door.

An exemption is a decision not to look. It is made once, carefully, by people who weighed the alternatives, and pausing the reconciler before each of Encore’s 160 on-sales a year had genuinely created more work than it seemed to save. After that the exemption stops being a question, which is its purpose, and the thing exempted stops being examined, which is its cost. Nobody at Encore had decided that the reconciler should drain nodes during a major on-sale. They had decided, long before, that it did not need to be asked.

What shipped before eight

The card’s freeze line has two halves: what changes before the freeze and what may change during it. The first half is the subject of Chapter 4, and the review took it change by change, because about two-thirds of Encore’s incidents in the past year had begun within an hour of a change (Section 4.1).

Four changes reached the purchase path between the review and the freeze. The new admission rate, a single setting in the tour’s event configuration, went out on Wednesday evening through the same configuration rings as code: first to the staff ring and the synthetic buyers that run against the hidden test event, then outward, with a bake at each ring. The flag state that put seatmap-renderer-v2 in front of the 12 dates went out on Thursday morning by the same road, recorded in the deploy history like any other production change. Bot Screening’s rule bundle carried two new rules, both in shadow mode, so that they would log what they would have blocked without blocking anything; the bundle traveled the rings on Thursday, and the Gate’s loader validated its shape, its size limit and its signature, keeping the last good bundle if anything failed (Section 4.3). And Orders & Payments shipped a small patch to its PSP client library on Thursday morning. The patch touched money, so it sat in the first review tier of Section 4.5, read by two reviewers, one of them the domain owner, before it passed the full canary schedule of Figure 4.3 in about 45 minutes.

The last purchase-path deploy finished its canary at 16:10 on Thursday, which left the new code a full evening and night of ordinary traffic before the freeze. That margin was deliberate. Canaries do not run during on-sales, because a surge of 500× leaves no stable baseline to compare against and no time to bake (Section 4.4), so whatever was going to be on the purchase path at 10:00 had to have been watched before then, on traffic that could still tell a bad build from a busy minute.

Elsewhere at Encore, change continued at its usual pace. About 180 production deploys land on a working day, across 40 teams, and Friday would be a working day for every team whose service was not on the frozen list. That is as it should be. A freeze is an expensive instrument, Section 4.4 argued, and the evidence favors a narrow one, aimed at the changes most likely to do harm, over a broad one that stops everything and then releases the backlog in a flood.

The freeze as written

The second half of the freeze line is the freeze itself, and Figure 9.3 shows it as the review read it.

Figure 9.3 — The freeze for Friday, as the review read it. Each row has a sound reason behind it, and the list of frozen services was written some years before this review.

Every row of the table has a sensible argument behind it, and the people who made those arguments were not careless. Certificate rotation must run whatever the calendar says, because credentials that expire on their own are only safe if they are renewed on their own (Section 7.2). The reconciler’s exemption we have just discussed. Catalog’s absence follows from what Catalog was when the list was drawn up, the service behind search and listings, and a fan can indeed buy a seat without searching for one. The list had been amended when a new service joined the purchase path by name, and not otherwise. Since it was written, the seat map had come to draw its geometry from a cache held in Catalog’s pods, and nothing in the way the list was kept would have prompted anyone to ask whether that changed Catalog’s place.

The last row is the one Section 4.4 counted. In the twelve months before this review, the on-sale channel had granted 41 exceptions, rising quarter by quarter from 3 to 11, 13 and 14, every one of them judged on its merits by whoever was senior and awake, and none of them followed by an incident. No one had added them up. The review did not add them up either, and it had no reason to: the review concerns one sale, and the exceptions belonged to other sales, each one closed and forgotten when its sale ended. Section 4.4 also noted the subtler weakness in a freeze drawn this way. Cloudflare’s outage of November 2025 began with a change to a database that the broken proxy depended on, and a freeze written as a list of protected services, with automation exempt, would have let that change through. As Section 4.4 concluded, a freeze drawn around services leaves open whatever those services depend on.

None of this was raised in the room, and I do not think it should surprise us that it was not. A readiness review is good at examining the things in front of it. The freeze’s boundary is not in front of it; the boundary is the frame within which everything else is examined, and a frame is the hardest part of a picture to see. The review asked whether the freeze would be in force from 08:00, confirmed that the pipeline would enforce it, confirmed that the standing exemptions were unchanged, and ticked the line.

Recap. Friday’s preparation is itself a governor, curtain-up, and the review checked its plan by asking who owns each field and confirming that intent reaches the owning loop through Git. Changes to the purchase path, configuration and flags included, traveled the rings and finished their canaries a night before the freeze. The freeze is narrow, as the evidence advises, and every row of it has a sound reason; but an exemption removes a loop from examination, a list of frozen services drifts away from the dependencies it was meant to describe, and exceptions granted one at a time are never seen as a total.

Exercise 9.3. List every loop and every route that can change your production system during your next freeze, including platform automation, certificate and secret rotation, flag changes and data pushes. For each, write down which calendar it reads, if any, and when its exemption, if it has one, was last examined. Then draw the frozen list from the dependency graph of your critical path rather than from memory, and compare the two.

9.4 Failure, Rehearsed — Stability Patterns and the Ledger

The registers and the ladder

The dependencies line of the card is Chapter 5’s, and it begins with the integration-point registers of Section 5.1, the table each service keeps of its dependencies, with the timeout, retry policy and breaker that guard each call, what the service does when the dependency is slow or down, and the date on which anyone last saw that behavior happen. The review read the registers of the frozen services. Orders & Payments’ 11 rows were all dated within the quarter, led by the two PSPs, whose slow behavior had been seen at last quarter’s tech rehearsal and whose secondary path is exercised every day by the small share of ordinary payments sent to it. Seat Inventory’s three rows were current. The card asks about the registers of the purchase-path services, and those were in good order.

The degradation ladder for payments came next, and the review walked it rung by rung, as Section 5.2 built it. The breaker on the primary PSP opens at a 40% failure rate, counting as slow any call over 600 ms, a threshold set between the PSP’s normal p99 of 450 ms and the 800 ms timeout so that a slow provider trips it as surely as a failing one. When it opens, new payment attempts move to the secondary PSP, and attempts already in flight finish where they began, because idempotency keys belong to one provider. If both providers are degraded, verified fans’ seats are held as a pending payment for up to 20 minutes instead of the usual five, capped per fan, and each fan receives a link to complete payment with strong customer authentication when a provider recovers; unpaid holds lapse and their seats return to sale. On the last rung, checkout fails fast with an honest message and the Gate slows admission. No rung issues a ticket without an authorized payment. Behind the Gate, services shed background work first and checkout last, Envoy’s adaptive concurrency filter bounds what each server holds in flight, and Seat Inventory has no breaker, relying on deadlines and the retry budget so that one sick shard cannot cut off the healthy ones.

Then the review did something Chapter 5 of the Operations book would approve of, and listed the generic mitigations, the moves that reduce harm without anyone first understanding its cause. Figure 9.4 is that list.

Figure 9.4 — The generic mitigations on the review’s page, with who may pull each one and when it was last pulled. Like every list on the card, it holds the moves Encore had needed before.

The last column is the one Section 5.1 taught us to value, and almost every row of it has a date. Two rows deserve a word. The shard failover’s time comes from last quarter’s tech rehearsal, at which Jonas, who will be Seat Inventory’s on-call on Friday, performed the failover by hand for the first time and found that the runbook named a flag removed two releases earlier; the step took 11 minutes instead of three. The runbook has since been corrected, and Jonas read through it again on Monday. And the admission dial at the top of the table is the plainest move Encore has: one setting, reversible in seconds, whose cost to fans is a slightly longer wait in a queue they already expect. It is on the list, it has an owner, and it was rehearsed on Thursday.

Thursday’s tech rehearsal

Encore’s quarterly game days are called tech rehearsals, after the theater’s practice of running every cue in sequence before opening night (Section 5.5). For its largest on-sales the Reliability team also holds a short one in the days before, aimed at the moves the sale is most likely to need. Thursday’s ran at 09:00 against the hidden test event, which was given four seat-ledger shards of its own for the occasion, with synthetic buyers arriving at 1,500 a second. It was written down in advance:

experiment: vesper-hale-ladder-and-dial
hypothesis: >
  Primary PSP breaker opens within 15 s; new payments move to the secondary;
  with both PSPs slow, holds become pending payments and links are issued;
  admission falls to 900 and returns to 1,500, each within a minute of approval;
  synthetic checkout success stays at or above 97%; no seat is sold twice.
target: test event only (tenant: encore-internal), home region EU-1
injections:
  - at: T+0     psp_latency: { psp: primary, added: 2s, share: 60% }
  - at: T+8m    psp_latency: { psp: secondary, added: 2s, share: 60% }   # both slow
  - at: T+15m   gate_admission: { event: test, rate: 900 }   # IC approves in channel
  - at: T+20m   gate_admission: { event: test, rate: 1500 }
abort_if:
  - real-event checkout SLO burn rate > 2x for 5m
  - any duplicate SeatSold for the test event
duration: 30m

Marta, on the incident commander rotation that week, played the commander in the channel, and Cue was switched off, as it is for half of all drills, so that the people would practice without waiting for an agent’s suggestion. The machines did what the hypothesis predicted. The breaker opened in 8 s and new payments moved to the secondary. When the secondary slowed too, 312 holds became pending payments and their fans’ links were issued; when the latency was removed, 290 of those payments were completed with authentication and the rest lapsed on schedule, their seats returning to sale. Marta approved the drop to 900 in the channel, the dial moved within 40 s, and synthetic errors fell within two minutes; the return to 1,500 was as quick. No seat was sold twice. Synthetic checkout success held at 98.6%.

The observers’ notes, as usual, were worth more than the confirmation. The most useful note was small. When Elif drafted the status-page message for the injected slowdown, the template told fans that some were seeing errors and when the next update would come, but said nothing about the question every fan in a queue is actually asking, which is whether to stay. She added a line saying that places in the queue were safe. A second note recorded a property of the setup. Since last quarter’s rehearsal, when Bot Screening’s velocity rule had quarantined several synthetic buyers that were retrying their rejected holds, the synthetic buyers carry a marker that the rules honor, so that rehearsals are no longer interrupted by them. The rule therefore took no part in Thursday’s rehearsal, by design. The rehearsal included no shard failover; last quarter’s had covered it, and the runbook it exposed had been fixed.

There is an old saying in the theater that a bad dress rehearsal means a good opening night. It is a superstition, and like most superstitions it carries a grain of truth inside a comforting falsehood: a rehearsal that goes badly has found things while they are cheap to fix, and a show that has found them does tend to open better. The falsehood is the implied converse, that a smooth dress promises nothing in particular, or promises trouble. A smooth rehearsal promises exactly what it tested and no more. Thursday’s showed that the payment ladder and the admission dial worked, under a commander who had now used the dial once, against synthetic fans. And there the analogy breaks in the place that matters most. A dress rehearsal runs the real show with the real cast. Encore’s rehearsal ran a test event with synthetic buyers, whose behavior is written in a script: they retry when the script says to retry, they do not refresh in frustration, they do not tell their friends, and they do not behave differently when they are afraid of missing out.

State that must survive the morning

The data line of the card is Chapter 6’s, and it is the line on which Encore’s preparation is deepest. All 12 dates are homed in EU-1, as every major on-sale before them had been. Each date’s four seat-ledger shards are consensus groups of five voting members: two in EU-1 in different zones, where the leader sits; two in EU-2; and a witness in EU-3 that stores the log and votes but serves no traffic. A seat write is acknowledged once three of the five hold it, about one cross-region round trip of 18 ms, and group commit keeps each shard’s ordered writes at its rated 1,500 a second. The losses this arrangement absorbs are the ones Section 6.4 worked through. A lost node or zone moves a shard’s leader to the other replica in EU-1, with about 40 seconds of rejected writes while the new leader is elected and fenced in, a cost the drain arithmetic of Section 9.2 had already allowed for. An isolated EU-1 loses its majority and stops writing, and the members in EU-2 and EU-3 elect leaders there, with a recovery point objective (RPO) of zero, no split brain, and a recovery time objective (RTO) of minutes. The review checked the witness’s health and the replication lag of every group, and asked Jonas what a lost node would cost on Friday. His answer was the one the design gives: about 40 seconds on one shard, and nothing worse.

The October region game day had left three findings at the edges, and the review checked each. The Gate’s pre-warm now runs in EU-2 as part of curtain-up. The Orders reconciliation job is no longer pinned to EU-1. The third finding, that Bot Screening’s risk data reaches EU-2 only by a daily copy, was still open, its ticket scheduled for the following month; Bot Screening’s lead accepted the risk for Friday by name, and the review recorded the acceptance as one of its two notes. A region failover remains, by Encore’s design, automated in its steps and human in its trigger, and the review named who on Friday would have the authority to pull it.

The rest of the line concerned recovery. The backup freshness alert of Section 6.2 was quiet for every system of record. The Finance ledger, which in its first full restore drill had taken nine hours against a two-hour RTO, now restores its current financial year first, with a warm standby behind it, and its second drill had brought that slice back in 1 hour 50 minutes; the standby approver rota named someone for Friday. The holds store, migrated to a key-value store with per-key expiry in Section 6.3, carries both the five-minute holds and the pending-payment holds of up to 20 minutes.

That left the derived data, which Section 6.1 warned is where teams are most often surprised. The seat-map geometry cache needs no backup, because its contents can always be computed again from Catalog’s data. What it needs, Section 6.1 said, is a rebuild whose duration is known. The only rebuild duration anyone had measured was curtain-up’s warm-up on a quiet morning, about 12 minutes for 12 venues with no fans asking for anything. That figure was accurate, and it was the right figure for the purpose it served, which was planning the warm-up. Whether it said anything about rebuilding the cache while a crowd was asking for it, nobody had had occasion to find out.

Recap. The review checked failure design as Chapter 5 built it: current registers for the purchase-path services, a payment ladder whose rungs were rehearsed on Thursday, and a list of generic mitigations with owners and the dates they were last pulled. The rehearsal confirmed what it tested, with synthetic fans whose behavior is scripted. The data line rests on the deepest engineering Encore has, a quorum ledger with zero RPO and rehearsed restores, while the one derived store on the opening path had a rebuild time measured only on a quiet morning.

Exercise 9.4. Write Figure 9.4 for your own system: every generic mitigation, who may pull it, how long it takes and when it was last pulled. Then design a short pre-peak rehearsal for the two moves you are most likely to need, and write down, beside its hypothesis, every way in which its synthetic users will behave differently from your real ones.

9.5 Adversaries and Assistants — and the Last Hour

Screening a crowd

The bots line of the card belongs to Chapter 7, and to the one adversary in this book that studies its opponent. The people who run bots against Encore rehearse, as Section 7.1 observed. They try their tools on minor on-sales, where little is at stake and much can be learned, and bring what works to the major ones, and an on-sale of 1.1 million fans for an artist of Vesper Hale’s standing is exactly the kind of morning they prepare for. Bot Screening tunes its rules weekly for that reason, and Noor, who will be its on-call on Friday, presented the team’s preparation.

The preparation was the one the spring dispute had produced. Two new rules, written against tools seen probing smaller sales that month, were in the bundle in shadow mode, so that on Friday they would record what they would have blocked and block nothing; if they proved sound, they would enforce at a later sale. The bundle had traveled the rings on Thursday, and the Gate’s loader, which keeps the last good bundle and, at a cold start with nothing good to load, fails open on bot scoring while still metering admission, had been exercised at the last game day. Every block now offers a challenge and a way back into the sale. The false-block rate stood at 0.4 per thousand. The rules already in force were unchanged since the last major on-sale, and the card asks about new rules, so there was nothing further to review.

It is worth pausing on how the instruments were built, because the review relied on them, and rightly. The false-block rate recognizes a person by the challenge they pass after a rule has turned them away. That makes it an excellent instrument for every rule that offers a challenge, and the arrangement after the spring dispute required every new rule to offer one. Whether every rule already in the bundle offered one was a fact about the bundle’s history rather than about its recent changes, and Section 1.3 recorded that history: among the rules in force was a velocity rule, written years earlier, that watches how fast and how often a session acts, which had never caused a dispute and so had never been reviewed or brought under the new arrangement. The review’s question did not reach it, because the question was about what had changed, and the velocity rule had not changed in years.

Two other facts sat nearby, each in a different team’s picture of the system. The fan app now retries a rejected hold for the fan, with jitter and a client-side throttle, the fix that last quarter’s rehearsal produced (Section 5.5); a retry spread out by jitter is kinder to a recovering service, and it is still a retry. And Bot Screening’s rules see sessions, not the purchase path: a rule watching how fast a session acts has no way to know whether the session is acting quickly because it is a bot or because the seats it tried to hold were refused. Neither fact came up on Wednesday, because neither was on anyone’s line.

The partner side of screening took a few minutes. Resale partners call the availability endpoints during on-sales, and since the Rowmate leak of Section 7.2 they do so with 15-minute tokens bound to a client certificate, at rates the gateway limits per partner for the duration of a major sale. The detections that caught Rowmate’s key, written against what each credential normally does, were confirmed active, with their baselines intact.

Credentials on a Friday

The rest of Chapter 7 reached the review through Security Engineering, and its preparation is a good example of an exemption that had been examined. Workload identities are SPIFFE Verifiable Identity Documents (SVIDs), certificates valid for an hour and renewed at their half-life, issued by SPIRE servers in both regions, with an alert when any certificate passes 75% of its life unrenewed. Certificate rotation is exempt from the freeze, and the review asked why, as it should. The answer was the right one: credentials that expire on their own are safe only if they are renewed on their own, and freezing their renewal would turn a security property into an outage at the end of the hour. This exemption had been examined and kept. The reconciler’s had been granted and then left alone. The difference does not lie in the exemptions themselves, both of which are reasonable. It lies in whether anyone had looked at them since.

Nothing else on the security line was due to move. No rotation of the PSP credentials or the wallet signing keys fell within two days of the sale. The break-glass path, two engineers, sixty minutes and an account outside single sign-on, had been walked at the last game day. The admission webhook for the purchase path fails closed, behind a policy engine replicated across zones. And if a critical vulnerability on the Known Exploited Vulnerabilities list arrived during the freeze, the top row of Figure 7.3 would still apply: the freeze bars non-emergency changes, a compensating rule at the edge comes first, and the 48-hour clock starts regardless of the calendar. The review recorded that Security Engineering’s on-call knew the freeze’s emergency path, and ticked the line.

The assistant’s quota

The AI line is Chapter 8’s, and it is short because Chapter 8 put the assistant’s operations into machinery. On a big on-sale morning support traffic triples within the hour, a sharp ramp of exactly the shape that providers’ rate limits punish, so Support Experience had forecast the morning’s tokens per minute, confirmed the quota for each model family with headroom, and bought reserved capacity for the morning, as Encore does for its largest sales. The fallback provider’s quota had been sized for failover load, on the understanding of Section 8.4 that the second rung is degraded by design and untested at the scale of a real provider outage.

The assistant’s behavior had been fixed in place well before the freeze. Its prompt bundle names its models by pinned snapshot, and no new bundle had shipped since Tuesday, because a bundle’s canary needs at least a day and enough conversations in every category to show a regression (Section 8.2). The golden set of 200 conversations had passed in full on Tuesday and Wednesday nights against the production configuration, each case run five times; the refund rule is explained from a template filled by the workflow’s verdict, so the model never paraphrases it; and the quality indicator on refund explanations, with its 98% objective and its 150-conversation minimum, was armed. The refund agent held no standing credentials and would hold, as always, one fan’s 15-minute token at a time, under a per-account limit on agent-initiated refunds. Support had doubled its staff for the morning, because the last rung of the assistant’s ladder ends in people, and because Support, more than any dashboard, hears what fans are actually experiencing.

What the review rested on

At 15:10 the chair read the card back from top to bottom. Every line was ticked. There were two notes: Bot Screening’s acceptance of the daily risk-data copy until the following month, and the Finance standby approver’s name for Friday. The review’s verdict, recorded in the on-sale channel, was ready. That evening the new admission rate went out through the rings.

It was a good review, and it is worth being exact about why. Nearly every line was answered with evidence, gathered in advance and attached. The arithmetic was redone at the new rate and checked by people who understood it. The load test was honest about its conditions. The ladder and the dial were rehearsed. The ledger, the backups and the restores had been proven in drills that were allowed to fail. Each team spoke for what it owned, and what each team said was true. If the purpose of this book has been to make a review like this one possible, then this book has done its job.

Now read the same afternoon a second way. Instead of listing what the review checked, list the beliefs its checks rested on: the things each answer took for granted in order to be an answer at all. Figure 9.5 is that list, as I would draw it from this chapter.

Figure 9.5 — Six beliefs beneath the review’s ticks. Each was reasonable and each had evidence; the last column is where the evidence stopped.

None of these is a mistake in the ordinary sense, and it would be a misreading of the table to treat its last column as a list of things somebody should have done. Each row describes a judgment that competent people made for good reasons, in a context that made it sensible, and each was confirmed, sometimes many times, by mornings on which nothing went wrong. That is the shape of what the Operations book will call drift, and it will give the idea the full treatment it deserves. Here I want to draw a narrower lesson, about what engineering can and cannot settle before a peak.

Everything in this book is a way of turning a belief about a system into evidence. An SLO turns “fans are happy” into a ratio. A capacity model turns “the shards can take it” into arithmetic, and a load test turns the arithmetic into a measurement. A rehearsal turns “the breaker will open” into an observation, a restore drill turns an RTO into a stopwatch reading, and an eval turns “the assistant explains the rule correctly” into a pass rate. A readiness review gathers that evidence in one room. What it cannot do is gather evidence about the beliefs nobody has noticed holding, and those are the beliefs embedded in the frame of the review itself: in what the card asks, in what the freeze lists, in what an exemption excludes and in what a team assumes about another team’s service. Forty teams each see their own part of Encore deeply, which is how a system this size can be built at all, and the price of that depth is that each team’s picture of the others is a sketch. The review is the place where the sketches meet. It is a better place than most, and the sketches are still sketches.

And the system does not hold still while it is being reviewed. Between Wednesday afternoon and Friday morning, Encore would ship hundreds of changes outside the frozen list, its loops would act on their own schedules, its adversaries would adjust their tools, and its fans would behave as frightened, hopeful people behave, which no script reproduces. Readiness is a statement about what we have examined. The system is under no obligation to stay within it.

The decision that a review produces has an older and far graver precedent, which I mention only for its shape, since the stakes bear no comparison. In the early hours of 5 June 1944, Eisenhower, who had already postponed the Normandy landings by a day because of a storm, heard his chief meteorologist, James Stagg, forecast a short break in the weather, and gave the order to go on the 6th. He had the best forecast anyone could give him, and he still could not know how the morning would go. He also wrote a short statement, to be released if the landings failed, that took the responsibility on himself. A go decision of that kind is a commitment to operate whatever the morning brings, made by people who have prepared as well as they know how and who understand that preparing is not the same as knowing. This is what Eisenhower meant, I think, in the line that opens this chapter. The plan for Friday would meet a morning it did not describe. What the planning had left behind were people who knew the system, a dial they had turned the day before, a ledger that would not sell a seat twice whatever happened above it, a comms lead who knew to tell fans that their place was safe, and an engineer who had failed over a shard by hand.

Ten to ten

At 07:30 on Friday, Rui opened curtain-up’s plan for the 12 events. He had been woken twice in the small hours that week by pre-warm alerts, the kind that fire around the preparations for a scheduled sale, and he read the plan through twice before he pressed go. By 07:55 the report was in the on-sale channel: the Gate’s instances provisioned and confirmed in both regions; the floors set through Git; the geometry cache warm for all 12 venues, with a hit ratio of 99%. At 08:00 the freeze began. The pipeline began refusing deploys to the four frozen services, and the control plane withdrew Cue’s write scopes on the purchase path.

Through the next two hours the waiting room filled. Fans arrived from every country on the tour, each given a place and a page that cost almost nothing to serve. The synthetic buyers walked the whole purchase path against the hidden test event once a minute, and every one of them bought its ticket. Bot Screening’s dashboards showed the expected probing by tools testing the edges of the Gate, and the two shadow rules began recording what they would have blocked. Support’s doubled queue was quiet. Marta, on the incident commander rotation, kept her laptop open at her desk in Orders & Payments; Elif had the status-page draft, with its new line about places in the queue, ready to publish if anyone asked her to.

Elsewhere, Encore’s Friday went on as Fridays do. Deploys continued to every service that was not on the frozen list, each through its own pipeline. In the core cluster in EU-1, the upgrade reconciler finished with one node, waited for its pods to settle elsewhere, and moved on to the next in its pool, as it had all week. In Bot Screening’s rule bundle, which had last changed on Thursday, the velocity rule sat where it had sat for years.

At 09:59 the waiting room held about 1.1 million people. The admission rate for every date read 1,500 fans per second. Jonas had the shard dashboards open, and on Rui’s screen the per-fan error ratio read zero, as it always does in the last minute before anyone has been let in.

Recap. The last lines of the card covered the adversary and the assistant: new bot rules in shadow mode, credentials whose exemption from the freeze had been examined and kept, a token quota with reserved capacity, and evals green against the production configuration. The review passed because nearly every answer was evidence. Beneath its ticks lay beliefs that the evidence did not reach, each reasonable and each confirmed by ordinary mornings, and the system kept moving after the review ended. Engineering turns beliefs into evidence; what remains after it has done so is the work of operating.

Exercise 9.5. Take the readiness review or launch checklist your organization last used and build Figure 9.5 from it: for each answer, write the belief it rested on, why that belief was reasonable, the evidence behind it and what the evidence did not reach. Choose the row whose last column worries you most, and decide who in your organization would be the first to notice, on the day, if that belief turned out false.

Kata: Midnight Doors

Your brief: “Tessera.” Tessera is a marketplace that sells to 38 million customers in 14 EU countries on behalf of 120,000 independent sellers. It is built and run by 1,100 engineers in 110 teams across three cloud regions. Once a year it holds Tessera Days, a sale that opens at midnight CET and runs for 36 hours, and its first 15 minutes are the hardest of the year: traffic rises 60× within ten minutes of midnight, and 5,000 “doorbuster” lots of limited stock, mostly game consoles, phones and limited-edition trainers, sell out in under a minute each. Every doorbuster unit must be sold exactly once, and reseller bots arrive before the customers do. Checkout depends on two payment service providers and a buy-now-pay-later provider, each with an agreed rate. A shopping assistant answers customers’ questions, and a pricing agent adjusts sellers’ prices within limits each seller sets.

Last year’s Tessera Days met its own event objective at 97.1% against 99%. Three things went wrong in the first half hour. A third-party recommendation widget’s configuration, pushed by its vendor at 00:03, doubled the weight of the product pages. The buy-now-pay-later provider rate-limited Tessera at 00:06, because nobody had sent it a forecast. And the pricing agent, misreading a supplier feed, cut the price of 300 lots to a tenth of their value before a person noticed at 00:40. Tessera runs a code freeze from 22:00, which recorded 63 exceptions over the last year’s events, all granted in chat; its platform automation is exempt as low-risk. Its readiness review is a spreadsheet of 140 lines that every team marks green the week before.

Tessera’s brief is Encore’s Wednesday at three times the size, and the comparison is the way in. Its doorbusters are seat-like: a scarce thing sold under a crowd’s pressure that must never be sold twice. Its widget configuration traveled the road that configuration and data travel in Section 4.3, outside the rings that its code obeys. Its pricing agent is an agent with permissions, whose blast radius is the subject of Section 8.5. And its spreadsheet of 140 green lines is a card that has grown long enough to be read without being thought about. The deliverables ask you to prepare Tessera Days the way this chapter prepared Vesper Hale, and then to do the one thing Encore’s review did not: write down what your preparation rests on.

Deliverables:

1. Event objectives and budgets — Write the event-window objective for the first hour after midnight, counted per customer, and turn its budget into a number of people. State the 28-day budgets of the journeys it touches, what Tessera’s error budget policy allows each team to ship in the week before, and who will watch which indicator at 00:00, with the page that stands behind them.
2. Capacity, test and cost model — Choose the unit of capacity that binds for doorbusters, derive the admission or rate limit that keeps it at a stated margin, and design the load test that measures the per-customer demand behind your arithmetic. State the test’s conditions, including cache states and third-party page weight, and confirm the result at production scale. Send forecasts to every provider with an agreed rate, and price the night.
3. Change and control-plane census — List every route and every loop that can change production between 22:00 and the end of the first day: code, configuration, vendor pushes, flags, data feeds, platform automation and agents. For each, say which calendar it reads, whether the freeze covers it, and when any exemption was last examined. Draw the frozen scope from the dependency graph of the doorbuster path, and replace the chat with a register of exceptions that shows its running total.
4. Failure and recovery plan — Write the integration-point registers for the doorbuster path with the date each failure behavior was last observed, the degradation ladder for the three payment routes, and the list of generic mitigations with who may pull each one. Design a rehearsal for the night before, and state how its synthetic customers differ from real ones. Show what keeps a doorbuster unit from being sold twice through a failover.
5. Adversary and agent plan — Review every bot rule in force, old ones included, against your challenge and appeal policy, and say what each rule can see of the checkout path’s errors. Bound the pricing agent: its credential, the largest change one session can make, the number of sessions it can run in an hour, and the person who approves anything outside those limits. Size the shopping assistant’s quota and reserved capacity for the ramp after midnight, and state its evals’ state on the night before.
6. The beliefs beneath the green — Build Figure 9.5 for Tessera from your own deliverables: each belief your plan rests on, why it is reasonable, its evidence and what the evidence does not reach. Then write the go decision as it would be recorded, with the names of the people who will operate the first hour and what each of them will watch.

Rubric:

Where you now stand

You have come to the end of this book, and it is worth taking stock of what you can now do. You can treat reliability as a product decision, set objectives that people would recognize as describing their own experience, and turn an error budget into an agreement teams will honor. You can reason about capacity in the unit that binds, test it honestly and read its cost beside it. You can see a runtime as a crowd of governors, give each field one owner and keep loops from fighting. You can ship change in small steps through rings and canaries, configuration and data included, and scope a freeze by what it protects. You can design for failure with budgets, breakers, shedding and bulkheads, and rehearse it until the rehearsal teaches you something. You can keep state recoverable and prove it with a stopwatch, operate security as routine work, and run models and agents whose words and actions you have bounded. And you can gather all of it into a readiness review that answers with evidence. That is the engineering of a resilient system, and Encore’s Wednesday shows how much of a morning it can secure in advance. It also shows what it cannot secure: the beliefs beneath the evidence, the changes that arrive after the review, and the hour in which a crowd meets the system for real. That hour belongs to the people who notice, coordinate, decide and learn while the system is running, and their work is the subject of the Operations book, Operating Production Systems — When Production Fails. Its fifth chapter opens Encore’s Gate at 10:00.

References

• Atul Gawande — The Checklist Manifesto: How to Get Things Right. Metropolitan Books, 2009.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering, ch. 27, “Reliable Product Launches at Scale”. O’Reilly/Google, 2016 — free online.
• Betsy Beyer, Niall Richard Murphy, David K. Rensin, Kent Kawahara, Stephen Thorne (eds.) — The Site Reliability Workbook. O’Reilly/Google, 2018 — free online.
• Alex Hidalgo — Implementing Service Level Objectives. O’Reilly, 2020.
• Arun Kejariwal, John Allspaw — The Art of Capacity Planning, 2nd ed.. O’Reilly, 2017.
• Amazon Web Services — AWS Infrastructure Event Management. AWS — free online.
• J.R. Storment, Mike Fuller — Cloud FinOps, 2nd ed.. O’Reilly, 2023.
• Argo CD — Sync Windows. Documentation — free online.
• Jez Humble, David Farley — Continuous Delivery. Addison-Wesley, 2010.
• Michael T. Nygard — Release It!, 2nd ed.. Pragmatic Bookshelf, 2018.
• Martin Kleppmann, Chris Riccomini — Designing Data-Intensive Applications, 2nd ed.. O’Reilly, 2026.
• Heather Adkins, Betsy Beyer, Paul Blankinship, Piotr Lewandowski, Ana Oprea, Adam Stubblefield — Building Secure and Reliable Systems. O’Reilly/Google, 2020 — free online.
• Chip Huyen — AI Engineering. O’Reilly, 2025.
• Shreya Shankar, Hamel Husain — Evals for AI Engineers. O’Reilly, 2026.