Running Production Systems in the AI & Cloud Era

← the whole book on one page

Chapter 1: What Production Is — Running Software in the Cloud Era

Code is finished when it ships. Production never is.

On the morning after Encore’s new Notification service went live, nothing happened, and that was the first thing worth noticing. Overnight a coding agent had rebuilt the service on a new framework, with better retry semantics and 40 files that no person had typed or read in full. The contract tests passed, the boundary rules passed, the security checks passed, and the canary, judged against its service level objective (SLO), promoted the new version. By breakfast the service had delivered about 180,000 tickets to fans’ phone wallets and email inboxes, and every dashboard was green.

One engineer had nonetheless spent 20 minutes on that change before it merged, reading the behavior it promised more than the code, and had found that the new retry semantics could, under the wrong timing, send a fan the same ticket twice. No automated check would have noticed, because a duplicate ticket is a perfectly healthy delivery, twice. She added a specification-level idempotency test, and the change went out. The green dashboards raise this book’s question, which is older than it looks. Who runs code that nobody fully read, and what do they need from it at 3 a.m.?

People have always run code they did not write, and how to do it well without understanding all of it is the founding question of our discipline. This chapter traces how it was answered, from the machine-room operators of the 1950s to Google’s first site reliability engineering (SRE) team in 2003, then sets out what users feel of production, the stack of practices we run it with, and the principles that make a system operable by strangers.

Encore, the System We Will Run

Encore is our companion again, as it was throughout the Architecture book, which followed it from nine engineers to 400. For most of the week it could pass for an ordinary online shop, selling seats at concerts across Europe, in euros. Then a famous artist’s tour goes on sale, nearly always at 10:00 on a Friday, and within minutes the traffic reaches 500 times its weekday level. A million people want a few thousand seats; every seat must be sold exactly once, never lost and never sold twice in a race; and the bots arrive before the fans do, because a reseller’s scripts are faster and more numerous than any person. Nearly every structure in Encore exists to answer one of those demands, and most of this book’s trouble lies where those demands conflict.

On such a morning the first problem is the size of the crowd, and only then the selling of seats, so Encore is built to manage the crowd first. Figure 1.1 is the map, and the part of it that matters most is short. The fan’s purchase path begins at the Sale Gate, Encore’s virtual waiting room, where fans meet a queue instead of a seat map and are let through at a metered rate, about 1,200 a second per event, chosen so that nothing behind the Gate runs hot. That makes the Gate an admission controller, a component that protects a scarce resource by deciding how much work may reach it, and its rate is the most powerful single control Encore has. Alongside it, Bot Screening judges, session by session, whether it faces a fan or a script. Past them, Seat Inventory is the one authority on whether a seat is free: it holds a fan’s chosen seats for five minutes and sells each one once, whatever else is failing. Orders & Payments turns the hold into a sale within a 3 s checkout budget, calling an external payment service provider (PSP), with a second provider ready to take new payments if the first falters. The PSP’s slowness once took the whole site down; the Architecture book told that story, and Chapter 2 reads it again.

Figure 1.1 — Encore’s system map. The purchase path runs down the middle; everything else either hangs off it or listens to it.

Figure 1.2 follows one fan along that path, from the queue to a ticket in the phone wallet.

Figure 1.2 — One fan’s purchase in eight steps, each tagged with the service that owns it. The admission rate and the checkout budget bound the rush from opposite ends.

Forty teams build these services and carry their own pagers, standing on a platform whose golden paths take a new service from template to production, and Figure 1.3 shows where it all runs. Within the platform group sits the Reliability team, ten engineers who own the tooling for service level objectives, the incident process and the readiness review held before every big on-sale. They advise, embed and rehearse; they do not run other teams’ services.

Figure 1.3 — Where Encore runs: two EU regions of Kubernetes cells and serverless edges, reconciled from one Git repository. The seat ledger commits each write to a majority of five replicas in three locations, two in each region and a witness in a third, which is why losing a region loses no sale.

One Friday runs through this book: the on-sale for Vesper Hale, a fictional artist whose 12-date European tour will bring about 1.1 million fans to the Gate in the same minute. Chapter 12 fights it and Chapter 13 learns from it, and earlier chapters point out small, reasonable decisions that will matter that morning. Everything else in Encore, from its tenants and cells to the change freeze and the assistant in the incident channel, is introduced in the chapter where it first does something, and the appendix keeps a glossary for the reader who wants it all on one page.

1.1 Production as a Sociotechnical System

What we mean by production

The word production comes from manufacturing, where it names the floor on which real goods are made, as distinct from the laboratory. Production is the environment in which software serves real users with real consequences, where a mistake costs money, trust or someone’s evening, and where nothing can be undone merely by closing a laptop.

The theatre teaches something the factory does not. A script is written once, but the production is made again every night by a crew that adjusts to a cold house or a missed cue, and a good stage manager keeps the show running through small failures the audience never sees. The analogy breaks at scale: a crew can see the whole stage, whereas Encore runs thousands of copies of its services for millions of people it never sees, and the gap between what is happening and what anyone can perceive is where most of this book’s techniques live.

People and machines, jointly

The term sociotechnical system comes from the Tavistock Institute in London, where Eric Trist and Ken Bamforth, studying British coal mines in 1951, found that the technically superior longwall method had made the mines worse by breaking apart the small, self-managing teams that kept the old method safe. Their conclusion, relearned in every industry since, was that the social and the technical parts of a work system cannot be optimized separately.

A sociotechnical system, then, is one whose behavior is produced jointly by its people and its technology, so that neither can be understood or improved without the other. Production is such a system, and its reliability is no more a property of the code than a ship’s safety is a property of its hull. It is produced continuously, by people knowing what the system is doing, changing it carefully, withstanding what goes wrong, responding when it does and learning afterwards. Those five verbs organize the rest of the book.

The Notification rewrite shows what this means. As an artifact the new service was better than the old one; as part of a sociotechnical system it was briefly worse, because its operators understood less of it. The engineer’s test moved a piece of human knowledge into a check that would outlast her attention, the oldest move in operations. In the Architecture book’s language, Encore’s 500× spike is essential complexity, whereas the gap between what a system does and what its operators perceive is usually accidental, and much of operations is the patient work of closing that gap.

Recap. Production is where software meets real consequences, a sociotechnical system whose reliability people and machines produce jointly. Much operational work closes the gap between what the system does and what its people can perceive.

Exercise 1.1. List the last five times a person, not an automated check, prevented or shortened a problem with a service you know. For each, ask whether that person’s knowledge now lives in a test, an alert or a runbook, or still only in their head.

1.2 From Operators to SRE — A Short History of Running Software

The machine room

In September 1947 someone working on the Harvard Mark II taped a moth into the machine’s logbook, beside the note “First actual case of bug being found.” The logbook is the more instructive object: running a computer has always meant recording what went wrong, and Chapter 12’s incident timelines continue that record.

Through the 1950s and 1960s, as machines like the IBM 704 and later the System/360 spread through universities, banks and government, computer operator became a job with its own shifts and skills. Most installations ran as a closed shop: programmers handed their card decks across a counter and came back hours later for the printout. Behind the counter, operators mounted tapes, watched the console and restarted failed jobs from instructions the programmers had left, often in a binder called a run book.

The operator stood where Encore’s on-call engineer stood after the rewrite, running programs they had not written and could not usefully read. What made that workable was a contract: the program declared what it needed, the run book said what to do when it stopped, and judgment covered the rest. The contract failed through stale instructions, programs written for the happy path, and improvisation at 3 a.m. while the authors slept.

Automating the operator

Much of computing’s history is the operator’s work being absorbed into software. In 1956 General Motors Research and North American Aviation wrote GM-NAA I/O for the IBM 704, one of the first operating systems, chiefly to cut the idle time between jobs. Spooling and time-sharing absorbed more of the role, and by the 1980s the work had passed to the system administrator, who tended fleets of smaller Unix machines through scripts.

Aviation went through a parallel change. Large airliners once carried a flight engineer to manage engines, fuel and electrics, until from the early 1980s new wide-body airliners, beginning with the Boeing 767 and the Airbus A310, were certified for two pilots and the engineer’s panel was folded into automation. The role dissolved into the aircraft’s design and the pilots’ training, as computer operations dissolved into operating systems, scripts and eventually the code. The analogy breaks because an airliner is certified once and software changes daily. Every era left something automation could not absorb: judgment for the failure nobody had written down.

The wall, and two ways over it

As software moved onto the web, developers were rewarded for change and operations staff for stability, and releases were thrown over the wall between them. The IT Infrastructure Library (ITIL), published by the UK government from the late 1980s, disciplined change management but often institutionalized the wall.

Two answers emerged. In 2003 Benjamin Treynor (now Ben Treynor Sloss) joined Google to run a production team of seven engineers, and staffed it with software engineers who treated operations as a software problem: automate the repetitive work, cap the manual share of each engineer’s time at half, and set the balance between change and stability by measuring it. He later put it in a line quoted ever since: “SRE is what happens when you ask a software engineer to design an operations team.” The other answer came from web companies. In 2009 John Allspaw and Paul Hammond described Flickr’s practice of deploying more than 10 times a day, and later that year Patrick Debois organized the first DevOpsDays in Ghent. DevOps tried to take down the wall, and Amazon’s “you build it, you run it”, which the Architecture book discussed, pushed the pager to the team that wrote the code.

Encore has inherited all three answers: teams run their own services, a small Reliability team works in the SRE tradition, and the incident commander rotation borrows trained engineers, most of them senior, from everywhere. The question is still the one the closed shop faced: how to run software whose authors are not in the room. With a coding agent as author they may not be anywhere. Figure 1.4 sets the lineage on one line.

Figure 1.4 — Who has stood between the code and its users, from the 1947 logbook to today.

Recap. Operators have always run code they did not write, relying on a contract with its authors and on judgment for the unwritten cases, which no era of automation has removed. SRE and DevOps are two answers to the wall between change and stability, usually combined today.

Exercise 1.2. Draw your organization’s closed-shop counter, where code passes from those who wrote it to those who run it. Who is on each side, what crosses it with the code, and what is lost in the crossing?

1.3 Reliability, Availability, and What Users Actually Feel

Definitions worth quoting

The reliability engineers of the 1950s, facing military electronics that failed at alarming rates, gave us a definition that has held up well: reliability is the probability that a system performs its intended function, under stated conditions, for a stated period. Intended function means that a system can be running and still be failing; stated conditions means that reliability on a quiet Tuesday says nothing about reliability during an on-sale; stated period means a window, never an instant.

Availability is narrower: the fraction of time, or of requests, for which the system is able to serve. Its classical steady-state form divides the mean time between failures (MTBF) by the sum of that and the mean time to repair (MTTR). Availability rises by failing less often or by recovering faster, and for modern systems, built from many parts that fail partially and often, designing for fast recovery is usually far cheaper, which is why “expect failures” was the first tenet of James Hamilton’s 2007 paper on internet-scale services. Figure 1.5 draws the arithmetic.

Figure 1.5 — Availability as mean time between failures over the sum of that and mean time to repair, with lines of equal availability. Halving recovery time raises availability as much as doubling the time between failures. Chapter 4 turns availability into targets.

The lesson concerns design; it does not make average recovery time a number to manage by, because incidents are few and their durations wildly skewed. Štěpán Davidovič showed by simulation at Google that even a genuine 10% improvement in incident durations is usually lost in the noise of MTTR, and the VOID, Courtney Nash’s public collection of thousands of incident reports, shows the same skew. We judge recovery by drills and by what each incident teaches, as Chapters 4 and 13 do.

A third property matters most to Encore. Correctness is the property that the system’s answers are right: a seat sold once, a ticket for every charged fan, each ticket delivered once. The dependability literature folds it into the definitions (Avižienis and colleagues’ 2004 taxonomy calls availability readiness for correct service), but availability as we usually measure it counts responses, recording whether the system answered while users judge what it said. The duplicate ticket would have appeared nowhere in that measure, and the dashboard would still have been green as fans found two copies of one seat in their wallets.

Vital signs and the patient

Medicine offers the right analogy. A hospital monitors pulse, temperature and blood pressure because they are cheap, continuous and correlated with health, yet a patient can have normal vital signs and be seriously unwell, and the patient’s own account is evidence too. Health checks, CPU graphs and pod counts are a system’s vital signs; what the fan feels in the queue and at checkout is the patient’s account, and surprising outages hide in the gap. The analogy breaks in our favor, since software can report its users’ experience directly, as Chapter 5 shows.

Figure 1.6 — Vital signs against the patient’s account. Each left-hand statement can be true while the right-hand one is also true.

Two features of user experience shape the rest of the book. Failure is usually partial, so the useful questions are “for whom?” and “how many?” And users feel their own latency, never the median, so in a purchase touching a dozen services a fan will often meet the slow tail of at least one, as the Architecture book’s tail-at-scale argument showed.

Recap. Reliability is the probability of doing the intended job under stated conditions over a stated period; availability is narrower, and correctness is a third property that availability as commonly measured cannot see. Design for fast recovery, but do not steer by average recovery time. Failure is partial and felt individually.

Exercise 1.3. For a user journey you own, write five rows of Figure 1.6. Which of the right-hand column could you detect today, and how?

1.4 The Modern Production Stack

Layers, and why we need them

A touring concert production is organized in layers: riggers hang the structure, a power crew brings in electricity, the sound and lighting desks run their systems, and the stage manager calls the cues, each layer with its own specialists and each depending on the ones beneath it.

Production is organized the same way, because hierarchy is how we master a whole too complex to hold at once: each layer hides detail from the ones above. The layered hierarchy has a precedent in Google’s Site Reliability Engineering (2016), whose Service Reliability Hierarchy, drawn by Mikey Dickerson, stacks monitoring, incident response, postmortems, testing and release, capacity planning, and finally development and product, since a team cannot respond to what it cannot see or learn from incidents it never handled. No standard stack exists; ours is a teaching device in that tradition.

At the bottom sits the runtime and its control plane: clusters, functions and managed services, declared as code and reconciled continuously. Above it, delivery carries change through pipelines, canaries and flags. Telemetry reports what the system is doing, and SLOs judge whether that is good enough for users. The incident process takes over when users are hurting, and learning decides what changes afterwards. Alongside runs the platform, packaging the lower layers into paved paths. Two concerns sit on no single layer: security operations, since attackers use our runtime, pipeline and credentials as readily as we do, and AI workloads, which run on all of them. Figure 1.7 lists its rows in the book’s order, which makes it a reading map.

Figure 1.7 — The production stack in the book’s reading order, not bottom to top; security and AI workloads cut across the other rows.

What the stack hides, and what it cannot

Layers help only while the interfaces between them behave as documented. A node drain in the runtime shows up as a latency spike in telemetry, an SLO burn, a page, an incident and, if we are diligent, a change to delivery. Here the touring analogy breaks: a tour has one stage manager calling every cue, whereas a cloud system’s cues are called by hundreds of control loops, none aware of all the others. No layer sees the whole, and the incident process exists partly to assemble it, as Figure 1.8 shows.

Figure 1.8 — One node drain, as each layer of the stack sees it. Every layer holds a piece of the event and none holds all of it, which is why an incident needs someone to assemble the whole.

The largest outages of recent years were of this kind: a content update to CrowdStrike’s Falcon sensor in July 2024, a policy change with blank fields in Google Cloud’s Service Control in June 2025 and a doubled bot-management file at Cloudflare in November 2025, all three told in Chapter 8, and a race in the DNS automation of AWS’s DynamoDB in October 2025, told in Chapter 3. None began with a broken machine; each began with a change to configuration or automation and crossed layers faster than anyone could follow. Modern production failure is mostly self-inflicted, and its blast radius is set by how far a change can travel before anyone sees it.

On the morning after the Notification rewrite, no layer required anyone to have read the 40 files, and none could say whether “acceptable” had been defined correctly. That took a specification of behavior, written by someone who understood what a ticket is for.

Recap. The production stack is a teaching hierarchy in the line of Google’s Service Reliability Hierarchy, with security and AI workloads cutting across it. The great outages of 2024 and 2025 each began with a change and crossed the whole stack.

Exercise 1.4. Fill in Figure 1.7’s “Encore today” column for your own system. Mark each row strong, adequate or missing, and pick the weakest row on your most important journey.

1.5 Principles for Operable Systems

A definition, and a stranger

Operators have never been able to understand everything they run. Operability is the degree to which a system can be understood, controlled and repaired in production by people who did not build it, under time pressure and with partial knowledge. Every author will one day be asleep or gone, and with coding agents the author may never have been a person at all.

None of what follows is new. Hamilton’s paper, drawn from running Microsoft’s Windows Live services, reads as a list of obligations a service has to its operators, under three tenets: expect failures, keep things simple, automate everything. AWS wrote much of the same advice into the operational excellence principles of its Well-Architected Framework: frequent, small, reversible changes, anticipating failure, learning from every operational failure. The five principles below are this book’s distillation of that tradition, and they apply equally to code written by people and by agents.

Legible. The system tells us what it is doing, in terms of its users’ experience, without anyone reading its source. Encore’s new Notification service emits a delivery record per ticket with the order and attempt number, so an escaped duplicate would show within minutes.

Steerable. The system offers controls that operators can turn safely and quickly while it runs: admission rates, feature flags, drains, load-shedding thresholds. The best controls degrade the product instead of dropping it, as Hamilton’s “big red switch” sheds non-essential work under extreme load, so that a ticket site under strain might draw a simpler seat map and keep selling. Controls are tested and access-controlled like any change.

Bounded. When the system fails, the damage stays inside a known perimeter: a cell, a shard, a tenant, a slice of traffic. The Architecture book drew these boundaries; they make a bad change survivable only if changes cross them one at a time, since a configuration pushed everywhere at once has no perimeter. A bounded system is also statically stable, in the Amazon Builders’ Library’s phrase: what is running keeps serving when a control plane or dependency fails. Chapter 9 develops both ideas.

Reversible. Every change can be undone faster than it can do serious harm, and the irreversible ones, mostly changes to data, are identified in advance and made in reversible steps. The Notification canary was safe only because the previous version could be restored in minutes.

Rehearsed. Recovery paths never exercised are untested. Hamilton insisted that rollback be tested and recovery scripts exercised in production, and Google’s Disaster Recovery Testing (DiRT) program made such exercises a yearly habit.

Ownership, which holds the principles together

The tradition insists on a sixth principle, which belongs to the organization. Every production system needs an owner, a named team that answers its pages, keeps its runbooks current and decides when its risks are acceptable. The Fan Messaging team woke that morning owning code it had not written, the operator’s oldest position. What made that tolerable was a service with the five properties, plus one person’s 20 minutes of specification.

Figure 1.9 turns the six into questions a readiness review can put to any service before launch, each answered with evidence.

Figure 1.9 — Operability as a readiness checklist. Chapter 15 returns to the review that asks these questions.

I have come to believe that this answers the question with which the chapter began. We cannot read everything we run, and we never could. We can insist that whatever we run, whoever or whatever wrote it, comes with what a stranger needs to operate it at 3 a.m. We can also insist that someone is accountable when that stranger is us. Operability is the contract between a system and the people who will run it, and like every contract it is worth exactly as much as its enforcement.

Recap. Operability is the degree to which strangers can understand, control and repair a system under pressure. Five principles distill Hamilton’s 2007 advice and AWS’s operational excellence principles, and ownership holds them together. Each holds only under its condition: tested controls, boundaries crossed one at a time, practiced rollbacks.

Exercise 1.5. Answer Figure 1.9’s questions for a service you own, with evidence, and score it from one to five on each principle. For the lowest score, describe when it would hurt at 3 a.m. and the smallest change that would raise it.

Kata: The Night Ferry

Your brief: “Tideway.” A ferry operator’s booking platform: 14 island routes, about 9,000 bookings a day. When storms cancel crossings, rebooking traffic jumps 40× within an hour, and island residents with hospital appointments cannot afford to be stranded. It has 22 developers in four teams and a three-person infrastructure team that receives releases on Thursday evenings and holds the only pager. Half of last quarter’s code came from a contractor’s coding agent; the contract ended in September. Outages are reported by port staff phoning the office. The board’s request: “fewer surprises, without slowing down.”

Deliverables:

1. Sociotechnical map — the people, automated systems and external parties in a storm rebooking, and who knows what about each.
2. Operations history — where Tideway sits in Section 1.2’s lineage, where its counter is, and what crosses it.
3. User-felt reliability — a Figure 1.6 table for storm rebooking, with at least two rows where availability looks healthy and residents are failed.
4. Stack map — a Figure 1.7 table rated strong, adequate or missing, with three layers chosen for investment.
5. Operability review — Figure 1.9 answered for the rebooking service, an owner for the contractor’s code, and a 90-day plan.

Rubric:

Where you now stand

You can now describe production as a sociotechnical system, place your organization in the history of running software, separate what a dashboard reports from what a user feels, map the production stack, and judge operability whoever wrote the code. What you lack is an account of how such systems actually fail, and Chapter 2 turns to safety science, from Three Mile Island onward, for that account.

References

• James Hamilton — On Designing and Deploying Internet-Scale Services. LISA ‘07, USENIX, 2007 — free online.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering. O’Reilly/Google, 2016 — free online.
• Amazon Web Services — Operational Excellence Pillar: Design Principles. AWS Well-Architected Framework — free online.
• Štěpán Davidovič — Incident Metrics in SRE. O’Reilly/Google, 2021 — free online.
• Courtney Nash — The VOID Report. Verica, 2021–2022 — free online.
• Algirdas Avižienis et al. — Basic Concepts and Taxonomy of Dependable and Secure Computing. IEEE Transactions on Dependable and Secure Computing, 2004 — free online.
• Becky Weiss, Mike Furr — Static Stability Using Availability Zones. Amazon Builders’ Library, 2020 — free online.
• CrowdStrike — External Technical Root Cause Analysis: Channel File 291. 2024 — free online.
• Google Cloud — Multiple GCP Products Are Experiencing Service Issues (incident report). 2025 — free online.
• Amazon Web Services — Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region. 2025 — free online.
• Matthew Prince — Cloudflare Outage on November 18, 2025. Cloudflare blog, 2025 — free online.
• E. L. Trist, K. W. Bamforth — Some Social and Psychological Consequences of the Longwall Method of Coal-Getting. Human Relations, 1951.