Building Resilient Systems in the AI & Cloud Era

← the whole book on one page

Chapter 1: Reliability as a Product Decision — SLOs, Error Budgets, and Risk

Every promise about reliability is also a promise about what we will stop doing.

How reliable should a system be? Ask a room of engineers and most will answer, “As reliable as we can make it.” The answer sounds responsible, and it is nearly always wrong. The trouble is that reliability is never free. We pay for it in money, in the attention of the people who build and run the system, and above all in the freedom to change things, because every change is a chance to break something, and so each extra degree of reliability makes change a little more expensive. A system made as reliable as possible has spent all three of those currencies on a property its users stopped noticing several nines ago. (A “nine” is one more 9 in the availability figure: 99.9% is three nines, 99.99% is four.) Once we see the cost, the question changes shape. It stops being an engineering question about how high we can climb and becomes a product question: how much unreliability can our users absorb before they would rather be somewhere else, and what will we buy with everything we save by not climbing further?

That question is older than our industry. The telephone network was the first to ask it at scale, and its answer, “five nines,” became a byword for reliability itself. We will follow the telephone engineers first, and then take up the chain that the Architecture book introduced. The chain runs from the service level indicator (SLI), a measurement of how well the service is treating its users, to the service level objective (SLO), a target for that measurement, and on to the error budget, the amount of failure the target allows. Section 1.2 and Section 1.3 define each link precisely. In this chapter we turn the chain into a working instrument: one that helps us decide what is worth measuring, settles arguments between teams on evidence rather than on who argues hardest, and wakes people at night only when it should.

Encore needs that instrument now. Today it has exactly one published objective, for the seat map, and it has 40 teams, each with opinions about what the other teams ought to promise. The Reliability team, the platform group’s ten site reliability engineers, has been asked to close that gap before the next big Friday on-sale. Their task is to give checkout, the on-sale window and every tenant an objective that users would recognize as describing something they have actually felt. (The section that follows introduces Encore for readers meeting it for the first time, with a map of where each of these pieces sits.) The rest of this chapter follows them through that work, and along the way we will see that each choice they face is also a choice about what Encore is willing to stop doing.

Encore, the System We Will Run

Encore is our companion again, as it was throughout the Architecture book, which followed the company from nine engineers to 400. Most of the week it sells concert seats across Europe, in euros, like any other online shop. Then a famous artist’s tour goes on sale, nearly always at 10:00 on a Friday, and within minutes the traffic reaches 500 times its weekday level. Three demands arrive together on that morning. The crowd is enormous, a million people for a few thousand seats. Every seat must be sold exactly once, never lost and never sold to two fans who clicked it in the same instant. And the bots arrive before the fans do. Nearly every structure in Encore answers one of those demands, and much of the engineering in this book lies where they pull against each other.

Figure 1.1 is the map. The fan’s purchase path, the chain of services a fan passes through between arriving and holding a ticket, begins at the Sale Gate, Encore’s virtual waiting room, which lets fans through at a metered rate of about 1,200 a second per event. The rate is chosen so that Seat Inventory, behind it, runs below 80% of what it can carry. That makes the Gate an admission controller: a component that protects a scarce resource by deciding how much work may reach it. A concert hall does the same with doors and ushers, and the comparison holds until we notice that an usher can see the hall filling, while the Gate cannot see the services behind it at all. Its rate is a judgment made in advance, from capacity arithmetic and load tests, and it is only as good as the conditions under which those were made. Beside it, Bot Screening judges, session by session, whether it faces a fan or a script.

Figure 1.1 — Encore’s system map. The purchase path runs down the middle; everything else either hangs off it or listens to it.

Seat Inventory is the single authority on whether a seat is free. It holds a fan’s chosen seats for five minutes while they pay, and it sells each seat once, whatever else is failing. A large event’s seats are split into four shards, each led by one elected leader that puts the shard’s writes into a single order, and a write commits only once a majority of five replicas, spread across three EU locations, has recorded it. Orders & Payments then turns the hold into a sale within a 3 s checkout budget, calling an external payment service provider (PSP). It has two providers, because a slow one once took the whole site down, as the Architecture book told. A circuit breaker sends new payments to the second provider when the first falters, and if both are degraded, verified fans’ seats are held as a pending payment for up to 20 minutes, to be paid once a provider recovers. No ticket is ever issued without an authorized payment. Figure 1.2 follows one fan along this path in the order a person meets it.

Figure 1.2 — One fan’s purchase in eight steps, each tagged with the service that owns it. The admission rate and the checkout budget bound the rush from opposite ends.

The admission rate bounds how many fans begin each second, and the checkout budget bounds how long any one of them may take. Between those two bounds lies everything this book must keep healthy, and many of the failures we will study come from something in the middle quietly breaking an assumption one of the bounds relied on.

Forty teams build these services and carry their own pagers. They stand on a platform whose golden paths, the supported routes from a service template to production, deliver everything through Git into Kubernetes cells and serverless functions across two EU regions, as Figure 1.3 shows. Venues that buy Encore as a service are housed in cells, self-contained copies of the stack that keep one tenant’s trouble away from the rest. Within the platform group sits the Reliability team, ten site reliability engineers who own the tooling for objectives, the incident process, the game days and the readiness review held on the Wednesday before each big on-sale. On the morning itself, a self-service workflow called curtain-up warms the purchase path, and a change freeze holds the path still from two hours before the sale until it ends. The Reliability team advises and rehearses but runs no one else’s services, so that understanding stays where the pagers are.

Figure 1.3 — Where Encore runs: two EU regions of Kubernetes cells and serverless edges, reconciled from one Git repository. The seat ledger commits each write to a majority of five replicas in three locations, two in each region and a witness in a third, which is why losing a region loses no sale.

One Friday sits on this book’s horizon: the on-sale for Vesper Hale, a fictional artist whose 12-date European tour will bring about 1.1 million fans to the Gate in the same minute. This book is about the engineering that prepares for such a morning, the objectives, capacity, runtime, delivery, failure design, data, security and AI practice that decide how much of it the system can withstand. Along the way it points out small, reasonable decisions, each defensible on its own day, that will matter when the morning comes. The morning itself, and what Encore learned from it, belong to the Operations book. Everything else is introduced where it first does something, and the appendix keeps the whole company on one page.

1.1 Why 100% Is the Wrong Target

The Bell System and its nines

For most of the twentieth century the Bell System ran the American telephone network as a regulated monopoly. The Bell System was a family of companies: AT&T, which operated the network; Western Electric, which built its equipment; and Bell Telephone Laboratories, which did its research and design. Its engineering culture rested on one expectation, that when a subscriber lifted the handset the dial tone would always be there. That expectation was written into its machines. The first No. 1 Electronic Switching System (ESS), in service at Succasunna, New Jersey, from 1965, carried a design objective of no more than two hours of total downtime in 40 years. Forty years is about 350,000 hours, so two hours of downtime works out to a little better than 99.999% availability, the “five nines” that later became a byword for reliability itself. The engineers met the objective by duplicating the switch’s central processor. One copy carried calls while the other stood ready, so that when a fault appeared in one processor it could be isolated and repaired while its twin kept the exchange running. It is worth noticing that the famous number was not achieved by making a single processor perfect. It was achieved by accepting that processors fail and building a structure in which one failure did not reach the subscriber.

Two details of this story teach more than the famous number does. The first is that the objective belonged to the switch, and the switch was only one link in a chain. To hear a dial tone, the subscriber also depended on the copper loop running from the exchange to the house, on the handset itself and on the weather, which could bring down a line in a storm whatever the switch was doing. A switch with five nines did not give the subscriber five nines; it gave the subscriber one very reliable link in a chain whose weaker links were elsewhere. The second detail is more surprising: the same network was deliberately engineered to fail some calls. Trunks, the circuits that connect one exchange to another, were expensive, and the number of them between two exchanges was sized to a grade of service, a target probability that a call is blocked in the busiest hour of the day. A common target was one call in a hundred. The engineers chose that figure on purpose, because building enough trunks that no call was ever blocked, even at the busiest moment of the busiest day, would have cost a fortune for capacity that sat idle the rest of the time. So the most reliability-minded engineering organization of its century accepted a small failure rate at its peak, chose it knowingly and put a price on it. That is an error budget in all but name. Chapter 2 takes up the mathematics of sizing capacity to a target like this; here the point is the attitude. The Bell engineers did not treat reliability as something to maximize. They treated it as something to choose.

What each nine costs

The nines are easy to say and hard to feel, because a percentage hides the quantity that operators actually live with, which is time. Figure 1.4 translates each level of availability into the downtime it allows, both per year and per the rolling 28-day window that Encore uses to measure its objectives.

Figure 1.4 — The nines as time. Each additional nine divides the allowance by ten; the cost of staying inside it rises by much more.

Read the right-hand column as an operator would, by asking what kind of bad day each row can absorb. At 99.9%, the allowance is 40 minutes in 28 days. That is enough for Encore to survive one bad deploy a month that takes half an hour to notice and roll back: someone is paged, looks at a dashboard, decides the new version is at fault and reverts it, and the month’s budget is bent but not broken. At 99.99%, the allowance is four minutes. Four minutes is less time than it takes a person to be paged, wake up, open a laptop and understand what they are looking at, so a human in the loop is no longer affordable. Detection and rollback must both be automatic, which means building and trusting the machinery that does them. At 99.999%, 24 seconds, even the automation must be very fast and the failure must usually be contained before most users meet it, which is roughly what the duplicated processor of the No. 1 ESS was for. This is why the caption says the cost rises by much more than tenfold. Each nine changes the kind of engineering a team must do, as well as its amount: a team moves from careful people, to automated recovery, to redundancy designed so that failure is absorbed before anyone sees it.

All of that cost would be worth paying if users could feel the difference, and mostly they cannot. Users cannot perceive reliability beyond that of their own path to us. A fan reaches Encore’s seat map through a phone, a mobile network or a home router, an internet service provider and the public internet, and each of those fails now and then for reasons that have nothing to do with Encore. This is the copper loop and the storm again, in a new form. A fan cannot tell a 99.99% seat map from a 99.999% one, as the Site Reliability Engineering book observes, because the unreliability of everything between the fan and Encore swamps the difference. The reliability we buy beyond what users can perceive is not free. It is paid for in change not made: in features delayed, in deploys held back, in engineering time spent defending a margin nobody notices. There are exceptions, and they matter. On paths where failure harms someone rather than annoys them, clinical or safety systems among them, what users can perceive is the wrong yardstick, and regulators set the floor. Encore sells concert tickets, and for its paths the yardstick holds.

Civil engineers face a version of the same choice and have long made it openly. A levee is designed against a flood of a stated size, not against every flood the river could ever produce, because a levee high enough for any flood would cost more than the town it protects. The design flood is a choice, made in public, about how much risk the town will carry and what it will spend instead. The analogy breaks in one instructive place. A river floods whatever the engineers do, but many of a software system’s failures come from the changes its own team makes. Setting the target therefore decides not only how much protection to build but also how freely the team may go on changing the thing it protects.

Seen this way, a target of 100% is not a very high standard. It is a refusal to choose. A team that promises 100% has promised something it cannot keep, so every incident becomes a breach, and there is no agreed line between an acceptable bad day and an unacceptable one. In the absence of such a line, the decisions still get made: whether to freeze releases, whether to cancel a launch, whether to add another layer of approval. They are made one incident at a time, under pressure, by whoever is most frightened in the room. I have come to believe that this is the real cost of the 100% target. It does not buy reliability; it hands the choice about risk to fear.

This is why reliability is a product decision. Choosing a target means trading something users value, a more reliable seat map, against something else users value, such as a feature fans have been asking for, and weighing one against the other is the product owner’s job. Engineers have an essential part in the decision, but it is a different part: they can say what each option costs, in engineering effort and in the freedom to change, and what each nine would demand of the system and the people who run it. Encore has made this decision once, for the seat map, whose objective allows about 40 minutes of full-outage equivalent in 28 days. The rest of this chapter is about making it well for everything else, beginning with what exactly we should be measuring.

Recap. The Bell System chose its nines, engineered to them and knowingly accepted blocked calls at the busiest hour. Each further nine changes the kind of engineering required, not merely its amount, and users cannot feel reliability beyond that of their own path to us. So the target is a product choice about where to spend the freedom to change, except on paths where failure causes harm.

Exercise 1.1. For a service you run, find the row of Figure 1.4 at which a single incident, handled by your current detection, paging and rollback, would consume the whole monthly allowance. Is your published target above or below that row?

1.2 SLIs and SLOs That Users Would Recognize

From journeys to indicators

Two definitions come first, because everything in this chapter rests on them. A service level indicator (SLI) is a measurement of service, expressed as the proportion of valid events that were good. A service level objective (SLO) is a target for that proportion over a stated window of time. Written down like that, both sound like bookkeeping, and the bookkeeping is in fact the easy part. The hard part is deciding which events to count and what to call good, and there the choices are many and most of them are subtly wrong.

The most useful test I know is a simple one. Imagine showing the SLI, in plain words, to a fan. Would the fan recognize it as something they have felt? “99.9% of requests to the Orders service returned without a server error” fails the test, because no fan has ever experienced a request to the Orders service. “Of the fans who tried to pay, 99.9% either got their tickets or were told clearly why not, within three seconds” passes it, because every fan who has bought a ticket has lived through exactly that moment. The test pushes us away from components, which are how we build the system, and toward user journeys, which are how people use it: joining the waiting room, holding seats, checking out. For each journey we then define three things: the event we count, what makes that event good, and the place where we measure it.

Checkout shows why this takes care. The naive SLI is the one engineers reach for first: count the HTTP 5xx responses from Orders and call everything else good. It gets several cases wrong, in both directions. When a fan’s bank declines the card, Orders returns a perfectly healthy response and the system has done its job, so the event is good, however unhappy the fan may be. When both of Encore’s payment service providers (PSPs), the outside companies that actually move the money, are degraded at once, and a verified fan’s seats are held pending payment, the fan has been told the truth about what is happening and has lost nothing, so that event is good as well. But a decline that Encore caused itself, by timing out while waiting on the PSP or by breaking the strong customer authentication step, is bad, even though the fan sees the same polite “payment declined” message as in the first case. The 5xx count would call it good, since no server errored. The fan’s experience and the server’s status code have quietly parted company, and only an indicator built around the journey can tell the two declines apart.

Putting those cases together gives Encore’s checkout SLO, which reads: of checkout attempts by admitted fans, 99.9% end in a confirmed order, a hold pending payment or an issuer’s decline, correctly explained, within 3 s, measured at the gateway over a rolling 28 days. Each clause does work. “Admitted fans” fixes the valid events, so that bots turned away at the door do not enter the count. The three good outcomes are the three honest endings a checkout can have. “Correctly explained” is what separates the bank’s decline from Encore’s own. And the 3 s is a budget the whole path shares; about 800 ms of it belongs to the PSP, which leaves Encore’s own services the remainder.

Ratios, thresholds and percentiles

Once we know what to count, we have to choose how to count it, and there are two broad forms. An event-based SLI, also called a ratio SLI, divides good events by valid events: of all the checkout attempts, how many were good. A time-based SLI divides time into slices and counts the good slices, such as the proportion of minutes in which the error rate stayed below 1%. The difference lies in how each one weights a busy moment. In the event-based form, a minute in which tens of thousands of fans tried to check out contributes tens of thousands of events to the ratio, while a quiet minute at 04:00 contributes a handful, so the busy minute dominates. That is usually what we want, because a failure when many people are using the service hurts many people. In the time-based form every minute counts once, however crowded it was. That suits services with little traffic, where a ratio of a few events would lurch with every failure, and it suits contracts, whose readers tend to think in minutes of downtime.

Latency is where well-meant indicators most often go astray. “p99 latency below 800 ms” and “99% of requests faster than 800 ms” sound like the same sentence said two ways, and for a single server over a single period they are. The trouble begins when we combine measurements, because percentiles cannot be averaged. Suppose one server handled 1,000 requests and another, unlucky, handled ten requests that were all slow. Their p99 values might be 300 ms and 2 s, and the mean of the two, 1.15 s, describes no request anybody made and says nothing reliable about how the 1,010 requests fared as a whole. The mean of ten servers’ p99 values is likewise not the fleet’s p99. Counts, by contrast, simply add. If 995 of the first server’s requests and none of the second’s came in under 800 ms, the fleet served 995 of 1,010 requests within the threshold, about 98.5%, and that is exactly the proportion fans experienced. The ratio form survives being summed across servers, regions and time; the percentile form does not.

Encore therefore states its latency objectives as ratios, and it records latency in histograms with a bucket boundary placed exactly at each threshold. The reason for that last detail is easy to miss. A histogram stores counts per bucket and nothing finer. If its buckets ended at 500 ms and 1 s, it could say how many requests took less than 500 ms and how many took less than 1 s, but any answer about 800 ms would be an interpolation, which is a polite word for a guess. A histogram can only answer questions about the boundaries it was given, so the boundaries have to be chosen with the objectives in hand.

Where we measure matters as much as what and how. Encore measures at the gateway, because a server timing itself cannot see the requests queued in front of it: a request that waited two seconds for a free worker and was then served in 50 ms looks fast to the server and slow to the fan. The gateway is closer to the fan and catches that queueing, but it is still not the fan. It cannot see a failed DNS lookup, a content delivery network fault or a phone that never managed to connect, since in each case the request never reached it. So the purchase journeys also carry client-side measurement from Encore’s app, which sees what the fan’s device saw. The ticket-delivery SLI goes one step further and is confirmed by the wallet’s receipt, because Notifications can only know that it sent something, never that it arrived. The general rule is to measure as close to the user as we can get, and to be honest about what each vantage point cannot see.

Windows, and the hour that matters

The third choice is the window, the stretch of time over which the proportion is computed. I find it helpful to think of the window as the SLO’s memory: it decides how long a bad day keeps counting against the service. A rolling window remembers continuously. A bad Friday weighs on the team for exactly the length of the window, and then it slides out of the back and is forgiven. Encore follows The Site Reliability Workbook in choosing four weeks, 28 days, and the choice is less arbitrary than it looks. Traffic at a ticketing company has a strong weekly rhythm, with on-sales clustered on particular mornings, and a 28-day window always holds exactly four of each weekday. A 30-day window would sometimes hold four Fridays and sometimes five, and the indicator would wobble with the calendar rather than with the service. A calendar window, such as the calendar month, suits reports and contracts, because people read and sign those by the month. But it creates strange incentives: a budget nearly spent on the 29th is restored in full on the 1st, so the same failure matters a great deal on one day and hardly at all two days later.

Fans, however, do not remember months. They remember 10:00 on the Friday their artist went on sale. A 28-day objective can stay green while that hour was a disaster, because the hour’s failures are diluted by 27 quiet days on either side. A touring musician understands this instinctively: a tour can average well across 30 nights, and the fans in the one city where the sound failed will still talk about nothing else. So for each on-sale classed as major at the readiness review the Wednesday before, Encore also evaluates an on-sale-window SLO: an objective computed over a fixed window around a scheduled event, in this case the first 60 minutes of the sale, with a budget of its own. The comparison with a tour holds only so far. A concert’s quality is judged by everyone in the room, whereas this SLO is judged by a ratio, and a ratio over an hour of very uneven traffic behaves differently from one over a month, as the next point shows.

The on-sale-window SLO is Encore’s own adaptation, and there is no published practice of that name behind it. It rests on the Workbook’s advice that a window should match how users experience the service, and its costs were accepted knowingly. There are three. First, a short window that is heavily front-loaded, with most fans arriving in the opening minutes, is statistically noisy: a few unlucky minutes can swing it, so a miss is a strong signal to investigate and a weaker one to conclude from. Second, burn-rate alerting, the usual way of turning an SLO into a page, cannot work inside a window this short, for reasons Section 1.4 sets out. Third, error budget policies are written for rolling windows, and per-event budgets have no policy precedent to lean on. So a missed on-sale-window SLO earns a postmortem and an item for the next readiness review, never a change freeze.

The on-sale SLI also counts fans rather than requests, and that choice changes the number considerably. Consider a fan who hits an error and retries ten times before the eleventh attempt goes through. Counted by requests, that fan generated ten bad requests, and would have done so whether or not the eleventh had succeeded, so a request-based indicator charges the service ten times for one frustrated person, and charges it most heavily for the fans who are most determined to keep trying. Counted by fans, that person had one bad on-sale, however many requests it took. That is what the fan will remember and what the SLO should measure. The price is a per-fan key shared by the Gate and the gateway, so that the admission the Gate granted and the requests the gateway served can be joined into one journey. It is harder to build than it sounds, because the two systems see the fan at different moments and for different purposes, and both must agree on who that fan is throughout the hour.

The table below gathers the objectives this section has argued for. The “shop once admitted” and major on-sale rows share one indicator, judged over two different windows.

Each objective in the table is a product decision, signed by the journey’s product owner, for the reason Section 1.1 gave: only the product owner can weigh a little more reliability against the features it would cost. Each began near what the journey had actually delivered in the previous quarter, so that the first target described a service the team already knew how to run, and each is reviewed quarterly. That review matters more than it seems. Users’ expectations move, traffic grows and the system changes beneath the number, and an objective nobody revisits slowly drifts into irrelevance, still green on the dashboard long after it has stopped describing anything a fan would recognize.

Recap. Good SLIs start from user journeys and define three things: the event, what makes it good, and a place of measurement as close to the user as we can get. Ratios add up correctly across servers and time, while percentiles do not, so latency belongs in ratios backed by histograms with boundaries at the thresholds. Rolling windows give an SLO a steady memory, and Encore gives its scheduled peaks a window shaped like the event, counted per fan, knowing the costs of doing so.

Exercise 1.2. Take one SLI you have. Find a case it counts as good that the user felt as bad, and a case it counts as bad that nobody noticed. Rewrite the SLI so it counts both correctly, and say where you would have to measure to make the rewrite possible.

1.3 Error Budgets as a Negotiation Instrument

A contract written in calm weather

Section 1.2 gave each of Encore’s journeys an objective. Every objective short of 100% carries a second number with it, the failure it leaves room for, and that number is where reliability stops being a matter of measurement and becomes a matter of negotiation. Two definitions will carry us through the section. An error budget is the amount of unreliability an SLO permits over its window: for a 99.9% objective, the 0.1% of events that may go wrong before the objective is missed. An error budget policy is the written agreement about what the teams involved will do as that budget is spent. The budget is arithmetic, and any dashboard can compute it. The policy is something else entirely: an agreement among people with different interests about how to share something scarce, and like every such agreement it is only as good as the willingness of the people who made it.

The policy’s value lies in its timing. Every engineering organization eventually has the argument about whether to keep shipping features or stop and stabilize, and left to itself that argument happens at the worst possible moment, during or just after an outage, when everyone is tired, someone is angry and the loudest voice tends to win. A policy moves the argument to a quiet Tuesday, when nobody’s work is on fire and the people at the table can still think about the general case rather than the particular wound. Touring productions learned the same lesson long ago: what happens if the headliner falls ill, or the venue loses power, is settled in the contract before the tour leaves, because nobody negotiates well at the stage door an hour before the show. The analogy holds as far as the timing and breaks at enforcement, since a tour contract has lawyers behind it and an error budget policy has only the goodwill and authority of the people who signed it, a point we return to in a moment.

Encore’s purchase-path policy is taken nearly unchanged from the example in The Site Reliability Workbook. It has three clauses from that example. While budget remains, teams ship at will. Once the budget is exhausted, only reliability work, security fixes and risk-reducing changes ship until it recovers. And any incident that consumes more than 20% of a budget requires a postmortem. Between the first two clauses Encore added a tier of its own: when less than 25% of the budget remains, every change to the journey is also reviewed, in the same pull request, by the site reliability engineer embedded with the team. The form of that review is deliberate. DORA’s research found that approval by a body outside the team, such as a change advisory board, slows delivery without making it safer. So the embedded engineer does not act as an outside approver. They review as a peer who knows how the journey tends to fail, they can point out the risky line in a change, and they hold no gate that could stop it from shipping.

A policy like this holds only because people with authority over both kinds of work have signed it. At Encore those people are the heads of engineering and product: one decides how much engineering effort goes into reliability, the other decides which features fans will get and when, and a freeze on feature work touches both. Without their signatures the policy is a suggestion, and suggestions give way the first time a launch date is at stake. Many organizations publish SLOs and never enforce any policy about them, and in those organizations the targets become decoration, numbers on a dashboard that nobody’s behavior depends on. Encore also follows the advice of Alex Hidalgo, in Implementing Service Level Objectives, that a policy should guide decisions rather than freeze teams mechanically. The thresholds tell people when to stop and talk; they are not meant to replace the talking.

There is a second half to the idea, and it is the half engineers find harder to accept: budgets are meant to be spent. The usual way to spend one is on faster change, shipping more often and taking more risks with each release, because the budget exists precisely so that the organization can buy speed with a known, bounded amount of failure. The Site Reliability Engineering book tells of Google’s Chubby lock service, which was so reliable in practice that its users came to depend on more availability than it promised. The service’s owners responded by taking it down with planned outages until its real availability matched its objective, so that the services depending on it would discover their hidden assumption on a planned day rather than an unplanned one. The lesson generalizes. Unspent budget is reliability nobody promised, and sooner or later someone will build on it as though it were a promise.

The idea underneath the budget is older than site reliability engineering. In the 1920s Walter Shewhart, working for the Bell System, devised control charts, which told managers when the variation in a manufacturing process was simply the process behaving as designed and when it had a cause worth hunting. An error budget makes the same distinction for a service. Failures that stay within the budget are the ordinary variation we agreed to accept, and chasing each one would cost more than it returns. Failures that burn the budget quickly are a signal that something has changed and deserves attention. What the budget does not settle on its own, as Encore was about to discover, is whose attention it deserves when the change that burned it belonged to someone else.

The dispute between On-Sale and Bot Screening

Encore’s policy met its first real test in the spring. Two teams were involved. The On-Sale team owns the waiting room and the journey of a fan who has been admitted to the sale. Bot Screening writes the rules that keep bots out of the waiting room and that continue to watch sessions after they are admitted. That season Bot Screening switched on a new device-fingerprint rule during a mid-size arena on-sale, one the readiness review had classed as major. The rule ran at the hold step, where a fan places a hold on seats before paying, which is after admission. In 15 minutes it blocked about 6,000 admitted fans. Most of them, as Support’s inbox later showed, were real people using an older phone browser whose fingerprint the rule mistook for a bot’s.

The cost showed up in On-Sale’s numbers rather than Bot Screening’s, and the reason is the way the indicator was written. The “shop once admitted” indicator, as it stood then, counted every journey that ended on a block page as bad, with no way to tell a blocked bot from a blocked fan. About 200,000 fans were admitted in the first hour of that sale. The 6,000 blocks alone were 3% of them, and together with the hour’s ordinary errors they brought the on-sale-window SLO in at 96.8% against its objective of 99%. The damage did not stay inside the hour. The 28-day objective for the same journey is 99.9%, and a typical four weeks admits about 8.6 million fans, so that objective allows some 8,600 bad journeys in its window. The 6,000 blocked journeys took 70% of that allowance in a quarter of an hour, which dropped the remaining budget below 25% and put the On-Sale team into the review tier. Figure 1.5 shows the burn.

Figure 1.5 — The On-Sale team’s 28-day error budget through the spring on-sale. Ordinary spending keeps the budget well above the steady burn line; one rule from another team takes 70% in 15 minutes and drops the budget below the policy’s 25% line.

Both teams had a fair case, which is what made the dispute instructive rather than merely awkward. On-Sale argued that a real fan blocked in the middle of a purchase is the worst error the system can make, worse than a slow page or even a failed one, and that it had now paid for someone else’s change with its own freedom to ship. Bot Screening argued that a block is the system doing its job, and that an indicator which charged it for every bot it caught would, over time, teach it to let bots through. Both were right, and the reason they could both be right lies in the shape of Bot Screening’s work. Its mistakes come in two kinds that are never equally visible. A bot let through shows up quickly and loudly, because the resale market reports it within the hour as tickets appear on secondary sites. A fan wrongly turned away is almost silent, because only the fan can report it, and many fans simply give up instead. A team that sees one kind of error clearly and the other dimly will drift, quite reasonably, toward the error it cannot see.

The Reliability team brokered the resolution, and it began with the question of what the indicator should count. The Workbook’s example policy already allows out-of-scope traffic, such as load tests and abusive clients, to be excluded from the budget. That clause is easy to write and hard to apply, because it does not say how to tell an abusive client from a customer. Encore gave the clause an operational test. Every block now offers a challenge and a way back into the sale. A blocked session that fails the challenge or abandons it is treated as bot traffic, and it leaves the journey’s valid events altogether, neither good nor bad. A blocked session that passes the challenge is a false block: a person the rules turned away. A false block counts as bad even though the fan got back in, since the fan had already lost the seats they were holding, whatever happened next. Applied to the spring, the test did not let anyone off. Most of the 6,000 blocked fans were real people and therefore false blocks, so the burn stood.

Once the indicator counted the right thing, the rest of the settlement followed. First, the policy now attributes consumption: the review tier applies to the team whose change caused the burn, so it was Bot Screening, not On-Sale, that went under review. That goes a step beyond the Workbook, whose example only spares a team whose budget was spent by another team’s outage and says nothing about putting the cause under review. The step carries a risk worth naming. Attribution can drift into blame, and blame assumes a single culprit, when, as the Operations book shows in its chapter on learning from incidents, a burn often has more than one cause. Encore uses it with care for that reason.

Second, Bot Screening gained an indicator of its own, the false-block rate: false blocks divided by admitted sessions screened, with an objective of no more than one in a thousand. This gives the team a number that rises when it errs toward caution, to set against the resale market’s loud report of the bots that got through, so that both kinds of mistake are finally visible to it. The rate is a proxy, and it errs in both directions. Solver farms, services that pay people or software to pass challenges, let some bots pass and be counted as false blocks, which inflates the rate; and some real fans give up rather than attempt the challenge, which hides them among the bots and deflates it. So the team checks the measure every month against a sample of Support complaints. Cloudflare offers the same measure as a “challenge solve rate” for tuning rules; giving it an objective is Encore’s own step.

Third, new rules now run in shadow mode through one on-sale before they enforce anything. In shadow mode a rule logs what it would have blocked without blocking it, so the team can count its would-be false blocks before any fan pays for them. This is an established practice, which AWS WAF calls Count and Cloudflare calls Log.

Nobody thought to ask the same of the rules already in force. Among them was a velocity rule, written years earlier, that watches how fast and how often a session acts. It had never caused a dispute, so it was never reviewed, and it was never brought under the new arrangement of a challenge and a way back. That omission will matter on the morning of the Vesper Hale on-sale, which the Operations book follows minute by minute in its Chapter 5.

Recap. An error budget policy is an agreement, written in calm weather and signed by people with authority over both features and reliability, about how teams share a scarce allowance of failure, and the allowance is meant to be spent. When a dependency spends another team’s budget, the answer has three parts: a clear definition of what counts, attribution of the burn to its cause, used with care, and an indicator that makes the dependency’s own errors visible to it.

Exercise 1.3. Draft a one-page error budget policy for a journey you own. State the thresholds, what changes at each one, who grants exceptions, which traffic is out of scope and how you would tell it apart, and how a burn caused by another team is attributed. Then ask that team to read it.

1.4 Alerting on Burn Rate

Fuel flow and the fuel gauge

The oldest question in operations is when to wake a human. For most of the field’s history the answer was a threshold on some cause: page when CPU passes 90%, when a disk fills, when a queue grows long. Such alarms are easy to write and hard to live with, because a busy CPU may harm no one while a quiet one may be serving errors to every fan in the queue. An SLO offers a more principled answer, because it lets us page on the thing users actually feel: wake someone when the error budget is burning fast enough to matter. To make “fast enough” precise we need one more term. The burn rate is the speed at which a service consumes its error budget, measured relative to the speed that would use the budget up exactly at the end of the window. A burn rate of 1 spends the whole budget in exactly one window; a burn rate of 10 spends it in a tenth of the window; a burn rate below 1 means the service is living within its means.

Pilots learn early to tell level from flow. The fuel gauge shows how much remains, while the fuel-flow indicator shows how fast it is going, and a leak shows up in the flow long before the gauge looks alarming. The two readings answer different questions, and an error budget has the same pair. Budget remaining is the gauge, the number the error budget policy of Section 1.3 acts on over days and weeks. Burn rate is the flow, the number an alert should act on within minutes. Where the comparison fails is just as telling: an engine burns fuel steadily, while errors arrive in bursts, so any single reading of burn rate is noisy. A five-minute spike can look like a catastrophe and be nothing at all; a slow leak can look like nothing and quietly drain a month. The Site Reliability Workbook works through simpler rules first, and finds that each one either pages too often or pages too late. It arrives at multi-window, multi-burn-rate alerting: page only when the budget is burning fast over a long window and is still burning over a short one. Two windows, read together, smooth the bursts without slowing the response, and the next subsection shows how.

The arithmetic

The Workbook expresses its rules in terms of budget consumed, which is the quantity an on-call engineer cares about, and converts each one to a burn rate, which is the quantity a monitoring system can measure. For a 30-day window, which holds 720 hours, it suggests three rules:

• page if 2% of the budget is consumed in one hour. That is a burn rate of 0.02 × 720 h ÷ 1 h, or 14.4; left alone, the service would exhaust its whole budget in 50 hours;
• page again if 5% is consumed in six hours, a burn rate of 6;
• open a ticket rather than page if 10% is consumed in three days, a burn rate of 1.

The pattern behind the three rules is a trade between urgency and confidence. A fast burn is dangerous and obvious, so a short look suffices; a slow burn is less urgent but harder to distinguish from noise, so it earns a longer look and a gentler response.

Each rule then pairs its long window with a short one, by convention a twelfth of its length, and both windows must exceed the threshold before the alert fires. The two windows do different jobs. The long window supplies significance: an hour of evidence is hard to fake with one unlucky minute. The short window supplies timeliness, and its real value shows at the end of an incident rather than the beginning. Once the bleeding stops, the long window still remembers the bad hour and stays above the line for a long time, but the short window forgets within minutes, drops below the line, and so resolves the page. Without it, the on-call engineer would fix the problem and then watch the alert keep firing for most of an hour, which teaches people to distrust their alerts.

Figure 1.6 — Multi-window, multi-burn-rate alerting, with the Workbook’s parameters for a 30-day budget.

Encore’s windows are 28 days rather than 30, but it kept the Workbook’s burn-rate multipliers, as most SLO tools do. The multipliers are what the monitoring system checks, so keeping them means keeping well-tested rules; the price is only that the budget fractions shift slightly, since 28 days hold 672 hours rather than 720. An hour at 14.4 now spends about 2.1% of the budget, and six hours at 6 about 5.4%. The next step is to turn a burn rate into a number a query can compare against. Take the seat map’s objective, 99.9% of requests within 800 ms, which leaves an error budget of 0.1% of requests. A burn rate of 1 means a bad-request ratio of exactly 0.1%; a burn rate of 14.4 means a ratio 14.4 times that, above 1.44%.

Encore’s monitoring runs on Prometheus, where a recording rule is a query evaluated on a schedule and stored as a new time series, so that expensive expressions are computed once and read cheaply many times. Encore records the seat map’s bad ratio once per window: 5 minutes, 30 minutes, 1 hour, 6 hours and 3 days. Each recording counts the requests that were good, meaning they did not return a server error and completed within the 0.8 s histogram bucket, divides by all requests, and subtracts from 1. The alert rules then compare those ratios against the thresholds:

groups:
- name: seat-map-slo   # 99.9% of requests <= 800 ms over 28 days: budget 0.001
  rules:
  - record: seatmap:bad_ratio:rate5m   # likewise for 30m, 1h, 6h and 3d
    expr: 1 - sum(rate(gateway_request_duration_seconds_bucket{route="seat-map",code!~"5..",le="0.8"}[5m]))
            / sum(rate(gateway_request_duration_seconds_count{route="seat-map"}[5m]))
  - alert: SeatMapBudgetFastBurn
    expr: (seatmap:bad_ratio:rate1h > 14.4 * 0.001 and seatmap:bad_ratio:rate5m > 14.4 * 0.001)
       or (seatmap:bad_ratio:rate6h > 6 * 0.001 and seatmap:bad_ratio:rate30m > 6 * 0.001)
    labels: {severity: page}
  - alert: SeatMapBudgetSlowBurn
    expr: seatmap:bad_ratio:rate3d > 0.001 and seatmap:bad_ratio:rate6h > 0.001
    labels: {severity: ticket}

Read the fast-burn alert as two sentences joined by “or”: the one-hour and five-minute ratios both exceed 14.4 times the budget, or the six-hour and thirty-minute ratios both exceed six times it. Either condition wakes someone. The slow-burn alert asks only whether the service has been spending faster than it can afford for three days and is still doing so now, and its answer goes to the team’s queue instead of anyone’s phone.

Nobody at Encore writes these rules by hand, and the reason is drift. A rule copied from a wiki carries the objective that was true on the day it was copied, and when the objective later changes, say from 99.9% to 99.95%, the copied rules go on enforcing the old one without complaint. So the golden path generates the rules from each service’s SLO entry in the service registry, and changing the objective changes the alerts in the same commit. Open-source generators such as Sloth and Pyrra work the same way, expanding a few declared lines into the full set of recording and alert rules. It is a small instance of a large principle: when two artifacts must agree, derive one from the other, because two copies kept in step by good intentions eventually stop agreeing.

Every alert rule is, in effect, a promise about how long a failure can go unnoticed, so the useful way to review one is to compute its detection times. Those times are the alert’s real specification. Suppose the seat map fails completely, so every request is bad. The one-hour ratio climbs by one percentage point for every 36 seconds of failure, and it crosses 1.44% after about 52 seconds; the five-minute ratio crosses sooner, so the page fires in under a minute. Now suppose a subtler fault, where one request in 50 is slow, a bad ratio of 2%. The five-minute ratio passes 1.44% within a few minutes, but the one-hour ratio rises slowly, because most of the hour it averages over came before the fault, and it crosses the line only after about 43 minutes. By then the incident has spent about 2% of the 28-day budget, which is exactly what the rule was designed to tolerate before waking anyone. Whether 43 minutes is acceptable is a product question, not a monitoring one, and that is the point of writing the number down. Figure 1.7 follows a third case, a partial outage with one request in ten failing for 40 minutes, through both windows: the page fires about nine minutes in, when the one-hour ratio crosses the line, and resolves about four minutes after the fix, when the five-minute ratio falls back below it.

Figure 1.7 — A fast-burn page through one incident. The one-hour ratio decides when the page fires and the five-minute ratio decides when it stops: without the short window, the page would outlast the fix by more than three-quarters of an hour.

Where the rules run out

Burn-rate alerting is a statistical method, and like every statistical method it needs a sample large enough to mean something. It needs traffic. At a few requests a minute, a single failure moves the ratio by whole percentage points, so one unlucky request can look like a fast burn and page someone for nothing. Section 1.5 meets the same problem again with small tenants, whose traffic is too thin for ratios to settle.

The rules also cannot protect an on-sale-window SLO, the 60-minute objective Section 1.2 introduced for major on-sales. Its target is 99% of fans served well over 60 minutes, which leaves a budget of 1% of the hour. A total failure spends that budget in 36 seconds, and sooner in practice, because admissions crowd into the first minutes of a sale and so do the bad requests. The alerting windows make things worse. The fast-burn rule’s long window is one hour, as long as the SLO window itself, so by the time an hour’s worth of evidence had accumulated the sale would be over, and the page could fire only after the budget was long gone. Shrinking the windows does not rescue the method, because over a few seconds the ratios become too noisy to trust. Here the arithmetic of burn rates simply runs out of room.

For major on-sales, therefore, Encore has people watch the signal. From the moment the Gate opens, the owning teams and an incident commander watch one-minute per-fan error ratios, the share of fans who hit an error, counted once per fan as Section 1.2 described. A page stays armed all the same, because attention fades over a long sale and trouble can begin in a component nobody happens to be watching. That page uses a rule rescaled to the hour: it fires at the On-Sale on-call when both the one-minute and the five-minute per-fan error ratios exceed 10%. Ten percent is ten times the hour’s 1% allowance, a burn rate of 10, at which the whole on-sale budget would be gone in six minutes. Google’s practice for launches, as the SRE book describes it, is the same arrangement: people watching closely, with the alerts still switched on. The two are complements. People notice what no rule anticipated, and rules keep noticing after people have stopped.

Recap. Burn rate measures how fast the error budget is being spent, and it tells us when to act long before the budget remaining looks alarming. Pairing a long window with a short one gives each alert both significance and timeliness: the long window keeps noise from paging, and the short one lets the page end when the incident does. Computing detection times turns an alert rule into a specification that a team can argue about, and generating the rules from the SLO keeps them faithful to it as it changes. Where the window is too short or the traffic too thin for the statistics to work, as in an on-sale hour, people watch, with a rescaled page as the backstop.

Exercise 1.4. For one SLO you own, compute the thresholds for the three rows of Figure 1.6, expressed as bad-request ratios for your objective. Then work out how long a total outage and a 5% partial outage would each take to page. Would your users accept those times?

1.5 SLOs Across Dependencies and Teams

Composition only subtracts

So far we have treated each SLO as if it belonged to one service standing alone, but no service meets its SLO alone. When a fan checks out, the request passes through the Gate, Seat Inventory, Orders & Payments and the payment service provider, and if any one of them fails, the fan’s checkout fails with it. We met the same truth at the start of the chapter: the Bell System’s famous objective belonged to the switch, while the subscriber also depended on the copper loop, the handset and the weather. The paper that gives this problem its clearest modern treatment, “The Calculus of Service Availability” (Ben Treynor and co-authors, 2017), states the hard limit plainly: a service can be no more available than its least available critical dependency.

The arithmetic behind that limit depends on how the dependencies fail. If they fail independently, so that one being down tells us nothing about whether another is down, then the chance that all of them are up at a given moment is the product of their separate availabilities. Four dependencies at 99.95% each give 0.9995 × 0.9995 × 0.9995 × 0.9995, which is about 99.80%. That is already below checkout’s 99.9% objective, before Orders has written a line of its own code or made a single mistake of its own. To feel what the gap means, translate it into time: over Encore’s 28-day window, 99.9% allows about 40 minutes of failure, while 99.80% means about 80 minutes. The dependencies alone would spend twice the budget.

If, on the other hand, the dependencies’ failures coincide, because they share a database, a network or a region, then their bad minutes overlap. When the shared database goes down, all four fail during the same minutes, so the journey loses those minutes once rather than four times. The composite is then better than the product, but it can never be better than the weakest link, here 99.95%, because the journey still fails whenever that one dependency fails. These two cases bracket the truth. For services that are run separately, on separate infrastructure, by separate teams, the independent case is the closer one, and so the product is the honest guide. Figure 1.8 draws the arithmetic.

Figure 1.8 — The availability of a chain of hard dependencies whose failures are independent. At 99.95% each, the chain reaches checkout’s objective at the second dependency and falls below it at the third; at 99.99% each, three still leave room for checkout’s own failures.

Concert sound has exactly this shape. A sound engineer knows that the finest mixing desk cannot fix a crackling cable: the audience hears the microphone, the cable, the desk, the amplifier and the speakers in series, and the chain sounds only as clean as its worst link. That is exactly the multiplication in Figure 1.8. But here the analogy breaks, and it breaks in our favor. A sound system has no way to carry on when a cable fails short of someone walking out on stage with a spare. A service, unlike a sound system, can often keep its promise when a dependency fails, provided it was designed to. That design choice is what separates two kinds of dependency. A hard dependency is one whose failure fails the journey. A soft dependency is one whose failure the journey survives, perhaps in a degraded form that users can still accept.

With that distinction in hand, we can look at the two remedies the paper offers, both of which Encore uses. The first is a rule of thumb: critical dependencies should be roughly one nine more reliable than the service that depends on them. Encore holds its internal purchase-path services to 99.99%, and three of them multiplied together come to about 99.97%. That leaves room inside a 99.9% promise for checkout’s own failures, about 30 of the 40 minutes in a 28-day window. But the paper also warns against applying the rule mechanically, demanding an extra nine at every layer down the stack, because each nine costs far more than the one before it, as Section 1.1 showed, and the layers at the bottom would be asked for reliability nobody can afford. Where a dependency cannot supply the extra nine, the second remedy applies: make the dependency soft.

The payment service provider is Encore’s clearest case. The PSP’s contract promises 99.9% per calendar month, and Encore cannot buy another nine from it. If checkout depended on a single PSP as a hard dependency, the arithmetic would be fatal: the three internal services at about 99.97%, multiplied by the PSP’s 99.9%, give about 99.87%, below checkout’s objective before checkout has failed on its own account. So Orders & Payments routes each new payment to whichever of two PSPs is healthy, as large merchants do. This is real protection, though not perfect independence, because both PSPs lean on the same card networks, and a failure there would reach both at once. For the moments when both PSPs are degraded, Encore has a further step. Verified fans’ seats are held for up to 20 minutes until they can pay, the fan completes payment with strong customer authentication once a PSP recovers, and holds that remain unpaid lapse and return their seats to sale. No ticket is ever issued without an authorized payment. The design turns what would have been an error page into an honest delay, and that is what makes the PSP soft. Figure 1.9 shows the result for every dependency of checkout.

Figure 1.9 — Composing checkout. Hard dependencies multiply and need an extra nine; soft ones are designed out of the product, and the PSP row is why the checkout SLO is achievable at all.

Read the table from the consumer’s side and something important appears. Orders & Payments could make the PSP soft only because it knew what the PSP promised, and the Gate’s consumers can plan around it only because they know it is held to 99.99%. A published SLO thereby becomes part of a service’s interface. Encore’s service registry shows each service’s SLOs beside its API contracts, so that a team about to take on a new dependency learns how much to trust it in the same place it learns how to call it, much as a type signature tells a caller what to pass. This is information hiding, in the sense the Architecture book took from Parnas, extended from behavior to reliability. A consumer should depend on the promise and nothing more. A service that has never failed in the year since we started calling it has made us no promise at all; its good record is only a coincidence that has lasted so far, and a team that builds on the record instead of the promise will learn the difference on the day the coincidence ends.

Tenants, contracts and small numbers

Dependencies are one way an SLO is shared across teams. Customers are the other, because Encore does not make one promise to everyone. It serves venues in three tiers. About 500 self-service venues are pooled in serverless cells of about 50. Professional tenants run in Kubernetes cells with a database each. And one Enterprise tenant, venue-chain-a, runs in a silo of its own. Self-service venues are promised 99.9%, and Professional tenants 99.95%. For venue-chain-a the promise is of a different kind: a service level agreement (SLA), a contractual promise with consequences, in this case 99.95% per calendar month, with service credits owed when Encore falls below it.

An SLA and an SLO serve different purposes, and the relation between them matters. The SLA is what Encore owes the customer, and missing it costs money. The SLO is what Encore asks of itself, and missing it costs only a change in how the teams spend their time. So the SLO is set tighter than the SLA, and the gap between them is where engineers notice trouble and fix it before anyone writes an invoice. For venue-chain-a, the internal SLO is 99.97% over a rolling 28 days. The gap is thin. In time, 99.97% over 28 days allows about 12 minutes of failure, while 99.95% over a 30-day month allows about 22, so the warning arrives with perhaps ten minutes of margin to spare. The windows also differ, and that difference can hide a breach: a rolling 28-day window and a calendar month cover different stretches of days, so a bad incident early in one month can drop out of the rolling window while still counting fully against the month the contract measures. For that reason Encore tracks the SLA on its own calendar month as well, beside the SLO, and does not assume one tells it about the other.

Small tenants raise a different problem, one of statistics. Consider a venue selling 300 tickets a month under a per-venue 99.9% objective. Its error budget is 0.1% of 300, or 0.3 failed purchases, so a single unlucky fan whose checkout fails would breach the objective for the whole month. Such an SLO measures chance. Hidalgo’s book gives a whole chapter to the probability involved, and its lesson is that an SLO needs enough events to tell a real signal from noise. Encore’s answer is to measure self-service SLOs per cell, because the cell is the unit that fails together: its 50 or so venues share the cell’s pooled serverless infrastructure, and a fault there reaches all of them at once. Measured that way, the cell has enough traffic for its ratio to mean something.

Aggregation, though, has its own blind spot. If one small venue in a cell is entirely down while its busier neighbors are serving fans normally, the cell’s ratio may barely move, because the dead venue contributes only a sliver of the cell’s traffic. The cell’s SLO would report health while every fan of that venue was failing. So Encore adds a synthetic probe: a scripted visitor that loads each venue’s storefront and seat map every minute from outside Encore’s network, as a fan would. It needs no volume of real traffic to be meaningful, because it asks a direct question of each venue, and it pages when a venue fails several probes in a row. The cell’s SLO answers how well the cell is serving its fans as a whole, and the probe answers whether any one venue has gone dark. Encore needs both questions answered.

The tiers, finally, share more than they seem to. Every tenant’s checkout, whatever tier it belongs to, passes through the same Seat Inventory. That core is therefore a hard dependency of every promise Encore makes, and by the arithmetic of the first half of this section it must be at least as reliable as the strongest promise it supports. Cells, Chapter 5’s subject, limit the blast radius of a failure, so that a fault in one cell harms only the tenants inside it. What they cannot do is lift any tenant’s reliability above the reliability of the shared core beneath them all.

Recap. A journey can be no more available than its weakest critical dependency, and when dependencies fail independently their availabilities multiply, so each critical dependency needs roughly an extra nine or must be designed to be soft. A published SLO belongs in a service’s interface, because consumers should build on the promise and not on a record of good luck. An SLA should sit outside its SLO with room to spare and be tracked on its own window, low-volume objectives need enough events to mean anything, and synthetic probes catch what aggregation hides.

Exercise 1.5. Draw the dependency chain for your most important journey, mark each dependency hard or soft with its published objective (or “none”), and multiply the hard ones. Does the product leave room for your SLO? Which of the dependencies share infrastructure, so that their failures would coincide? And which one would you make soft first?

Kata: The Eight O’Clock Slots

Your brief: “Kindred Health.” Kindred is a telehealth platform that runs video consultations between patients and about 2,400 doctors in three EU countries, some 6 million consultations a year. Bookings arrive by two routes. Patients book in Kindred’s app, and 85 insurer and clinic partners book on their behalf through an API, under a contractual service level agreement (SLA) that promises them 99.9% availability each month. A single consultation leans on several systems at once: Kindred’s own booking service; an external video relay that carries the call; a national electronic identity (eID) login, which patients in two of the three countries must pass before they see a doctor; and an insurer eligibility check that confirms the visit is covered. Payments go through an external payment service provider (PSP). The busiest moment of every weekday is fixed and known in advance. At 08:00 the day’s same-day slots open, and in the flu season traffic rises 30× for about 40 minutes as patients who woke up ill compete for the morning’s appointments. Kindred has 35 engineers in six teams, and their alerting is not their own: they inherited it from a vendor template that puts a 99.99% alert on every service, whatever that service does for patients. On-call took 58 pages last month. Meanwhile the clinical lead and product disagree about whether to freeze changes during the flu season, and nothing written down helps them settle it.

Deliverables:

1. Journey SLOs — Write service level objectives for four journeys: a patient booking a consultation, joining it, the video staying usable once the call has begun, and a partner booking through the API. For each, give the service level indicator (SLI) as good events divided by valid events, the point where you measure it (as close to the patient as you can get), the target and the window. Then say where clinical safety justifies a stricter objective than patients would notice on their own. A dropped call with a doctor midway through a diagnosis is a different kind of failure from a slow seat map, and the objective should show that you know the difference.
2. The eight o’clock SLO — Write an event-window objective for the morning slot opening, in the manner of Encore’s on-sale-window SLO in Section 1.2. Count it per patient, so that a patient who retries a failing booking ten times counts as one bad experience and not ten. State its statistical limits honestly: a window of about 40 minutes, crowded at the front, is noisy, and a burn-rate rule cannot protect it in time (Section 1.4). Then say how people will watch it live during the flu season, and what page stays armed behind them for the moment their attention drifts.
3. Error budget policy — On one page, write the agreement that governs the budget. Give the thresholds of remaining budget and what happens at each; the traffic that sits outside the SLO and why; how a burn caused by one team’s change, or by an external dependency such as the video relay, is attributed; and who signs the policy. That last point matters most here, because the signatories are the people who must stop arguing about the flu-season freeze, and the policy is the place where they settle it in advance.
4. Burn-rate alert set — Write multi-window, multi-burn-rate rules for the joining-a-consultation SLO, using the arithmetic of Figure 1.6. Then work out the detection time for two failures: a total outage, and a partial one in which 3% of attempts to join fail. Those two numbers are the real specification of your alerts, so ask whether a patient waiting in a virtual waiting room would accept them.
5. Dependency composition — Draw the chain a consultation depends on. Mark each dependency hard (the consultation fails when it fails) or soft (the consultation can carry on without it), multiply the availabilities of the hard ones as in Section 1.5, and note which dependencies share infrastructure, since their failures will not be independent. Finally, choose one hard dependency and redesign the journey so it becomes soft, saying what the patient experiences when it fails.
6. Partner SLA and internal SLO — Set the internal objective that sits behind the 99.9% partner SLA, and explain the margin between the two. Remember that the SLA counts calendar months and carries consequences, while the SLO is where engineers notice trouble and fix it before any partner has a claim.

Rubric:

Where you now stand

You can now choose indicators from what users actually experience, and decide which traffic belongs in them. You can give scheduled peaks, an on-sale or a morning rush, a window of their own instead of letting them dissolve into a quiet month. You can write an error budget policy that teams with different interests are willing to live with, alert on how fast that budget is burning, and reason about reliability across the dependencies and contracts no service escapes. These are real powers, but they have two limits. An SLO can tell us that users are suffering, and how much, but not why; finding out is the craft of observability, which the Operations book treats in its Chapter 4. And an objective is only a promise, which has to be paid for in machines. The 99% that Encore now owes its fans in the first hour of a major on-sale says nothing about how many fans the Gate may admit each second, how close to its limit each Seat Inventory shard may run, or what the spare capacity costs through the quiet weeks between sales. Those are questions of queues, load and money, and telephone engineers were the first to ask them. Chapter 2 takes them up.

References

• Betsy Beyer, Niall Richard Murphy, David K. Rensin, Kent Kawahara, Stephen Thorne (eds.) — The Site Reliability Workbook. O’Reilly/Google, 2018 — free online.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering. O’Reilly/Google, 2016 — free online.
• Alex Hidalgo — Implementing Service Level Objectives. O’Reilly, 2020.
• Ben Treynor, Mike Dahlin, Vivek Rau, Betsy Beyer — The Calculus of Service Availability. ACM Queue, 2017 — free online.
• DORA — Streamlining change approval. dora.dev — free online.
• Prometheus — Histograms and summaries. Prometheus documentation — free online.
• Sloth — Sloth: Prometheus SLO generator. sloth.dev — free online.
• Cloudflare — Challenge solve rate. Cloudflare documentation — free online.
• Amazon Web Services — Testing and tuning your AWS WAF protections. AWS WAF Developer Guide — free online.