Building Resilient Systems in the AI & Cloud Era

← the whole book on one page

Chapter 2: Performance, Capacity, and Cost

Every queue is a wager that the future will be less busy than the present.

In 1908 a young Danish mathematician named Agner Krarup Erlang joined the Copenhagen Telephone Company. The company put to him a question that sounds like bookkeeping and turned out to be the beginning of a science: how many lines, and how many operators, does a city need? Build too few, and callers hear a busy tone at the moments they most want to talk. Build too many, and copper and wages sit idle through every quiet afternoon, paid for and unused. Nobody could answer the question by counting subscribers, because subscribers do not call on a schedule. Erlang’s first insight, published in 1909, was that this randomness has a shape. Calls from many independent subscribers, each deciding on their own when to pick up the receiver, arrive as a Poisson process: random in the timing of any single call, yet describable in aggregate by one number, the average rate. Once arrivals could be described, they could be reasoned about, and by 1917 Erlang had derived two formulas from them. The first gave the chance that a call would be refused because every line was busy. The second gave how long a call would wait if the exchange held it in line instead of refusing it.

The profession honored him by turning his name into a unit. One erlang is the amount of traffic that keeps one server continuously busy. A system’s offered load in erlangs is its arrival rate multiplied by the mean time each arrival holds a server. A small worked figure makes the idea concrete: a switchboard that receives 600 calls an hour, each lasting three minutes on average, receives ten calls a minute, each holding a line for three minutes, so at any moment about 30 calls are in progress, an offered load of 30 erlangs. It is tempting to conclude that 30 lines will do. They will not. Thirty lines would be enough only if the calls arrived in perfect succession, and random arrivals bunch together, so some moments bring 35 calls and others 25. A switchboard offered 30 erlangs therefore needs more than 30 lines, and how many more depends on how long callers will tolerate waiting and how often we are willing to turn one away. Notice what has happened in that sentence. How long a caller waits is a question of performance. How many lines we provide is a question of capacity. What those extra lines cost through the quiet afternoons is a question of cost. Erlang’s work shows that these are one problem seen from three sides, and this chapter treats them that way.

The question Erlang was handed is the one Encore faces before every big on-sale, with fans in place of callers. When tickets for a popular tour go on sale, the fans gather first in the Sale Gate, Encore’s virtual waiting room, which admits them to the purchase path at about 1,200 fans per second per event, however many are waiting. Behind the Gate sits Seat Inventory, the ledger whose one essential job is to sell each seat exactly once. For a large event that ledger is split into four shards, and each shard is a small consensus group whose elected leader puts every seat write for that shard into a single agreed order. The Architecture book chose those numbers, 1,200 fans a second and four shards, but did not fully show where they come from. Why not admit 2,000 fans a second and sell out sooner? Why not run the shards closer to their limit and pay for less hardware? This chapter answers those questions with Erlang’s reasoning and its modern descendants, building on the service level objectives (SLOs) of Chapter 1, which say what fans must experience, and on the telemetry that lets us see what they actually do experience, which the Operations book treats at length in its chapter on observability. (The map of Encore in Chapter 1 shows where each piece sits.)

2.1 Performance Methodology — USE, RED, and Latency Distributions

Two checklists, two points of view

Most performance investigations fail because they begin with a hunch. Someone suspects the database, so they look at the database. There they find something mildly alarming, because a busy production system always has something mildly alarming in it, and they tune it. The graphs wiggle, the team feels productive, and the real cause sits untouched somewhere nobody thought to look. Brendan Gregg called this the streetlight method, after the old joke about the drunk who searches for his lost keys under the streetlight. He did not drop them there; it is simply where the light is. The pattern is so common because looking where we already have dashboards is easy, and looking everywhere else is work.

Gregg’s USE method, published in 2012, is his answer to the streetlight. Rather than starting from a suspicion, it starts from an inventory. We list every resource in the system and ask the same three questions of each one:

• its utilization: the fraction of time the resource was busy doing work;
• its saturation: how much work is queued up waiting for the resource because it was already busy;
• its errors: how often the resource failed at what it was asked to do.

A resource, in this sense, is anything that can run out. The obvious ones are physical: CPUs, memory, disks and network links. The ones that cause the most surprising outages are logical: connection pools, thread pools, file descriptors, rate-limit quotas, and, at Encore, the ordered write log of a Seat Inventory shard, which can accept only so many seat writes per second no matter how many CPUs stand idle beside it. The operating system will report CPU utilization for free. Nobody reports the saturation of a connection pool unless we add that instrumentation ourselves, and so the logical resources are usually the ones we cannot see when we need to.

The method is deliberately dull and deliberately exhaustive, and that is its virtue. Because it walks every resource whether or not we suspect it, it finds the problem we did not suspect. A theatre’s stage crew works the same way before a show: they check every fly line, every dimmer and every radio pack, and not only the one that failed last night, because last night’s failure is the one they are least likely to repeat. The analogy breaks in one useful place. A stage crew’s list is fixed for the run of the show, whereas a software system’s list of resources changes with every deploy. A new dependency brings a new pool, and the checklist is only as complete as our latest inventory of what can run out.

USE looks at the system from the resource’s point of view. Tom Wilkie’s RED method looks from the request’s side, and asks three questions of every service:

• its rate: how many requests per second it is serving;
• its errors: how many of those requests fail;
• its duration: the distribution of time each request takes, from arrival to response.

RED descends from the four golden signals of Google’s site reliability engineering (SRE) book, and it maps naturally onto the service level indicators (SLIs) of Chapter 1: a request rate, an error ratio and a latency threshold are exactly the raw material an SLI is built from. RED fits request-driven services best. Batch jobs and stream processors don’t serve requests in that sense, and they need other signals, such as lag: how far behind the newest input the processing has fallen.

The two methods are complementary because they answer different questions. RED tells us where users feel the problem: which service is slow or failing, and for whom. USE tells us which resource is causing it: the pool, the shard or the disk that has run out. Used together they give an investigation a direction, from the symptom inward to the cause. Picture an Encore engineer paged at 3 a.m. by a seat-map burn-rate alert. They begin with RED for the seat-map service, to confirm that duration has indeed grown and to see whether every request is slow or only some. Then they walk USE down the resources behind it: the service’s own pods, its cache, its connection pools, and the Seat Inventory shards it reads from. At no point do they have to guess. The method tells them where to look next, which is a great kindness at that hour.

Latency is a distribution

Of RED’s three signals, duration is the one most often mismeasured, and the mistake usually begins with the habit of summarizing it as a single number. A service’s latency is not a number. It is a distribution: the whole spread of times that individual requests took. That spread is usually skewed, with most requests bunched near the fast end and a long tail stretching toward the slow end. It often has two humps. A seat-map request that finds its geometry in the cache returns quickly, a request that misses the cache must fetch and assemble it, and the two kinds of request form two separate clusters with a valley between them.

To describe such a shape we use percentiles. The percentile pN is the latency value below which N% of requests completed. If the p99 of the seat-map service is 2 s, then 99% of requests finished in under 2 s and the slowest 1% took longer. The p50, or median, describes the typical request; the p99 and p99.9 describe the requests at the tail.

The mean, the number most dashboards show first, is the worst summary of a two-humped distribution. A seat-map service whose mean is 120 ms can have a p99 of 2 s. The mean is pulled up by the slow cache misses and down by the fast hits, and it settles in the valley between them, at a latency that almost no actual request experienced. The fan who waited 2 s for the seat map does not experience the mean. They experience 2 s, and at Encore’s scale the slowest 1% of seat-map requests is still a great many fans (Figure 2.1).

Figure 2.1 — A seat-map service’s latency, on a log scale. The mean falls in the valley between the cache hits and the misses, where almost no request lands; the p99 lives in the misses’ long tail.

Two working habits follow from treating latency as a distribution.

The first is that percentiles do not aggregate. The p99 of a service is not the average of its pods’ p99s. A percentile is a position in a sorted list of latencies, and averaging two positions taken from two different lists does not give the corresponding position in the combined list. A quiet pod and a busy pod get equal weight in the average even though the busy one served far more requests, and a single pod with a terrible tail is diluted by its healthy neighbors. The correct approach is to record each pod’s latencies as a histogram, a set of buckets that counts how many requests fell into each latency range. Histograms can be merged honestly, because counts simply add: the service’s histogram is the sum of its pods’ histograms, and we read the service’s percentiles from that merged whole. The price is precision. We know only which bucket a request fell into, not its exact latency, so the bucket layout limits how accurately any percentile can be read. Exponential histograms, whose buckets widen in proportion to the values they cover, reduce that error across the wide range from fast cache hits to slow misses.

The second habit is that we read the tail in proportion to the fan-out. The more backends a single request depends on, the more the backends’ tails matter to it. Suppose a purchase calls 20 backends in parallel and must wait for all of them before it can answer. For any one backend, the chance of answering faster than its own p99 is 0.99. For all 20 to answer faster than their p99s, the chance is 0.99 multiplied by itself 20 times, which is about 0.82, or 82%. So in roughly 18% of purchases, nearly one in five, at least one backend lands in its slowest 1%, and the purchase waits for it. A latency that each backend team thinks of as rare, one request in a hundred, has become an ordinary experience for the user. Keep adding backends and the arithmetic keeps going: at about 70 backends, 0.99 to the seventieth power falls to about one half, which means that half of all requests now touch some backend’s p99, and a backend’s p99 has become the purchase’s median. “The Tail at Scale” gives the figure for a fan-out of 100: 63% of requests wait on at least one backend’s slowest 1%. The practical rule is that a service with wide fan-out must care about its dependencies’ p99 and p99.9 as much as a simple service cares about its own median.

There is a subtler trap, and it lies not in how we summarize latency but in how we produce the measurements in the first place. Gil Tene named it coordinated omission: a measurement error in which the load generator waits for each response before sending the next request, so that the requests that would have arrived during a stall are never sent and never measured. The load generator, in effect, coordinates with the system under test. When the system stops, the generator politely stops too, and the stall leaves almost no trace in the data.

A small worked example shows how large the error can be. Suppose a benchmark means to send 100 requests a second for 60 s, 6,000 requests in all, and the system normally answers in about 40 ms. Halfway through, the system freezes for 2 s. A closed-loop generator, one that sends the next request only after the previous one returns, sends a single request into the freeze, records one slow response of 2 s, and then carries on. Of the roughly 5,800 requests it records, only one is slow, so its p99 is still about 40 ms. The report says the system is fine. Now consider what real users would have seen. The 200 requests that were due during those 2 s would have arrived anyway, and they would have waited anywhere from 2 s down to almost nothing, depending on when in the freeze each one arrived. Those 200 slow requests are more than 3% of the 6,000, so the true p99 is no longer 40 ms; it lands at about 1.4 s. The closed-loop benchmark did not round the stall down. It erased it.

The difference is between two models of load. In a closed model, a fixed population of clients each waits for an answer before asking again, so a slow system automatically receives less work. In an open model, arrivals keep their own schedule regardless of how the system is doing. Bianca Schroeder and her colleagues showed in 2006 how far the two kinds of model diverge, and how misleading it is to study a system with the wrong one. Encore’s fans are an open population, as Erlang’s telephone callers were: when the on-sale opens at 10:00, fans click because the clock says 10:00, not because the previous fan’s request has finished. A closed-loop test of an open system measures a gentler world than the one the system will meet. The remedy is to give each request in a test an intended send time from a fixed schedule, and to time its latency from that intended moment rather than from when the generator finally got around to sending it. That way the time a request spends waiting for a stalled system counts as latency, which is exactly what it is.

Recap. Methods beat hunches because they cover the whole system whether or not we suspected the right part of it. RED describes what requests experience, and USE describes what resources endure; together they lead from the symptom to the cause. Latency is a distribution rather than a number: it is measured with merged histograms rather than averaged percentiles, read at the percentile that matches the fan-out, and taken from load that arrives on its own schedule, as real users do.

Exercise 2.1. For one service you own, list every resource the USE method should cover, including pools and logical limits such as quotas and write logs. For how many of them can you produce utilization, saturation and errors today? Then count how many backends your busiest request type calls and must wait for, and use the arithmetic above to work out what fraction of those requests touches at least one backend’s p99.

2.2 Queues, Load, and the Universal Scalability Law

The hockey stick

Erlang’s formulas gave the telephone company its answers, but the deepest lesson they hold lies in the shape they draw. When requests arrive at random and each takes a varying time to serve, a server can be idle part of the time and still make some requests wait, because arrivals do not politely space themselves out: two often turn up together, and the second must stand in line while the first is served. That waiting grows faster than the load does. To see how much faster, take the simplest model the theory offers: random arrivals, random service times and a single server. Write the server’s utilization, the fraction of time it is busy, as ρ (rho). The mean response time, which is the time a request spends waiting in the queue plus the time it spends being served, is then the service time divided by one minus the utilization. The arithmetic is worth doing slowly, because it surprises almost everyone the first time. At 50% utilization a request takes twice its service time, half of it spent waiting. At 80% it takes five times its service time, and four of those five parts are spent in the queue. At 90% it takes ten times; at 99%, a hundred. Plotted against utilization, the curve stays nearly flat for a long while and then turns sharply upward, like the blade of a hockey stick, and the bend where it turns is what engineers call the knee. Real systems differ from this model in nearly every detail, since their arrivals are not perfectly random and they have many servers, caches and pools rather than one, but they share its shape, and the shape is what we use.

In 1961 J. F. C. Kingman sharpened the point. For a single server under heavy load, he showed, the expected wait depends on two things multiplied together. One is the utilization term we have already met, ρ/(1−ρ), which explodes as ρ approaches one. The other is the variability of the arrivals and of the service times: the more irregular they are, the longer the wait at the same utilization. Two systems running equally busy can therefore have very different queues, and the one fed by lumpier traffic suffers more. This matters to Encore because an on-sale’s arrivals are about as bursty as traffic ever gets: a million fans watching the same clock tick toward 10:00 do not arrive at random, they arrive together. It helps to hold two quantities apart. Utilization is what the machine feels, a measure of how busy it is. Queueing delay is what the fan feels, a measure of how long she waits. On a quiet day the two move gently together, but near the knee they diverge sharply, and they diverge most at exactly the moment the business is busiest, which is when a fan’s wait matters most. Figure 2.2 draws both curves: the gentle model and the steeper one that burstier arrivals produce.

Figure 2.2 — Response time against utilization for one server with random arrivals, and the steeper curve that burstier arrivals draw. Encore’s write shards are planned at 80%; the same curve explains why a hot shard at 96% hurts so much more than 16 points should.

The caption’s last claim deserves its arithmetic. At 80% utilization the simple model gives five times the service time; at 96%, one minus the utilization is 0.04, so the response time is 25 times the service time. Sixteen more points of busyness have made each write wait five times longer. Section 2.3 shows how a shard comes to run that hot, and why Encore plans for it.

Every motorway at rush hour draws the same curve. Add a few more cars to a road that is already near its capacity and the traffic does not slow a little; it collapses into a jam that takes far longer to clear than it took to form. Many cities answer with ramp metering, a traffic light on the on-ramp that admits cars one at a time at the rate the motorway can absorb. Metering keeps the main road below its knee, where traffic flows, and the price is a wait on the ramp that every driver can see. The trade is a good one, because a short, visible, orderly wait for some is far better than a jam for everyone. The Sale Gate is Encore’s ramp meter, and its limit of about 1,200 fans per second per event is the timing of the green light. Metering does a second, quieter job as well, the one Kingman’s formula predicts: by admitting fans at a steady rate, the Gate strips much of the burstiness out of the arrivals, so the shards behind it see a smoother stream than the crowd in front of it. That is also why the Gate itself must not become the bottleneck. Its edge runs as serverless functions, pre-warmed before each scheduled on-sale and able to grow to 400 instances, because a waiting room that collapses under its own crowd is worse than having no waiting room at all.

Like every analogy, this one breaks somewhere, and here it breaks in a useful place. A driver held at the ramp light waits in the car; she cannot vanish and reappear a minute later as a second car. A fan who is refused can do exactly that, and does: she presses the button again, and her retry arrives as new load on top of the load that caused the refusal. Chapter 5 takes up that problem. Erlang’s waiting formula, which describes many servers sharing one queue, adds one more lesson that bears on how we carve up capacity. At the same utilization a large pool of servers keeps requests waiting far less than a small one, because when one server is tied up with a long request the others are likely to be free, whereas in a small pool one slow job blocks a larger share of the whole. So splitting capacity into isolated cells, for all the containment it buys, has a price: each small cell needs its own headroom to keep its own queue short, and the cells together need more spare capacity than one large pool would.

How far will it scale?

Queueing theory describes a single resource under load. Scaling asks a different question: what happens when we add more workers, threads or connections to share the work? Intuition says that doubling the workers should double the throughput, and experience says it never quite does. Most teams answer the question empirically, running a load test at increasing concurrency until the system stops improving and reading the knee off the graph. That is honest work, but a graph of points tells us where the system stopped scaling, not why, and without the why we cannot tell whether a change of design or of configuration would help. Neil Gunther’s Universal Scalability Law (USL) is a small model for interpreting those points. It is used mostly by capacity planners in his tradition and in database performance practice, where Baron Schwartz and the MySQL community applied it widely. The law describes the relative capacity C(N) of a system at concurrency N, where C(1) is the throughput of a single worker, and it charges two separate penalties against the ideal of perfectly linear growth:

C(N) = N / (1 + σ(N−1) + κN(N−1))

Each penalty corresponds to a different kind of cost, and keeping them apart is the whole value of the model. The first is the contention coefficient σ (sigma), the fraction of the work that must be serialized: done one worker at a time, while the others wait for a lock, a single log or a shared queue. If we set the second coefficient to zero, the law reduces to the one Gene Amdahl stated in 1967, and throughput approaches a ceiling of 1/σ times a single worker, no matter how many workers we add. Contention, in other words, bends the curve toward a flat line but never turns it down. The second penalty is the coherency coefficient κ (kappa), the cost of keeping parallel workers consistent with one another, as when they exchange messages, invalidate each other’s caches or retry after colliding. That cost grows with the number of pairs of workers, which is why the term contains N(N−1): every new worker must, in some measure, coordinate with every worker already there. Because pairs multiply faster than workers, coherency eventually outruns the benefit of adding workers, and throughput peaks and then actually falls once N passes √((1−σ)/κ). Gunther calls this retrograde scaling, and it is the most counterintuitive and most practically important thing the law says: past a certain point, adding concurrency makes the system slower in total, not merely slower to improve.

An orchestra offers a fair picture of the two penalties. Contention is the single conductor: every player watches the same baton, and no number of extra violinists lets the piece go faster than one person can beat time. Coherency is the listening: each player adjusts to the others, and in a large ensemble the effort of staying together grows faster than the number of players. The analogy breaks, as they do, because an orchestra adds players for richer sound, not for throughput, but it captures why the two costs differ in kind.

Seat Inventory has both. Each shard’s leader appends every seat write to a single ordered log, so writes take their turn at one point, and that is contention. When several fans try to hold the same popular seats at the same moment, their holds collide and the losers retry, and the work of resolving those collisions grows as more writers compete, which is coherency. To learn how large each cost is, the Inventory team ran a load test against a single shard at six concurrency levels and fitted the USL to the measured throughput:

Figure 2.3 — The USL fitted to one Seat Inventory shard. Peak throughput falls near 44 concurrent writers; beyond that, more concurrency buys less.

Read the fitted coefficients first. A σ of 0.03 means about 3% of each write’s work is serialized; on its own, that would cap the shard at 1/0.03, about 33 times a single writer, or roughly 4,000 writes/s. The κ of 0.0005 looks tiny, but it is multiplied by the number of pairs, and at 64 writers there are thousands of them: the coherency term adds about 2.0 to the denominator at that concurrency, more than the contention term’s 1.9. That is why the measured throughput at 64 writers is lower than at 32. The fitted curve, drawn in Figure 2.4, then tells the team three things.

First, the shard peaks at roughly 1,630 writes/s near 44 concurrent writers, the point where √((1−0.03)/0.0005) comes to about 44. The published figure of 1.5k writes/s per shard is therefore a measured ceiling with a little margin below the peak, not a guess. Second, raising the concurrency cap above its current 32 would gain almost nothing, and pushing it far past 44 would lose throughput, so the cap is a setting whose change belongs in configuration review rather than in an engineer’s afternoon of tuning. Third, if Encore ever needs more write capacity per event, it must come from more shards per event, a change of design that no tuning knob can stand in for. This last conclusion is the one I value most, because it turns a vague hope that the system can be tuned a little further into a clear statement of where the limit lies and what it would take to move it.

The model earns that trust only under stated conditions, and they are worth naming because each one is easy to violate. The fit needs a measured point at a concurrency of one, since C(N) is defined relative to a single worker and every coefficient is scaled from it. It needs a steady workload with a fixed mix of operations, since a test that blends holds and reads differently at each level is measuring a different system at each point. And it must not be extrapolated far beyond the largest N actually measured, because the coefficients describe the range we observed and nothing guarantees that a new kind of cost does not appear beyond it. The USL predicts throughput, not latency, so it says nothing directly about the 3 s checkout budget. And it must be refitted whenever the code changes, because a new lock or a new retry path changes σ and κ, and yesterday’s coefficients then describe a system that no longer exists.

Figure 2.4 — The USL fit of Figure 2.3 against linear scaling and Amdahl’s law. Contention alone would bend the curve; coherency turns it over, and the concurrency cap of 32 sits just below the peak.

Recap. Response time rises with utilization along a hockey stick, and variability in the arrivals steepens it, so the fan’s wait grows far faster than the machine’s busyness. Admission control, like a ramp meter, keeps a system below its knee and smooths its arrivals, at the cost of a visible queue. Load tests find where scaling stops, and the USL, where its conditions hold, explains why by separating the cost of waiting in turn from the cost of staying in step, which tells us whether the remedy is a setting or a redesign.

Exercise 2.2. Load test one component at four to six concurrency levels, including a concurrency of one, and plot throughput against concurrency. Fit the USL to the points (a spreadsheet’s solver will do). Which penalty dominates, contention or coherency? Is your current concurrency setting on the rising or the falling side of the peak?

2.3 Capacity Planning and Load Testing

On-sale arithmetic

Erlang’s question was how many lines a city needs, and ours is how many fans Encore can let through the Gate each second. Capacity planning begins with the same first step he took: choosing what to count. The unit of capacity is the measurable quantity that actually limits the business workload, the thing that runs out first when demand rises. It is rarely processor time. Arun Kejariwal and John Allspaw insist that we find that unit’s ceiling by measurement, because a ceiling we have only estimated is one we have not yet met. For Encore during an on-sale, the unit is ordered seat writes per shard. Reads can be spread across replicas and caches, but every write to a shard passes through one elected leader that puts it in order, and Section 2.2 showed that such serialized work stops scaling. The unit also matters to the business directly: a delayed seat write is a fan who cannot buy.

To plan in this unit, we need to know how much of it each fan consumes. A fan admitted by the Gate spends about three minutes choosing seats and checking out, comfortably inside the 5-minute hold. In that time the fan generates about four ordered writes: a first hold, often a second hold after losing a seat to someone faster, the sale itself, and, for the holds that fans abandon, a release or an expiry. The same fan generates about 26 availability reads, as the seat map refreshes and the fan looks around. If the Gate admits fans at a steady rate, the shards see that rate multiplied by each fan’s demand, and the arithmetic is short enough to fit in a table:

Figure 2.5 — Encore’s on-sale capacity arithmetic. Writes bind first, and the admission rate is set so that they sit at 80% of the shards’ rated capacity.

Read the table one row at a time. Each of the four shards is rated at 1,500 ordered writes a second, so together they can absorb 6,000; 1,200 fans a second, each making four writes, ask for 4,800, which is 80% of that. Reads come in at 78% of their capacity, close behind but still lower, which is why we say the writes bind: if we raised the admission rate, the write ceiling is the one we would hit first.

The last row applies Erlang’s unit to today’s fans. Recall that offered load in erlangs is the arrival rate multiplied by the time each arrival holds a server. Here 1,200 fans arrive each second and each holds a session for about 180 s, so the offered load is 216,000 erlangs of fans. Little’s law says that the average number of items inside any stable system equals their arrival rate multiplied by the time each one spends there, so 216,000 is also the number of fans we should expect to find in seat selection or checkout at any moment. This number does not limit Encore directly, but it tells every service that keeps per-session state how many sessions it must hold at once.

Now return to the 80% and ask what it costs a fan. Section 2.2’s hockey stick says that at 80% utilization the mean response time of a seat write is about five times its service time, one part service and four parts waiting. Five times a few milliseconds is still a few milliseconds, and the 3 s checkout budget absorbs that easily. No standard prescribes 80%. It is a rule of thumb, and it is defensible here for a specific reason: the wait it produces fits the budget. The remaining 20% of write capacity looks idle on a cost report, and anyone reading only that report could fairly ask why Encore pays for it. The answer is that the 20% has two jobs.

The first job is absorbing skew. Our arithmetic assumed that each shard receives a quarter of the writes, but fans do not choose seats uniformly. The front-floor sections sell first and hottest, and the seats in them may live mostly on one shard. If that busiest shard takes 30% of the writes instead of 25%, it receives 1,440 writes a second against its 1,500 ceiling and runs at 96%. This is the hot shard that Figure 2.2 anticipated: as Section 2.2 worked out, 96% on the hockey stick means a response time of about 25 times the service time, five times worse than its neighbors, although its utilization is only 16 points higher. The average across the four shards still reads 80%, which is why a dashboard that shows only the average will hide exactly the shard that fans are waiting on.

The second job is absorbing failure. When the node holding a shard’s leader is drained for maintenance or dies, the shard’s consensus group elects a new leader in the same region. The fencing that keeps a seat from being sold twice works by rejecting writes from any leader whose term has ended, and during the handover it rejects all writes on that shard, for about 40 s. The fans whose holds failed do not go away; they try again. So a shard carrying its even share of 1,200 writes a second comes back from those 40 s owing about 48,000 writes (1,200 multiplied by 40), on top of the new writes that keep arriving. The new leader can work off that backlog only with its spare capacity. With 300 writes a second to spare, it drains 48,000 in 160 s, under three minutes, and it runs at its ceiling the whole time, so every fan on that shard waits at the far end of the hockey stick until the queue is gone. The hot shard is in a much worse position. It owes more, about 57,600 writes from its 1,440 a second, and it has only 60 a second to spare, so it would need about 16 minutes, more than a quarter of an hour, which on an on-sale morning is most of the sale.

This is old wisdom with several names. Google’s SRE book plans for N+2: enough units that the service survives losing one to maintenance and another to failure at the same time. Amazon’s Well-Architected framework calls the idea static stability, meaning a system that already holds the capacity it will need when part of it fails, rather than one that must go out and acquire capacity in the middle of the failure. Encore’s version is specific to its unit: every shard’s headroom must be able to absorb the backlog of its own failover, quickly enough that the fans on that shard are not waiting through the sale. Seen this way, headroom is the price of a short queue, paid in advance.

Load tests that resemble Friday

The arithmetic is only as good as its per-fan numbers, and those come from load tests. Encore’s harness, which the On-Sale team calls Soundcheck, replays recorded on-sales, scrubbed of personal data, against a production-sized copy of the purchase path. Replaying a real on-sale matters because it carries the real mix of browsing, holding, losing seats and abandoning carts, which no hand-written script reproduces faithfully. Four further choices make Soundcheck’s results trustworthy, and each answers a lesson from earlier in this chapter.

First, Soundcheck generates load from an open model, in which each simulated fan arrives on a schedule of its own, whether or not earlier requests have been answered; the harness uses k6’s arrival-rate executors to do this. Section 2.1 explained why: a closed-loop generator waits for each response before sending the next request, so during a stall it simply stops sending, and the stall never appears in its measurements. An open-model generator can still fall into the same trap if it runs short of resources and quietly skips arrivals it was scheduled to send. Soundcheck therefore fails any run in which the generator dropped scheduled arrivals, since dropped arrivals mean coordinated omission has reappeared inside the rig.

Second, its arrivals are bursty, because Kingman’s variability term is real: at the same average rate, bursty arrivals produce longer queues than smooth ones, and a test with smooth arrivals would flatter the system. Third, its seat choices are skewed toward the best sections, because uniform random seats spread writes evenly across the shards and would understate the hot shard we met above. Fourth, it stubs the payment service provider (PSP), the outside company that processes card payments, with a latency distribution taken from production rather than a fixed fast response. A stub that always answers in a few milliseconds would hide what happens when the PSP slows down, and the first outage in the Architecture book began with exactly that, a PSP that slowed down.

Even a faithful copy is not production. The copy may have the same number of machines, but it does not share production’s network paths, its noisy neighbors, its cloud account limits or its real dependencies, and some problems appear only there. Other companies with sharp, predictable peaks have reached the same conclusion: Shopify has written about running its Black Friday scale tests against production infrastructure, and Hotstar’s engineers about load-testing at the full scale of a cricket audience. Encore does the same twice a year. It runs Soundcheck at off-peak hours against the hidden test event that Encore’s synthetic buyers already use, so that the test exercises the real purchase path without selling any real seat. One cost-saving detail follows from the purpose of the test: the load generators may run on cheap spot capacity, which the cloud provider can reclaim at short notice, because losing a generator only ends a run early. The system under test does not run on spot, because its behavior is the thing being measured.

One lesson concerns state rather than traffic. When a team runs a load test twice, the second run meets a system that the first run has already warmed: caches are full, connection pools are open, and code paths have been compiled and loaded. On-sale morning is often the first run. At Encore the clearest case is the seat-map geometry cache, which holds each venue’s layout (rows, sections, the shape of the hall) but never which seats are free. When that cache is cold, every request misses it and goes to Catalog, the service that owns venue data, and it does so at the worst possible moment, in the opening seconds of the sale when the most fans arrive at once.

A touring band understands this problem from the stage. The crew soundchecks in an empty arena in the afternoon, knowing the room will sound different at nine o’clock, when 15,000 bodies absorb the reflections that bounced off empty seats. Experienced engineers adjust for the difference, because they know which way it runs. The analogy breaks in a direction that should worry us: the hall grows kinder when it is full, while a cold cache makes the system harsher just when it is busiest. A load test against warm caches therefore errs in the dangerous direction, telling us the system is stronger than it will be at curtain-up. So every test plan at Encore must say which cache state it assumed, and on the day itself the cache is warmed before the doors open.

The Wednesday review

All of this arithmetic and testing would be wasted if it never met a decision, and at Encore it meets one each week before a big on-sale. The Reliability team holds an on-sale readiness review on the Wednesday before each big on-sale, which usually opens at 10:00 on a Friday. It is a launch review of the kind Google’s SRE book describes, covering capacity, dependencies and who will be on call. The review’s most important rule concerns the admission rate, since that single number drives every row of Figure 2.5. A proposal to raise it must arrive with the arithmetic redone at the new rate and a load test run at the new point, and the review asks what cache state that test assumed, because a test against a warm cache answers a different question from the one the on-sale will ask.

The review also checks the limits that belong to someone else, which are easy to forget because no Encore dashboard shows them. The cloud account caps how many serverless functions may run at once, and the Gate runs on those functions; Chapter 3 tells how Encore learned that one. The load balancers have their own limits, and each PSP has agreed to a request rate that Encore must stay under. The review requires each of these quotas to stand at least 1.3 times the forecast peak, so that a busier night than forecast still fits. Forecasts are sent in advance to any provider the night depends on, so that the provider is not surprised by Encore’s traffic, a practice Amazon formalizes as Infrastructure Event Management. The core of the checklist fits on a card:

On-sale readiness — Wednesday, T-2 days
[ ] Admission rate unchanged, or new arithmetic + load test attached
[ ] Load test: open model, bursty, skewed seats, cache state stated
[ ] Writes per shard ≤ 80% at planned rate; failover drain < 5 min
[ ] Pre-scale plan: Gate concurrency, Inventory floors, cache warm
[ ] Quotas: function concurrency, LB, PSP rates ≥ 1.3 × forecast peak

Each line on the card points back to something this section has argued. The first two lines guard the per-fan numbers and the honesty of the test that produced them. The third holds the 80% and asks that every shard, the hot one included, could drain a failover’s backlog in under five minutes. The fourth names the capacity that must be in place before the doors open, which Section 2.4 explains, and the fifth covers the limits other people set. I have come to believe that a review like this earns its keep less by catching errors on the day than by making each assumption visible enough that someone will ask about it when it changes.

Recap. We plan capacity in the unit that limits the business, which for Encore is ordered writes per shard, and we keep headroom for skew, for variability and for the failover that will eventually come, sizing it so that every shard can drain its own failover backlog quickly. Load tests use open, bursty arrivals, realistic seat skew and realistic dependency latency, state the cache state they assumed, and are confirmed periodically at production scale. Before a scheduled peak, the limits other people set are checked as carefully as our own.

Exercise 2.3. Write your system’s equivalent of Figure 2.5 for its busiest hour: name the unit of capacity, the demand each user generates, and the utilization that results. Then compute how long your binding resource would take to drain the backlog of its own failover at that utilization, and check whether your last load-test report says whether caches were warm or cold.

2.4 Autoscaling and Elasticity

The lag in the loop

Section 2.3 did its arithmetic in advance, on a Wednesday, for a Friday that everyone could see coming. A natural question follows: why plan at all, when the cloud will add machines as the load arrives? Autoscaling does make that promise, and for much of the year it keeps it. What it removes, in practice, is the need to plan for demand that changes slowly. Demand that changes quickly is another matter, and the reason lies in what an autoscaler actually is.

An autoscaler is a feedback loop, and like every feedback loop it acts on the past. It cannot see the load that is coming; it sees only a measurement of the load that has already arrived, and it reacts after a chain of steps, each of which takes time. First the metric must rise, and the monitoring system must scrape it. Then a stabilization window must pass, so that the autoscaler does not chase every momentary blip. Then a new pod must be scheduled, perhaps on a node that must first be bought from the cloud provider and booted. Then the container image must be pulled, the process must start, and the new instance must warm its caches and connections before it serves at full speed. Each step takes seconds. Together they often take minutes. Suppose, for illustration, that each of those six steps took a modest fifteen seconds: the loop would need a minute and a half to complete one cycle, and it would add capacity in proportion to the load it saw a minute and a half earlier. An on-sale that rises 500× in minutes is past its first surge before the loop has completed even that one cycle, and the fans who arrived during that minute and a half have already met a system sized for the quiet morning.

Pilots learn the same lesson from their engines. A turbofan at idle takes several seconds to spool up to full thrust, so a pilot who waits until the aircraft is low and slow on approach before asking for power will get it too late. That is why the stabilized approach is flown with the engines already spooled, ready to answer at once. Encore’s equivalent is pre-scaling: provisioning capacity ahead of a load we can predict, instead of waiting for the load to ask for it. Encore’s curtain-up workflow, the self-service routine that warms the purchase path on the morning of a sale, does exactly this before each scheduled on-sale. It raises the Gate’s provisioned concurrency toward its 400 instances, sets Seat Inventory’s minimum replica counts (its floors), and warms the seat-map cache, all before the doors open at 10:00. Hotstar’s engineers reached the same conclusion for cricket, writing that capacity must be in place ahead of the match. The analogy is worth following to the point where it breaks, which is the end of the flight. A pilot lands and shuts the engines down, and the flight is over. A service, after its peak, must also scale back down, and scaling down has hazards of its own: instances being removed must first drain their open connections without dropping requests in flight, and the crowd does not always leave on schedule. A late second wave of fans, for instance those returning as abandoned holds lapse and seats come back on sale, can arrive just as the capacity that would have served them is being handed back.

Reactive scaling still has its place, because not every surge is announced in advance. When it is used for unscheduled demand, it fails in three characteristic ways, and it helps to know them by name.

It can lag, as we have just seen: the loop reacts to the load of a minute or two ago.

It can thrash, scaling up on a burst, down in the lull that follows, and up again on the next burst, paying the full startup cost each time and never quite having the right amount of capacity. The usual remedies damp the loop. A stabilization window makes the autoscaler wait before acting on a change. An asymmetric policy scales up quickly and down slowly, on the reasoning that too much capacity for a few minutes costs a little money while too little costs failed requests. And a floor keeps a minimum number of instances running however quiet the moment looks. Kubernetes follows the asymmetric reasoning by default, scaling up at once and waiting five minutes before scaling down. Chapter 3 treats thrashing more fully, as a problem of control loops in the lineage of the governor.

And it can amplify, which is the most dangerous of the three, because it turns a small problem into a large one. In Encore’s first outage, told in the Architecture book, the PSP slowed from 200 ms to 8 s. Requests waiting on the PSP piled up, connection pools filled, and health checks began to fail. The autoscaler read this distress as a need for more capacity and added instances. But each new instance opened still more connections to the PSP, which was the very bottleneck causing the trouble, and so the autoscaler’s help made the bottleneck worse. The lesson generalizes beyond payments: adding instances to a service raises its capacity only if whatever it depends on can serve the extra work. The autoscaler can only buy what the dependency can sell.

Scaling on the right signal

If an autoscaler can be fooled into amplifying a failure, the defense begins with what it measures. We should choose the scaling signal as carefully as we chose an SLI in Chapter 1, by asking what the number really says about the work the service is doing.

The most common signal is CPU utilization, and it suits work that really is computation, such as rendering, encryption or compression, where a busier service burns more processor time. It misleads badly for services that spend their time waiting. A thread blocked on a slow dependency is not using the processor at all, so a service whose every thread is stuck waiting for the PSP looks idle on CPU while being saturated on everything else: its connection pool is full, its request queue is growing, and its callers are timing out. An autoscaler watching CPU would conclude that all is well, or even scale the service down.

Signals that count waiting work track the truth more directly. Concurrency in flight is the number of requests a service has accepted and not yet finished; queue depth is the number of requests or messages waiting to be started. Kubernetes’ Horizontal Pod Autoscaler (HPA) can scale on such custom metrics, KEDA scales on the depth of event sources such as queues and streams, and Knative scales on concurrency in flight. These signals carry their own hazard, though, and it follows directly from Little’s law, which we met in Section 2.3: the number of requests in flight equals the arrival rate multiplied by the time each request takes. When a dependency slows, each request takes longer, so concurrency rises even if not a single extra fan has arrived. A concurrency-driven autoscaler therefore sees a sick PSP as a surge in demand, and it will chase that PSP exactly as the old one did unless something stops it. A better signal makes the autoscaler more accurate in normal times, but it does not by itself make the autoscaler safe when a dependency fails.

That something is a ceiling. Kubernetes requires every HPA to declare a maximum number of replicas, and most teams pick that maximum by feel, a round number that seemed generous at the time. Encore derives its ceilings from the arithmetic of Figure 2.5 instead. The four shards of a large event can absorb a rated 6,000 ordered writes a second, so no service that sends work toward those shards is allowed to scale past the point where its replicas could generate more than the shards can absorb. However distressed the metrics look, the autoscaler cannot add the instance that would overload the shards. The Wednesday review rechecks each ceiling whenever either side of the calculation changes, whether that is a new admission rate, a different number of shards or a change in how many writes each fan generates. Connection pools and client-side concurrency limits do the same job one layer down, capping how much any single instance can ask of a dependency at once, so that the limit holds even inside an instance the autoscaler has already added.

There is a broader point here about the word “elastic”. A system is elastic only as far as its least elastic part. At Encore, the Gate’s serverless edge can grow to 400 instances and the stateless services can add pods within minutes, but the shard count of Seat Inventory does not grow during an on-sale at all; as Section 2.2 showed, more write capacity means more shards per event, which is a design change made in advance. Everything upstream of the shards can stretch only as far as the shards allow, and the ceilings simply make that fact explicit in configuration. I have come to believe that this is the right test of anyone’s claim about elasticity: anyone who calls a system elastic should be able to name its slowest-scaling component and say how long it takes to grow.

Recap. An autoscaler is a feedback loop that acts on the past, and it can lag behind a fast surge, thrash between too much and too little, or amplify a failing dependency by adding load to it. We pre-scale for the peaks we can foresee, scale reactively on signals that measure waiting work, and cap every service at what its dependencies can absorb, because any signal that rises when a dependency slows will otherwise drive the autoscaler to feed that dependency more work.

Exercise 2.4. For one autoscaled service, measure the time from a sudden step in load to a new instance serving at full speed, and compare it with how fast your steepest real peak rises. Then find the service’s maximum replica count: what downstream limit justifies that number, and what would happen downstream if the service reached it?

2.5 FinOps — Cost as an Operational Signal

The night’s settlement

After every concert on a tour, the tour manager and the local promoter sit down and settle the night. They set the ticket revenue against the cost of the hall, the crew and the local expenses, and they do it show by show, while the numbers are fresh and the people who spent the money are still in the building. The point of the ritual is that nobody discovers in December that the Tuesday shows lost money all year. FinOps, the practice of making cloud spend a shared, continuous responsibility of engineering and finance, is that settlement brought to software. The analogy is worth taking a step further, because it also shows where software differs. A promoter’s costs are mostly agreed before the doors open: the hall’s fee and the crew’s wages are in a contract. A cloud bill is written by the system’s own behavior, minute by minute, by every retry, log line and idle replica, and it arrives hours after the spending happened. So the settlement cannot wait for the end of the night; it has to run continuously, like any other operational signal. The Architecture book introduced FinOps’ central move, measuring cost per ticket sold. In this section we go further and treat cost as a telemetry stream with its own anomalies and its own choice of denominators, read alongside the capacity arithmetic of the sections before it. Erlang’s question at the opening of this chapter was, after all, partly a question about money: lines and operators sitting idle through a quiet afternoon were a cost, and so were callers hearing a busy tone.

J.R. Storment and Mike Fuller describe FinOps as a lifecycle of three phases: Inform, Optimize and Operate. Inform means making spend visible to the people who cause it; Optimize means reducing it; Operate means building those habits into the organization’s routine. The first phase is the hardest, and everything else depends on it, because nobody can optimize a cost they cannot see or do not own. Informing requires allocation, attributing each part of the bill to an owner, a service or a tenant. Most of that work is done through cloud accounts, tags and labels, so that a database instance carries the name of the team that runs it. The difficult case is shared infrastructure, above all the Kubernetes nodes on which many teams’ pods run side by side, where the provider bills for the node and says nothing about who used it. Tools such as OpenCost solve this by apportioning each node’s cost according to what each workload requested: a pod that requested a quarter of a node’s capacity is charged a quarter of its price. Encore allocates a little over 90% of its bill in this way, per service and per tenant, which is about as far as mature practice gets. The remainder, the genuinely shared costs that no single owner caused, is not spread thinly across everyone; it is shown as its own line, so that its size is visible and someone is accountable for it.

Once the bill has owners, we can divide it by the work it bought, and here the choice of denominator matters more than it first appears. The figure the Architecture book followed, which drifted from €0.11 to €0.19 per ticket, was the blended cost per ticket: the month’s whole bill divided by the month’s tickets. A blended figure is easy to compute and easy to explain, but it mixed two quite different causes into one number. Part of the rise was real inefficiency: a metrics forwarder, billed per invocation, that cost about €9,000 a month, and logging that was more verbose than anyone needed. Part of it was simply a larger Encore, which had added a second serving region and new tenants, with that larger fixed base spread over a quiet quarter in which fewer tickets were sold. The first cause is a defect to fix; the second is a consequence of growth and of the calendar, and the blended figure cannot tell them apart. So Encore now reports two figures instead of one. The first is the fixed monthly cost of being open for business at all, whatever the month’s sales. The second is the variable cost per ticket, the spend that rises and falls with the number of tickets sold, divided by that number. The split is approximate, and it is worth being honest about why. Much cloud spend is step-fixed: flat until demand crosses a threshold, and then a whole new node or a whole new shard at once. If Encore ever added a fifth shard for a season of large events, that shard would be not quite fixed, since it would exist because of ticket volume, and not quite variable, since it would not shrink when one week is quiet. Someone has to decide which side of the line each step belongs on, and write the decision down. After the forwarder fix, Encore’s variable cost per ticket stands at €0.12, broken down in Figure 2.6. In a typical month the fixed line adds about four cents per ticket, so the Architecture book’s blended figure would now read about €0.16.

Figure 2.6 — Encore’s variable cost per ticket after the forwarder fix. Tokens are one line among six; they get the same treatment as the others.

The table repays a slow reading. The largest line, purchase-path compute, carries a note that ties it directly to Section 2.4: the capacity that pre-scaling buys before an on-sale is billed from the moment it is warm, whether or not fans have arrived yet, and that is a cost Encore pays knowingly. Telemetry, at nearly a fifth of the total, is a reminder that the instruments that let us see a running system are not free, which is why their sampling policy has a review date. And the token line, the newest kind of spending, is small and sits in the same table as everything else, which is exactly where it belongs.

Splitting the two figures pays off because each one, when it moves, points to a different kind of cause. A rising variable cost per ticket is an engineering signal: some part of the design has become less efficient, and each ticket now costs more to sell than it did. A rising fixed cost while ticket volume stays flat is a planning signal: capacity was bought for demand that has not come, or has not come yet. The first kind of problem lives in code, and so it can be found the way performance problems are found. Continuous profiling, which Google described in 2010 as sampling the call stacks of every production machine all the time, makes this practical: because the profile is always running, a team can compare the profile from before a release with the profile after it and see which change made a request more expensive, in CPU seconds and therefore in euros. The second kind of problem is not found in any profile. It lives in forecasts and purchase decisions, and it is settled in planning meetings rather than in pull requests.

The token line is the youngest in the table, and it deserves a closer look precisely because it is new. It can be attributed at all only because every model call Encore makes passes through its AI gateway, which tags each call with the feature that made it, the tenant it served and the model it used; without that single chokepoint, token spend would be scattered across many teams’ invoices and nobody could say what the support assistant costs per conversation. Three levers move the token line, and they are not equally mature. The first is batch processing. The large model providers price their batch interfaces at about half the interactive rate, in exchange for returning results later rather than at once, and that suits work that can wait, such as the nightly summaries of the day’s support tickets. The second is prompt caching, also a priced feature offered by providers. Every turn of the support assistant begins with the same long block of instructions, and caching lets that unchanging prefix be resent cheaply instead of being paid for in full each time. The third lever is routing: sending most turns to a smaller model at about a tenth of the price, and reserving the larger model for the turns that need it. This lever is the least proven of the three. The best published account, RouteLLM (2024), reports large savings on benchmarks, but production accounts are few. Encore routes about 80% of turns to the cheaper model, and it does so only because each routing rule must hold its score on the 200-conversation golden set of Chapter 8 before it ships. The order matters: we pull the levers whose behavior we understand first, and we pull the young one only with a test that would tell us if it had started to cost us quality instead of money.

Commitments and anomalies

Storment and Fuller draw a distinction that keeps two conversations from blurring into one. Using less is engineering: fixing the forwarder, sampling telemetry, routing tokens. Paying less for what we do use is purchasing, and its main instrument is the pricing model. Cloud providers offer three broad kinds. Commitments, such as reserved instances and savings plans, trade a promise of steady spend for a lower rate. On-demand capacity promises nothing in either direction and costs the most per hour. Spot capacity is the cheapest of all, but the provider can reclaim it at short notice when it needs the machines back. A good portfolio follows the shape of demand. We commit to the floor that runs all year, because that capacity will be used whether or not the promise is made; we pay on demand for the scheduled peaks, because they are tall and brief; and we put interruptible work on spot, meaning work that can be stopped and restarted without harm, such as batch analytics and Soundcheck’s load generators. (The system under test never runs on spot, for the reason Section 2.3 gave.) Encore commits about 60% of its steady compute, which is less than the 70–90% many practitioners aim for, and it does so by choice. Its floor is still moving, as workloads shift between containers and serverless functions, and a commitment to capacity it will soon stop using would lock a forecast error into the bill. It revisits the figure each quarter, once the next on-sale calendar is known and the floor can be forecast with more confidence. The trade-off is a familiar one: a lower price is bought with less freedom to change the architecture, and a team in the middle of changing it should buy less of that kind of discount.

Commitments are about the price of the spending we expect; anomaly detection is about the spending we do not. Cost anomaly alerts are built like the burn-rate alerts of Chapter 1. A forecast says what the month’s spend should be, and the alert fires when actual spend runs ahead of that forecast fast enough to matter. The difference lies in the data. Billing data arrives between 8 and 24 hours late, so detection works on a scale of hours rather than minutes, and no cost alert can catch a problem in its first few minutes the way an SLO burn alert can. That delay also shapes how urgent the response should be. Most organizations route cost anomalies to tickets or chat, on the reasoning that money lost overnight is recoverable in a way that a failed on-sale is not. Encore’s choice is a little sharper. A spend rate that would exhaust the month’s budget within a week, which means spending at roughly four times the planned rate, sends an urgent notification to the owning team during business hours, though it does not page anyone at night. A sudden step in a team’s line opens a ticket. The reason for having any alert at all is that a cost regression, unlike a latency regression, makes no noise. No fan sees an error page and no dashboard turns red; the money simply accumulates until someone looks, as the forwarder’s €9,000 a month did for as long as nobody was looking.

There is also a judgment that no dashboard can make for us, and it brings this section back to the arithmetic of Section 2.3. The 20% of write headroom in Figure 2.5 appears on the bill as capacity that, on most days, does nothing. A well-meaning optimization pass, reading the cost report on its own, could propose to remove it and would see only savings. It would not know that this idle-looking capacity is what absorbs the backlog of writes after a shard failover, and that without enough of it a shard would take far longer to recover, as the hot shard’s quarter of an hour already showed. The cheapest configuration and the safest configuration are seldom the same, and the difference between them is not a matter of efficiency; it is a decision about how much risk to carry. I have come to believe that the most useful thing FinOps does is to make that decision visible, so that it is made deliberately by people who understand both sides of it, rather than by accident in a cost-cutting spreadsheet that has never heard of a failover.

Recap. FinOps begins by allocating the bill to owners, then splits fixed from variable cost, using judgment for the costs that come in steps, and watches the variable cost per ticket as an engineering signal. Commitments follow the shape of demand and are held back while the architecture is still moving. Token cost is reduced with the proven levers first and the young one only under evaluation. And cost is always read beside capacity, because headroom shows up on both, and only the capacity model explains why it is there.

Exercise 2.5. Split last month’s cloud bill into fixed and variable costs, and compute the variable cost per business unit, whatever your system’s equivalent of a ticket is. Where did you put the step-fixed costs, and would your finance partner agree with that choice? What fraction of your steady compute is covered by commitments, and who decided that fraction, from what forecast?

Kata: Filing Season

Your brief: “Ledgerline.” Ledgerline sells tax-preparation software as a service in one EU country. It serves 2.4 million filers a year, and they do not arrive evenly: 35% of them file in the final six days before the deadline, and on the last evening the load reaches 14× its normal level. Every filing runs through a calculation engine that takes about 400 ms of compute, and the engine runs again each time the filer edits a figure, so a hesitant filer costs several times what a decisive one does. Every one of those runs works against a shared PostgreSQL cluster with a single primary, which is the one place in the system where all the work converges.

Thirty engineers build and run all of this, and last season their preparation did not survive contact with the deadline. Their load tests were closed-loop, waiting for each response before sending the next request, and ran on a staging cluster a quarter of production’s size; they reported a p99 of 180 ms. On deadline night, support logged thousands of complaints from filers who waited 10 s. The autoscaler, which scaled on processor utilization, oscillated every few minutes and twice exhausted the database’s connection limit. Meanwhile the cloud bill rose 40% in a year in which filings rose only 12%, and a new AI feature that explains each calculation to the filer has added its own token line to the invoice, one that nobody forecast.

Each deliverable below applies one part of this chapter to Ledgerline, in roughly the order the chapter developed them, and together they should tell a single, consistent story about deadline night.

Deliverables:

1. USE and RED inventory — List every resource and service on the filing path, from the calculation engine’s instances to the database’s connection pool and its primary. For each resource give its utilization, saturation and error signals, and for each service its rate, errors and duration, then mark the signals Ledgerline cannot produce today. The missing ones are where last season’s surprises were hiding.
2. Capacity model — Choose the unit of capacity that actually limits deadline night, which may not be processor time. Estimate the demand one filing places on it, counting the re-runs that edits cause, and compute the utilization of the binding resource at the last evening’s peak. Then add headroom for skew and for a failover of the database primary, and show the arithmetic so that a colleague could check every step.
3. Scaling curve — Load-test the calculation engine against the database at five or more concurrency levels and plot the throughput. Identify the knee, the point beyond which adding concurrency stops helping, and, if the Universal Scalability Law’s conditions hold, fit it to the points to explain what is bending the curve.
4. Test and readiness plan — Design a load test from an open model, in which arrivals keep their own schedule, with bursts, skewed demand and a stated cache state, and say how its results will be confirmed at production scale. Add a review of the quotas and limits that other people set. Finally, explain why last year’s tests reported 180 ms while filers waited 10 s.
5. Autoscaling policy — Name the signal the autoscaler will scale on and say why it is better than processor utilization. Set floors and a pre-scaling schedule for the final six days, stabilization settings that stop the oscillation, and a maximum instance count justified by what the database can absorb.
6. Cost plan — Express cost per filing, split into its fixed and variable parts, so that the 40% and the 12% can be compared on one footing. Propose commitments, anomaly alerts, and the token levers you would pull first.

Rubric:

Where you now stand

You can now investigate performance by method rather than by hunch, and read latency as a distribution whose tail matters in proportion to how many backends each request fans out to. You can explain why waits explode as a resource nears saturation, plan capacity in the unit that truly limits the business, with room for skew and for the failover that will eventually come, and rehearse a peak with load that arrives on its own schedule, so that the rehearsal does not flatter you. You can keep an autoscaler from amplifying the very trouble it was meant to absorb, and you can read the invoice as one more stream of telemetry, always beside the capacity it pays for. The autoscaler was our first glimpse of a larger family: much of what we measured here is run, day to day, by loops that observe the system and correct it. Chapter 3 turns to those loops in full, the reconciliation engines of Kubernetes, serverless platforms and GitOps.

References

• Brendan Gregg — Systems Performance, 2nd ed.. Addison-Wesley, 2020.
• Arun Kejariwal, John Allspaw — The Art of Capacity Planning, 2nd ed.. O’Reilly, 2017.
• J.R. Storment, Mike Fuller — Cloud FinOps, 2nd ed.. O’Reilly, 2023.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering. O’Reilly/Google, 2016 — free online.
• Jeffrey Dean, Luiz André Barroso — The Tail at Scale. Communications of the ACM, 2013.
• Bianca Schroeder, Adam Wierman, Mor Harchol-Balter — Open Versus Closed: A Cautionary Tale. USENIX NSDI, 2006.
• Neil J. Gunther — Guerrilla Capacity Planning. Springer, 2007.
• Brendan Gregg — The USE Method. brendangregg.com — free online.
• Shopify Engineering — Performance Testing at Scale. Shopify — free online.
• Hotstar Engineering — Scaling Is Not An Accident. Hotstar — free online.
• Amazon Web Services — AWS Infrastructure Event Management. AWS — free online.
• FinOps Foundation — FinOps Framework: Unit Economics. FinOps Foundation — free online.
• Gang Ren, Eric Tune, Tipp Moseley, Yixin Shi, Silvius Rus, Robert Hundt — Google-Wide Profiling. IEEE Micro, 2010.
• Isaac Ong et al. — RouteLLM: Learning to Route LLMs with Preference Data. arXiv, 2024.
• Anthropic — Prompt caching. Anthropic documentation — free online.