Building Resilient Systems in the AI & Cloud Era

← the whole book on one page

Chapter 3: The Runtime and Its Control Plane — Kubernetes, Serverless, IaC, and GitOps

A controller is an opinion about the world, held continuously and acted upon without being asked.

We still speak of a runtime as the place where our software sits, as if it were a shelf. A modern runtime is better pictured as a crowd of small machines, each of which keeps noticing that something in the world differs from what was asked for and then sets about correcting it. A scheduler notices a pod with nowhere to run and places it on a node. An autoscaler notices that CPU is running hot and adds replicas. A GitOps agent notices that someone has hand-edited a setting in the cluster and puts it back the way the repository says it should be. An upgrade reconciler notices that a node is on an old version and, through the night, drains it and replaces it, one node after another. None of these machines waits to be told; each holds its opinion of how things ought to be and acts on it continuously.

That changes what operating a system means. A generation ago we operated servers: we logged in, we restarted processes, we resized disks. Today we mostly operate the things that operate servers, and our work is to understand, configure and sometimes restrain them. The question of how to supervise a machine that is itself correcting another machine sounds thoroughly modern, but it is older than the computer, and it is worth going back to where it was first asked carefully.

In 1868 James Clerk Maxwell published a short paper in the Proceedings of the Royal Society called “On Governors.” Its subject was the centrifugal governor that James Watt had fitted to his steam engines some 80 years earlier. The device is simple enough to picture: two iron balls hang on hinged arms from a spindle, and the engine itself spins that spindle. As the engine speeds up, the balls fly outward and rise; through a sleeve and a lever, that rising motion closes the steam valve, and the engine slows. As the engine slows, the balls fall back, the valve opens, and the engine speeds up again. The engine, in effect, regulates itself. Engineers who lived with these machines had learned two uncomfortable things about them. A governor sometimes hunted, swinging the engine above and below the intended speed instead of settling on it. And making a governor more sensitive, so that it reacted more strongly to small errors, often made the hunting worse rather than better. The intuitive cure was aggravating the disease. Maxwell’s contribution was to stop treating engine and governor as two separate things. He wrote down the equations of the engine and the governor together, as one system, and showed that whether the swings die away or grow depends on the roots of a single characteristic equation for that combined system. Stability, in other words, was not a property of the governor alone, nor of the engine alone, but of the way the two were coupled.

The paper is usually cited as the beginning of control theory, and when Norbert Wiener named the new science of cybernetics in 1948, he pointed back to it. I find it a useful lens for everything in this chapter, because the analogy holds closely. Every reconciliation loop in a cloud runtime is a governor: it senses the state of something, compares it with what was intended, and pushes to close the gap. Like Watt’s governor, every such loop can hunt, overshooting and correcting and overshooting again. And two governors attached to the same shaft, each with its own idea of the right speed, can fight each other indefinitely while the engine lurches between them. Where the analogy breaks is in number: Watt’s engine had one governor, and a cloud runtime has dozens, written by different people for different purposes, most of whom never imagined the others.

Encore runs dozens of these loops. They live in its Kubernetes cells, the self-contained copies of the stack that keep SaaS tenants apart from one another; in its serverless Sale Gate, the waiting room fans queue in before an on-sale; and in its GitOps repositories, which describe what every environment should look like. Most of these loops were written by someone else, by the Kubernetes project, a cloud provider or a tool vendor, and Encore’s engineers meet them as configuration rather than as code they can read at a glance. This chapter is about operating them well. For each loop we will ask three questions: what it owns, how fast it acts, and what it cannot see. Those three questions are enough to explain most of the surprises a runtime delivers. (The map of Encore in Chapter 1 shows where each piece sits.)

3.1 Declarative Systems and Reconciliation Loops

The anatomy of a loop

Before we can operate a crowd of governors, we need a clear picture of a single one. A reconciliation loop is a process that repeatedly observes the actual state of some part of the world, compares it with a declared desired state, and takes the smallest action it knows that moves the actual state toward the desired one. Then it does the same thing again, and keeps doing it for as long as it runs. The Architecture book introduced Kubernetes as such a loop and was content to let it work. Here we take the loop itself apart, because most of the surprises in running a modern platform come from parts of the loop that our configuration files never show.

Every loop has the same four parts as Watt’s governor, and it helps to set them side by side. The sensor reports what is true now: on the steam engine it is the pair of spinning flyballs, and in a Kubernetes controller it is a watch on the Kubernetes API, a stream of updates about the objects that controller cares about. The set point is what ought to be true: the intended engine speed, or the spec of a resource, the part of its manifest where we write down what we want. The comparator works out the difference between the two: the sleeve and lever that turn the height of the balls into a position of the valve, or a function in the controller that computes a diff between the spec and what the watch reports. The actuator changes the world: the steam valve, or an API call that creates or deletes a pod. When a deployment asks for five replicas and the watch shows three, the comparator finds a shortfall of two and the actuator creates two pods. That is the whole mechanism, and it is simple enough that it is tempting to stop looking at it too soon.

The loop also has two properties that the YAML never mentions. The first is its gain: how hard it pushes for a given error. A governor with long lever arms throws the valve wide at the slightest change of speed; an autoscaler that adds many replicas for a small rise in CPU is high-gain in just the same way. The second is its delay: how long the loop takes to sense the effect of its own last push. On the engine, steam takes time to reach the cylinders and the flywheel has inertia. In a cluster, a new pod must be scheduled, started and warmed before its share of the work shows up in the metrics the autoscaler reads. Maxwell’s result, translated out of his equations, is that a loop with high gain and long delay overshoots, notices the overshoot only after it has gone too far, and then pushes back too hard the other way. Picture an autoscaler that sees CPU high and adds pods; because the new pods are still starting, CPU still looks high on the next check, so it adds more; when they all arrive at once, CPU falls sharply and it removes too many. That is hunting, and the engineers of Watt’s day would have recognized it at a glance.

We should be honest about what this way of thinking buys us. Nobody tunes a Kubernetes controller by solving Maxwell’s characteristic equation, and the analogy is a way of seeing rather than a tuning method. What it gives us is a pair of questions to ask before trusting any loop: how hard does it push, and how long before it sees what its push has done? The answers are rarely in the manifests we write. They live in defaults, in documentation, and in the physics of whatever the loop is acting on, which is exactly why they are so often overlooked.

The idea did not arrive with Kubernetes. Its software lineage runs through Mark Burgess’s CFEngine (1993), which described the state that Unix machines should converge to and, on every run, repaired whatever had drifted from it. The shift was a quiet but deep one. A traditional script is a list of steps, run once, that assumes it knows the starting point; a convergent description says where the machine should end up, and can be run again and again because a machine that is already correct needs nothing done to it. Kubernetes and GitOps agents sit squarely in that tradition, which the OpenGitOps principles now write down in four words: state is declared (we describe what we want rather than how to get there), versioned (kept in Git, with its history), pulled (an agent inside the environment fetches it, rather than a pipeline outside pushing it in) and continuously reconciled (compared and repaired over and over, never applied once and forgotten). What has changed since CFEngine is the tempo. A configuration pass every few minutes has become hundreds of loops reacting within seconds, and interactions between loops that were once too slow to notice can now play out several times inside a single on-sale.

Level-triggered, not edge-triggered

How a loop decides when to act matters as much as what it does, and the vocabulary for that comes from electronics. A digital circuit can be built to respond to the moment a signal changes from low to high, or to whether the signal is high right now. Software borrowed both terms. An edge-triggered system acts on changes: it hears “a pod died” and starts a replacement. A level-triggered system acts on state: it looks, finds three pods where five were asked for, and creates two, whatever history led there. Kubernetes controllers are level-triggered. They do subscribe to watch events, but they treat each event only as a hint to look again sooner; the decision always comes from comparing the current state with the desired one. So if a controller misses an event, or restarts in the middle of its work, it loses nothing that matters. On its next pass it simply looks again, and the only price is the time before that pass comes round.

A theatre crew offers a useful picture. Before every performance the crew walks the stage with a preset sheet, checking each prop and piece of scenery against where it should be, and they do not need to remember what the actors moved last night, because the sheet and the stage tell them everything. That is level-triggered work. A crew that relied only on hearing each change called out over the headset would be edge-triggered, and one missed call would leave a chair in the wrong place for the rest of the run. The analogy breaks at one revealing point. A human crew member who finds something out of place may stop and ask whether the director moved it on purpose. A controller never asks.

Encore learned the difference the hard way. Its first tenant-provisioning automation was edge-triggered: a TenantCreated message started a script that created a database, a namespace and a DNS record. When the script failed halfway, nothing noticed, because the message had already been consumed and nobody was looking at the state it was meant to produce. Three Professional tenants lived for a month with a namespace and no database. The replacement is a small operator that reconciles a TenantDatabase custom resource on every pass. Its steps are idempotent, meaning that running a step twice has the same effect as running it once (“create the database if it does not exist”), so the operator can safely repeat its whole sequence each time it looks. Deletion goes through a finalizer, a marker on the resource that Kubernetes requires to be removed before the object can disappear, which gives the operator its chance to clean up the database and DNS record first. Now a half-finished tenant is simply a resource whose actual state differs from its desired state, and the next pass finishes the job without anyone being paged. A level-triggered loop forgives the past, because it never consults it.

That forgiveness has a price, and the crew analogy has already shown us where it lies. Because a level-triggered controller judges only the present state against its desired state, it cannot tell accidental drift from a deliberate human change. A hand-edited setting that is not in the desired state looks to the controller exactly like damage, and it will undo that change on every pass, as often as the human makes it. When that human is an engineer trying to help during a busy evening, and the controller is one of several on the same system, the result is the subject of the next part of this section.

When governors fight

Two definitions will keep the discussion precise. A controller in Kubernetes is a reconciliation loop for one kind of resource, and an operator is a controller that encodes one application’s operational knowledge behind a custom resource, as the TenantDatabase operator does for Encore’s tenant databases. A production cluster runs many of them at once. Besides the built-in controllers, Encore’s clusters run a Horizontal Pod Autoscaler (HPA) for each scaled deployment, the cluster autoscaler, the GitOps agent (Argo CD), the TenantDatabase operator and the platform’s node-upgrade reconciler, which drains and replaces nodes one at a time to keep each pool at its target Kubernetes version. Each of these is reasonable on its own terms. Trouble begins where two of them claim the same thing, which is to say where two actuators write to the same field, each steering toward a set point of its own.

The largest public example is recent: the race in Amazon DynamoDB’s DNS automation in October 2025, which the Operations book tells in full in its chapter on automation, in which two instances of one controller interleaved their work and left a regional endpoint’s record empty. Neither instance was wrong on its own; the fault lay in how their actions were interleaved. That is the uncomfortable lesson of control systems in general. We can verify each loop carefully in isolation and still know very little about how the loops behave together.

Encore’s version was small, and all the more instructive for being ordinary. On a Thursday before an on-sale, a Catalog engineer expecting a surge of browsing ran kubectl scale to take Catalog from 20 replicas to 60. Argo CD runs with self-heal on, which means it continuously compares the live cluster with Git and reverts any difference; it found replicas: 20 in the manifest and restored that count within seconds. Meanwhile, as browsing ebbed, the HPA, set to aim for 60% CPU with a floor of 12 replicas, began recommending fewer pods. An HPA does not scale down the moment its recommendation drops. It keeps a scale-down stabilization window, five minutes here, and scales down only to the highest recommendation it has made within that window, so that a brief lull does not shed capacity that will be needed a moment later. For five minutes, then, it held. Once the window was full of low recommendations, it trimmed toward 12. Argo CD saw a count that differed from Git and restored 20 within seconds, and the HPA, whose window still held nothing but low recommendations, trimmed again at its next evaluation. For the rest of the half hour the two traded the count back and forth several times a minute, and each trade started pods with cold caches. Nothing failed, which is why the episode is easy to dismiss. But the engineer believed the service was running at 60 replicas when it never was for more than a few seconds, and on a busier day real decisions would have rested on that belief. Figures 3.1 and 3.2 show the half hour and the governors involved.

Figure 3.1 — Catalog’s replica count during the fight, sketched. The spike to 60 is reverted in seconds; the autoscaler waits out its five-minute window and then trades the count with the GitOps agent several times a minute. Notice that the count never returns to the 60 the engineer expected.

Figure 3.2 — The governors on Catalog’s shaft. Two rows claim spec.replicas, and the manifest claimed it too. The last row has no record and no delay, which is why it loses.

Read Figure 3.2 as Maxwell might have read a drawing of an engine. Each row is a governor, and the columns give its lever, its delay and the place its set point is kept. The fight happened because three set points (the number in Git, the HPA’s target and the number in the engineer’s head) were all attached to the same field, and the engineer’s was the only one written down nowhere, so the two machines kept erasing it.

The repairs follow directly from that reading. First, one field, one owner. Encore removed replicas from every manifest whose deployment has an HPA, as both the Kubernetes and Argo CD documentation advise. The team followed the documented migration steps while doing so, because simply deleting the field can let the next sync reset the deployment to the default of one replica. Policy now rejects any manifest that sets replicas on a deployment an HPA also manages. Second, express intent to the owner. A planned surge is no longer a hand-typed replica count; it is a change to the HPA’s floor, made in Git through the curtain-up workflow that pre-warms each on-sale, which also warms node capacity ahead of the sale and lowers the floor again when the sale ends. The engineer’s intention is the same as it was on that Thursday, but it now reaches the one loop entitled to act on it, and it leaves a record. Third, respect the damping. The scale-down window plays the part of a dashpot on the flyballs (a piston moving through oil, which resists sudden motion), slowing the governor’s response so that it does not chase every flicker, and teams that shorten it without measuring tend to rediscover hunting.

Underneath the three repairs lies one structural idea. Two loops pulling on one lever oscillate. Two loops on two levers, with the slower one setting the faster one’s set point, form a hierarchy instead: Git, changed slowly and with review, sets the HPA’s floor; the HPA, acting in seconds, sets the replica count; the cluster autoscaler, acting in minutes, adds nodes when pods cannot be placed. Each loop has its own job and its own timescale, and none writes a field that another owns. Hierarchy is how control engineers have long mastered complexity, and it serves us just as well when the governors are made of software (Figure 3.3).

Figure 3.3 — Catalog’s governors as a hierarchy. Each slower loop writes the set point of a faster one, and only the dashed accent arrow claims a field another loop owns. Notice that it is also the only arrow that leaves no record.

Recap. A reconciliation loop observes, compares and acts, and its gain and delay decide whether it settles or hunts, just as Maxwell showed for the governor. Level-triggered loops forgive missed events and half-finished work, but for the same reason they undo any human intent that has not been declared. When two loops own one field they fight, so we give each field a single owner, express our intent by changing that owner’s set point, and arrange the loops in a hierarchy in which slower loops steer faster ones.

Exercise 3.1. Pick one production deployment and list every controller, script and human that can change its replica count or its nodes. For each, note its set point, where that set point is recorded and how quickly it acts. Is any field claimed twice? If so, decide which loop should own it and how the others should express their intent instead.

3.2 Kubernetes in Production

Requests, limits and the honest workload

Section 3.1 took the reconciliation loop apart in general. Kubernetes is where most of us meet such loops every day, and the first of them, the scheduler, reasons from information we hand it ourselves. Kubernetes schedules on promises. A pod’s resource request is the amount of CPU and memory it promises to need, and the scheduler places the pod only on a node where that much is still unclaimed. A pod’s resource limit is the most it will ever be allowed to take, however much is free. Notice what the scheduler does not look at when it places a pod: what the pod actually uses. It adds up the requests already on each node and compares them with the node’s capacity, much as a hotel takes bookings against its number of rooms without checking how many guests actually sleep in them.

Requests and limits together also set the pod’s quality-of-service class, and that class decides the order of eviction when a node runs short of memory. A pod whose limits equal its requests is guaranteed its share and is evicted last. A pod that requests less than its limit is allowed to burst, and is evicted earlier when the node is under pressure. A pod that declares nothing at all is evicted first. The numbers in a manifest are therefore not paperwork. They are the pod’s statement of how it should be treated on a crowded day.

Bilgin Ibryam and Roland Huß call the discipline of declaring these numbers the Predictable Demands pattern, and the reason it matters follows directly from the fact that the scheduler plans with promises. If a team requests far less than its service uses, the scheduler, believing the requests, keeps packing pods onto a node that looks half empty on paper but is in fact full, and the pods on it then fight over resources nobody reserved. If a team requests far more than it uses, the opposite happens: nodes look full while their processors idle, and the company pays for capacity it never touches. Either way the scheduler makes sound decisions from false information. This is why Encore right-sizes its requests every quarter from observed usage, so that the promises the scheduler plans with stay close to how the services really behave.

The two resources fail in very different ways when a container reaches its limit, and the difference shapes how we set them. A container that exceeds its memory limit is killed. That failure is loud and visible: the pod restarts, a counter goes up, and someone notices. A container that reaches its CPU limit is not killed but throttled: the kernel hands out CPU time in short accounting periods, and a container that has spent its quota for the current period simply waits until the next one begins. Nothing restarts, no counter that anyone usually watches goes up, and the only symptom is that some requests take longer than they should. Because the wait falls on whichever requests happen to be running while the quota is spent, the damage shows up in the tail of the latency distribution, at the p99, while the average looks healthy.

Encore learned this the slow way. The Orders service had a spiking p99 for a month before someone put a graph of throttled periods beside the latency graph and saw the two line up. The house rule that came out of that month is simple to state. Memory limits are set equal to memory requests, so a service is never surprised by a memory ceiling it did not plan for. CPU limits are left off latency-sensitive services, so that a short burst of work can borrow processor time sitting idle on the node, instead of waiting for the next accounting period.

The rule is contested, and I think rightly so, because it is safe only where CPU requests are honest. When a busy node has more demand than processor time, the kernel divides the time in proportion to the requests, so the request is what guarantees each pod its share. A service that drops its CPU limit while understating its request has given itself freedom to take from its neighbors and only a weak claim to protect itself from them. Noisy batch workloads at Encore therefore keep their CPU limits. A batch job that would happily use every core it is offered gains little from finishing sooner, and on shared nodes one team’s burst is another team’s tail latency. The rule is a trade between latency for the services fans wait on and fairness among neighbors, and it holds only while every team keeps its promises.

Disruption is a normal day

Requests and limits decide where a pod lives. The next question is what happens when the cluster needs to take that home away, which in a well-run cluster happens all the time. Kubernetes distinguishes two kinds of disruption. A failed node is an involuntary disruption: it cannot be negotiated with, and our only defense is having enough replicas elsewhere. Voluntary disruptions are the ones the platform causes on purpose, such as a drain so a node can be upgraded, or the cluster autoscaler removing a node it considers underused. These can be negotiated with, and the instrument of negotiation is the PodDisruptionBudget (PDB): a declaration that at most so many pods of a given set may be voluntarily unavailable at once.

The mechanism is straightforward. A drain does not simply delete pods; it asks the eviction API to evict them, and the eviction API checks each request against the relevant budget. If evicting a pod would leave more of its set unavailable than the budget allows, the request is refused and the drain waits, retrying until a replacement elsewhere has become ready. A stage crew works the same way during a set change: it may strike only so many pieces at once, because the scene still in front of the audience needs the rest to stand.

But managed upgrades wait only so long, and here the budget’s protection runs out. Google Kubernetes Engine (GKE) respects a PDB for up to an hour during a node upgrade and then evicts by force; Amazon’s managed node groups fail the update after 15 minutes unless told to force it. The providers have good reason to behave this way, since a single carelessly written budget, one that allows no pod to be unavailable, would otherwise block an upgrade indefinitely. The consequence for us is that a PDB is a request for patience, honored for a bounded time, and a service that relies on it must be able to become safe to evict well within that time.

A budget also protects only the count of pods, and nothing inside them. When a pod is evicted, Kubernetes does two things at about the same moment: it sends the process a SIGTERM, telling it to stop, and it begins removing the pod from the endpoints that services and load balancers route to. Those two messages travel by different paths and arrive at different times. The process may receive its SIGTERM while requests are still in flight, and while load balancers that have not yet heard the news are still sending it new ones. A pod that exits the moment it is told to drops that work on the floor, and a budget that counted perfectly will not have helped.

The proven remedy is graceful termination, which has three parts that work together. First, a readiness probe that fails as soon as shutdown begins, so the pod announces that it should receive no new traffic. Second, a short preStop pause, which delays the stop signal for a few seconds so the endpoint removal reaches every load balancer before the process stops accepting work. Third, a grace period long enough for the process to finish the requests it already holds. With all three, a drained pod leaves the way a good session musician leaves the stage: it stops taking new requests, finishes the phrase it is playing, and only then puts down the instrument.

The budget makes one more assumption that is easy to miss: that its pods are spread out. If two members of one group share a node, one drain takes both, and a budget that imagined them as independent is quietly wrong. Topology spread constraints close that gap by telling the scheduler to place a set’s pods across nodes and across zones, so that no single node or zone holds more than its share. Encore’s golden-path template sets up both graceful termination and topology spread for every new service, so a team has them from its first deploy without having to learn either lesson from an incident.

Even with all of this in place, a PDB is a promise about how many, never about which. Seat Inventory shows why the distinction matters. It is the service that sells each seat exactly once, and it runs each shard as a small replicated group with a fenced leader, its members spread across zones. Its PDB allows one pod unavailable per shard, so no drain will take a shard below quorum, the majority it needs to keep accepting writes. In counting terms the shard is safe. But the budget treats every member of the group as interchangeable, and they are not, because one of them is the leader. A drain may remove the leader, and a leader’s departure, though safe, triggers a failover, with a short window of rejected writes while a new leader is elected and fenced in. Database operators such as CloudNativePG know this and move leadership to another member before a drain, so the departing pod is an ordinary follower by the time it goes. Seat Inventory has no such operator yet.

On a quiet Tuesday the window is invisible: few fans are buying, and a retry a moment later succeeds. Whether it stays invisible every day of the year depends on which other loops run at the same time and what they know about the calendar, since nothing in a drain knows whether a major on-sale is under way. The Operations book, in its account of the Vesper Hale on-sale, shows the cost when nobody asks.

Upgrades as a standing condition

Drains are frequent for a simple reason: the platform underneath never stops moving. Kubernetes ships three minor releases a year, each patched for about a year, and managed providers eventually upgrade clusters that fall behind, ready or not. An organization that runs Kubernetes is therefore never finished upgrading; it is always somewhere in the middle of the next one. The useful image is a ship’s hull, which must go to dry dock on a schedule because fouling and corrosion accumulate whether anyone is watching or not. The analogy breaks in an instructive place. A ship must leave service to be docked, but a runtime can be docked one compartment at a time while it sails, and the craft of Kubernetes upgrades lies in choosing those compartments and the order in which they go.

Encore’s Runtime team chooses them by upgrading in waves. Each region runs a core cluster for Encore’s own services and six cells for Professional tenants. A new version reaches a canary cell of internal test tenants first, where nothing a customer depends on can break, then the other cells, then the core clusters, with the node-upgrade reconciler from Section 3.1 draining node pools through the PDBs described above. The ordering is deliberate: each wave tests the next, and the clusters whose failure would hurt most take the version last, after it has run on clusters whose failure would hurt least.

Before each wave a check scans manifests, add-ons and custom resource definitions for APIs the new version removes. Kubernetes retires old API versions as it evolves, and the commonest upgrade failure is not a fault in Kubernetes itself but a chart still asking for something that no longer exists. Finding those requests before the wave turns a failed upgrade into an ordinary pull request. Figure 3.4 shows the waves.

Figure 3.4 — A Kubernetes upgrade moving through Encore’s 14 clusters in three waves. The canary cell takes a new version first, and the core clusters, which carry Encore’s own services, take it last.

Many clusters, deliberately

The waves are possible only because Encore runs many clusters, and that was a choice with a cost. One large cluster would pack better, since spare capacity in one corner could absorb demand from another. But its control plane would be a single blast radius, so one bad configuration could reach every service and tenant at once, and each of its upgrades would be a single event with nowhere to rehearse it. Cells, which Chapter 5 treats in depth, trade some efficiency for containment: a failure, or an upgrade gone wrong, stays inside one cell.

The price of containment is paid in tooling, because 14 clusters tended by hand would become 14 slightly different clusters within a year. Encore pays it with one GitOps layout, one set of policies and one set of dashboards, which make the 14 clusters one fleet whose members differ only where a difference was intended. This is decomposition used well: the fleet is split so that failures stay small, and held together by shared structure so that the parts do not drift apart.

Recap. Requests and limits are the promises the scheduler plans with, so they must be honest, and a CPU limit fails quietly, as tail latency, where a memory limit fails loudly. PDBs govern how many pods a voluntary disruption may take, for as long as the upgrader will wait, and say nothing about which; graceful termination and topology spread protect what the count cannot. Upgrades never end, and they are best absorbed in waves through cells run as one fleet.

Exercise 3.2. For one latency-sensitive service, graph CPU throttling against p99 latency for a week, and list its PDB, its spread constraints and its shutdown sequence. Which pod’s loss would hurt most, and does anything in your cluster know that?

3.3 Serverless and Managed Services — Operating What You Don’t Run

The shared responsibility line

So far in this chapter we have operated machinery we could open: clusters whose nodes we drain, controllers whose settings we tune, manifests we can read line by line. A growing share of every modern system is machinery we cannot open at all. A serverless function platform runs our code on servers we never see, and a managed queue, database or object store is a service whose internals belong entirely to someone else. Encore’s Sale Gate, the waiting room we met at the start of the chapter, is built this way, and so the question of this section is a strange one for an operations book: what does it mean to operate something we do not run?

Cloud providers answer part of that question with what they call the shared responsibility model: the provider is responsible for the security and operation of the cloud, and the customer is responsible for what they put in it. The provider keeps the data centers powered, the hypervisors patched and the function platform scheduling; the customer chooses the configuration, writes the code, guards the credentials and decides who may call what. Amazon has since extended the same division to resilience, so that the provider answers for the resilience of its infrastructure and the customer for the resilience of the workloads built on top of it. The model is a useful map of who fixes what. What it does not mention, though, is the user, and responsibility to the user is never shared. A fan who refreshes a frozen waiting-room page does not know or care which side of the line the fault lies on. When the Sale Gate is slow, fans blame Encore, and they are right to, because Encore chose the provider, chose the configuration and promised them a fair place in the queue.

Operating a managed service therefore means operating everything around a box we cannot open: how we configure it, the quotas that bound it, the ways it fails as seen from outside, and how we respond when it does. Touring crews know this work well. Before a show arrives in a new city, a production manager advances the venue, which is to say she learns ahead of time its power limits, its load-in doors, its rigging points and its curfew, so that nothing about the building surprises the crew when the trucks pull up. She cannot rewire the hall, but she can know its limits precisely and plan the show to fit inside them. We do the same for a provider, reading its quota pages and documented limits and, because documents are never the whole story, taking our own measurements of how it actually behaves. The analogy breaks in one instructive place. A venue’s limits are written on a spec sheet and stay put for the night, and the production manager can walk the building to check them. A provider’s internals cannot be walked, and its behavior can change beneath us without notice, so our measurements are never finished; they are a standing practice, renewed the way a crew re-advances a venue it has played before.

Quotas are part of the architecture

Every managed service has limits, and the most consequential ones are often invisible until we hit them. Function platforms, for example, cap concurrency, the number of function instances that may be running at the same moment, per account and per region. That cap is a single pool shared by every function in the account, unless some of it is set aside for a particular function. Two functions that have nothing to do with each other in our architecture diagram are therefore neighbors on the provider’s ledger, drawing from the same well.

Encore learned this during an on-sale rehearsal. A Sales Analytics backfill, deployed as functions in the same cloud account as the Gate, consumed most of the regional concurrency just as the Gate’s synthetic fans began arriving. Nothing was broken in either system; each did exactly what it was built to do. The failure lay in a dependency that no diagram showed, the shared concurrency pool, and it is a small lesson in one of the oldest themes of our discipline: the accidental complexity of a system often hides in the resources its parts share rather than in the parts themselves.

Three changes followed, each aimed at a different part of the problem. First, the Gate now runs in its own cloud account, as does each serverless cell for self-service venues, as Amazon’s multi-account guidance recommends; an account boundary turns one shared pool into separate pools, so a busy neighbor can no longer drain the Gate’s well. Second, the Gate has reserved concurrency, a portion of the pool set aside for it alone. A reservation cuts both ways, because it is a ceiling as well as a floor: the function is guaranteed that many instances, but it can never use more. So the reservation is sized to the forecast peak and revisited whenever that forecast changes, since a reservation sized for last year’s biggest show would quietly cap this year’s. Third, a quota inventory is generated nightly and kept beside the capacity models of Chapter 2. It records the limits on the path a fan travels, compares each one with the forecast peak, and alerts when the forecast comes within 30% of any limit.

That threshold is deliberately stricter than the warning a provider typically offers, which fires at around 80% of current use, and the difference is one of timing. A warning based on current use tells us we are close to a limit only once we are already close to it, which during an on-sale is far too late. A warning based on the forecast tells us well ahead that the next big show will come close. Encore needs that lead time for two reasons. Raising a limit means asking the provider, and an increase can take days to be granted. And some limits do not appear in the provider’s quotas API at all, so no automated warning would ever fire for them; they live in the inventory only because someone read the documentation and wrote them down.

Cold starts deserve the same treatment. A cold start is the delay a function suffers when the platform must create a fresh instance to serve a request, loading the runtime, the code and its dependencies before any work can begin; a warm instance, already running, skips that delay. For a waiting room, where a crowd of fans arrives within seconds, a wave of cold starts comes at exactly the wrong moment. The Gate therefore pre-warms with provisioned concurrency, instances the platform creates and keeps ready in advance, before every scheduled on-sale. The pre-warming begins early enough for the allocation to complete, because allocating provisioned concurrency itself takes minutes, and a warm-up that finishes after the doors open has warmed nothing. An unscheduled surge, such as a surprise announcement that sends fans to the site without warning, still meets cold functions. Encore cannot prevent that, so it plans for it: its latency service level objective (SLO) includes Encore’s own allowance for cold starts, and the team measures real cold-start time monthly by deliberately forcing one, rather than trusting a figure that may have drifted since the last check.

When the provider is down

Quotas and cold starts are limits we can learn and plan within. The harder case is the day the provider itself fails, and one such day is well documented. On 28 February 2017, a mistyped command during routine maintenance removed more capacity than intended from Amazon S3 in the US-East-1 region, and much of the internet that depended on S3 failed for about four hours. One detail of that day has stayed with operators ever since. The administration console of Amazon’s own Service Health Dashboard depended on S3, so for a while the provider could not post the outage on the very status page its customers were refreshing to learn whether there was one.

That detail carries the general lesson. A provider incident is one that the team owning the symptoms can mitigate but cannot repair. Encore’s engineers cannot restart a provider’s storage fleet or roll back its deployment; the fix belongs to someone else, on someone else’s schedule. What remains in their hands is everything on their own side of the line: noticing early, limiting the harm to fans, and keeping people informed. Those are exactly the decisions that go badly when improvised in the moment, under pressure and with incomplete information, so Encore makes them in advance.

Each managed dependency in Encore’s runbooks therefore comes with three pre-decided answers. The first is how we detect its degradation ourselves: through our own synthetic checks, which exercise the dependency the way fans do, whatever the provider’s status page says, because 2017 showed that the status page can be the last to know. The second is what we degrade to, chosen so that the experience worsens gracefully instead of collapsing: the Gate holds fans in the queue with admissions paused, preserving their places instead of turning them away, and the Catalog serves cached pages, which may be slightly stale but are far better than an error. The third is how we tell fans and organizers what is happening, since a silent waiting room breeds more refreshes, more support calls and more distrust than an honest message would.

I have come to believe that this is the real shape of the bargain behind every managed service. A managed service removes the work of running it and leaves us the work of surviving it. The first kind of work is real, and handing it to a provider is often the wisest decision a small team can make. The second kind stays with us whatever we buy, and a team that forgets it will rediscover it on the day of its biggest on-sale.

Recap. The provider shares responsibility for the machinery, but responsibility to users stays with us, because the product they experience is ours. Quotas, reservations and cold starts are properties of our own architecture, to be kept in an inventory, budgeted and tested, and measured against the forecast rather than against today’s use. We can mitigate a provider incident but never repair it, so we decide in advance how we will detect it ourselves, what we will degrade to, and what we will tell the people waiting.

Exercise 3.3. List every managed service on your most important user path, together with its tightest quota, your forecast peak measured against that quota, and what your system does when the service is down for an hour. Mark any entry where the honest answer is “we would find out.”

3.4 Infrastructure as Code and GitOps

State, drift and the reality behind the file

The Architecture book made the case for infrastructure as code (IaC) and GitOps: describe the environment in files, keep the files in version control, and let tools make the world match them. That case was about design. Operating these tools is about something less tidy, namely the gap between what the code says and what actually exists, and that gap is where most of this section lives.

The gap shows itself first in a file most engineers would rather not think about. A Terraform configuration, or one written for its open-source fork OpenTofu, says something like “there is a database called ledger_db with these settings.” The cloud provider knows nothing of that name; it knows an instance with an identifier it generated itself. So the tool keeps a state file, a record of which real resource corresponds to which declaration in the code, and consults it on every run to decide whether a declaration is already satisfied, needs changing or needs creating. The state file is the tool’s memory of the world, and it is the tool’s only way to connect a line of code to a thing that costs money and holds data.

That makes the state file production data in every sense that matters. Two engineers running the tool at once can corrupt it, so it needs locking. Losing it leaves the tool unable to recognize resources it created, so it needs backups. And because the tool records the attributes the provider returns, such as a database password generated at creation time, the state often holds secrets that the code itself never contains. It therefore needs encryption, access control and an audit log, exactly as a customer database does. Teams that keep state on a laptop or in an unprotected bucket have, without meaning to, created the most sensitive and least guarded file in their estate.

Once the tool has a memory of the world, the world can disagree with it. Drift is any difference between the declared state of infrastructure and its actual state. It matters how the drift arose, because the right response depends on who or what made it. Drift made by people, like the console fix applied at 2 a.m. during an incident, should be either reverted or written back into code the next morning; left alone, it is a change nobody reviewed and the next run of the tool will silently undo. Drift made by other loops is a different matter. An autoscaler that resizes a node group is doing its job, and the IaC tool should be told not to care, which Terraform expresses with an ignore_changes setting on that attribute. If we forget, the IaC tool becomes another governor on the autoscaler’s shaft, in the sense of Section 3.1: two loops claiming one value, each undoing the other. The rule of one owner per field applies to infrastructure code as much as to Kubernetes manifests.

Drift is easy to find if we look for it, and the way to look is to ask the tool what it would change. Encore runs a plan against every stack nightly, without applying it. A plan that finds nothing to do means code and world agree; a non-empty diff means something has drifted, and Encore files it as a ticket for the team that owns the stack. Most of those tickets are dull, a tag edited by hand or a setting nudged during an incident, and that dullness is the point: drift found the next morning is a chore, while drift found months later, in the middle of an unrelated change, is a puzzle.

The blast radius of a plan

A plan is the tool’s statement of intent: the list of resources it will create, change and destroy if we approve it. Its most important line is the summary at the end that counts what it will destroy, because creation is usually reversible and destruction of a stateful resource often is not. The blast radius of a plan is the set of resources a single apply could affect, and it is bounded by the stack, the unit of code the tool plans and applies at once. A stack that holds one service’s queues can, at worst, break that service; a stack that holds the whole company can break the whole company.

That is the operational reason behind Kief Morris’s case for small stacks. Morris also warns against cutting too fine, since every boundary between stacks becomes a dependency, with outputs from one stack feeding inputs to another, and those dependencies become the new risk. The question is where to cut. Encore cuts along ownership and rate of change: resources that the same team owns and changes at the same pace live together, and resources that change rarely, like networks, sit apart from those that change weekly. In practice that means stacks follow cells and services, which is also how the rest of Encore’s runtime is decomposed.

Encore learned why the summary line matters before it had any of this. Its earliest infrastructure lived in a single stack, and a harmless-looking module rename once produced a plan that would destroy and recreate the Finance ledger’s database. The reason lies in how the state file identifies resources. It keys each one by its address in the code, so when the module was renamed the tool saw a resource vanish under the old address and a new one appear under the new address, and it proposed what any literal-minded bookkeeper would: delete the first and create the second. Nothing in the plan said “rename.” A reviewer caught it by reading to the bottom of a long plan, where the destroy count sat. Encore was lucky in its reviewer, and it decided not to depend on that luck again. Three safeguards now do her job.

The first removes the misunderstanding at its source. A rename is declared with a moved block, which tells the tool that the old address and the new one name the same real resource, so the plan shows a move and nothing destroyed:

moved {
  from = module.finance_db.aws_db_instance.this
  to   = module.ledger_db.aws_db_instance.this
}

The second guards against destruction even when the code asks for it. Stateful resources carry prevent_destroy in code and deletion protection on the resource itself. Either alone is weaker than it looks. prevent_destroy makes the tool refuse any plan that would destroy the resource, but it lives in the very code a careless edit removes, and a refactor that deletes the resource block deletes the guard with it. Deletion protection lives with the provider, outside our code, so the provider refuses the deletion whatever our code says. Together they mean two independent mistakes are needed to lose the ledger’s database.

The third changes what the reviewer sees first: plans are reviewed with the destructive summary at the top, so the most consequential fact is the first thing read rather than the last.

Surgery offers an older version of this discipline. Surgical teams adopted a practice for the moment before the first incision, the time-out, in which the team stops and confirms aloud the patient, the procedure and the site. The World Health Organization’s checklist study of 2009 found that complications and deaths fell where the checklist was adopted. The lesson I take from it is that the worst errors are made confidently, at the moment of commitment, by people who have no doubt they are about to do the right thing; a ritual pause at that moment catches what confidence hides. A plan review is our time-out. The analogy has a limit worth noticing, though. In an operating theater the whole team stands in the room and speaks; a plan review is often one tired engineer skimming a diff alone, and a pause that depends on one person’s attention is a weak pause.

So we automate the part of the time-out that can be automated. Policy as code means rules written as code and evaluated by a machine against every plan and manifest before it is applied, so the check happens every time whether or not anyone is alert. Open Policy Agent (OPA) is a common engine for such rules. Encore’s rules deny three things: any plan that destroys a resource labeled stateful without a separately approved override; any Kubernetes manifest that sets replicas beside an HPA, which is the configuration that set Argo CD and the autoscaler fighting over Catalog in Section 3.1; and any public storage bucket. Each rule records a lesson that someone, at Encore or elsewhere, learned the hard way, which is perhaps the best way to read any organization’s policy set.

GitOps repositories and promotion

Infrastructure code shapes the cloud resources a system runs on; GitOps applies the same thinking to what runs on the clusters themselves. The term GitOps was coined at Weaveworks in 2017, for an older idea made precise. The older idea is that the desired state of a system should be written down and versioned. The precision is in how changes reach the cluster. In a push model, a pipeline outside the cluster holds credentials and applies changes to it. In GitOps, a pull-based agent inside each cluster watches a Git repository and continuously reconciles the cluster toward it, so that people change the repository and nothing else changes the cluster. The agent is one more governor of the kind Section 3.1 described, with Git as its set point.

Encore splits its repositories by purpose. Application repositories build container images and publish them by digest, the content hash that names one exact image, rather than by a tag like latest that can be moved to point at something else. A separate environment repository, encore-deploy, holds every cluster’s desired state. It is organized as a shared base, describing what every environment has in common, with one overlay per environment and cell that records only how that place differs. Because each overlay states its differences explicitly, environments cannot quietly diverge: any difference between staging and production is a line someone wrote and someone can read. An Argo CD instance in each cluster reconciles from encore-deploy.

With that layout, releasing a new version means changing a digest in one directory, and promoting it means changing the same digest in the next directory. Figure 3.5 shows the stages.

Figure 3.5 — Promotion as a series of pull requests. Each stage is a commit someone can read and revert, and the gates are the same SLO signals Chapter 1 defined, applied to the deployment itself.

Two of these gates need more explanation than a table can give. The canary cell serves only internal test tenants, so its natural traffic is a trickle, and an SLO that holds for an hour on a trickle proves little; almost anything survives being barely used. Encore therefore drives synthetic buyers through the canary cell to give the new version real work, and it treats the first Professional cells, where paying tenants arrive, as the real test. That is why a person on the owning team approves each step from there on, watching each cell’s burn rate before moving to the next.

The second subtlety concerns what the repository can tell us. The repository records what was intended to run, which is not always what did run. A sync can fail halfway; a resource can be applied and then crash; someone can change the cluster directly. Argo CD reports two statuses for each application that close this gap. Its sync status says whether the live cluster matches Git, and its health status says whether the resources it applied are actually working. Encore alerts on any application that has been out of sync for more than 15 minutes and on any whose health Argo CD reports as degraded. Only those two signals, read together, tell us what is really running.

Emergencies are where GitOps is most often misunderstood. The model says nothing changes the cluster except Git, but during an incident a person may need to act faster than a pull request can be written, reviewed and merged. The danger is that the agent does not know there is an incident. Encore’s break-glass procedure grants a named engineer an hour of direct cluster access, logged and reviewed afterwards, and its first step is to suspend automated sync for the affected application. Without that step, Argo CD’s self-heal treats the engineer’s fix as drift and reverts it within seconds, as it reverted the Catalog scale-up in Section 3.1, long before anyone could write the change back to Git. The procedure ends where the model requires it to: with a commit that records the fix in encore-deploy, and with automated sync restored. The order matters. A person acting during an incident must first tell the loop to stand aside, and the loop must be brought back only once Git agrees with what the person did.

Break-glass is for the unplanned. For planned quiet periods, Argo CD offers a gentler tool: its sync windows declare, per project, when automated syncs are allowed or denied. Together with GKE’s maintenance exclusions and their equivalents at other providers, they are the established form of calendar-aware automation: automation that knows what day it is and holds still when the business needs stillness. Tied to Encore’s on-sale calendar, a deny window takes only a few lines:

kind: AppProject
metadata: {name: purchase-path}
spec:
  syncWindows:
  - kind: deny
    schedule: "0 8 * * 5"      # T−2h before a Friday 10:00 on-sale
    duration: 4h
    timeZone: Europe/Paris
    applications: ["*"]
    manualSync: true           # break-glass can still sync by hand

Read in plain terms, the window opens at 08:00 Paris time every Friday, two hours before the usual 10:00 on-sale, and lasts four hours, until noon. During it, Argo CD will not automatically sync any application in the purchase-path project, while manualSync: true leaves the door open for a person following the break-glass procedure.

Encore has the tool and has not wired it to the calendar. Its freeze lives instead in the delivery pipeline and in the on-sale channel, as Chapter 4 describes, which means it holds back the changes people make and leaves the agents to their own schedule. And even a sync window, had Encore configured one, would hold only what Argo CD applies. The node-upgrade reconciler is not an Argo CD application but a platform controller, and it reads disruption budgets and nothing else; it does not know an on-sale is under way, and no sync window would hold it. A freeze is only as wide as the set of loops that obey it, and it is worth listing, for every loop that can change production, which calendar, if any, it reads.

One more piece of state never belongs in the repository at all. Secrets stay in a secrets manager, and an operator in each cluster materializes the references declared in Git into the secrets the workloads use, so the repository names a secret and never holds it. Anyone who can read encore-deploy learns which secrets exist and nothing about their values. Chapter 7 takes up rotation.

Recap. IaC state is production data, the tool’s only memory of which real resource each line of code describes, and it deserves the protection we give any database. Drift is reverted, written back or deliberately ignored, depending on whether a person or another loop made it. Small stacks, moved blocks, deletion protection held by the provider and policy as code together make it hard for a destructive plan to pass review unnoticed. GitOps promotion is a chain of readable pull requests; only sync and health status say what is actually running; and an emergency change must first tell the loop to stand aside, then end by writing the change back to Git.

Exercise 3.4. Count the resources in your largest stack, and write one policy rule that would have blocked the worst infrastructure change your team has applied. Then walk through your break-glass procedure step by step: at which step would your GitOps agent undo the fix, and what in the procedure stops it?

3.5 The Platform as a Paved Road

A product with a road crew

Look back over this chapter and count the machinery: reconciliation loops and the fights between them, requests and limits, disruption budgets, cluster upgrades in waves, provider quotas, Terraform state, GitOps promotion and sync windows. Every piece of it matters, and almost none of it is what Encore’s 40 stream-aligned teams are paid to think about. A stream-aligned team is one organized around a single flow of business change, such as checkout, the seat map or notifications, and its attention belongs to that flow. If each of those 40 teams had to master the runtime as deeply as this chapter has, Encore would be paying 40 times over for the same expertise, and paying it in the currency it can least spare, which is the attention of people who should be thinking about fans buying tickets. That is why Encore has a platform at all. The Architecture book described how Encore built its platform; the question for us now is how that platform is run, day after day, by the people responsible for it.

Matthew Skelton and Manuel Pais gave the platform team’s purpose its clearest modern statement in Team Topologies. A platform team exists, they argued, to reduce the cognitive load of stream-aligned teams, by which they mean the amount of knowledge and mental effort a team must carry in order to do its work. A team can hold only so much; whatever room the runtime takes up is room the team’s own domain cannot use. From that purpose follows their second idea, the thinnest viable platform: the smallest set of services that lets stream-aligned teams deliver without deep knowledge of the layers beneath them. It is tempting to read “thin” as a matter of taste or budget, but I have come to believe it is first of all an operating virtue. Every capability a platform offers is a promise, and every promise must be run, upgraded, patched, supported at awkward hours and, one day, retired. A platform that offers 20 capabilities has made 20 such promises, and the platform team will keep paying for each of them long after the excitement of building it has faded. Thinness is how a platform team keeps its promises few enough to keep well.

The phrase paved road is Netflix’s term for this kind of platform; Spotify’s “golden path” is its close cousin, and both describe a supported, well-lit route from an idea to a running service. The metaphor deserves to be taken seriously, because it teaches more than it first appears to. A sensible city paves the routes people actually walk, and it often learns where those routes are by watching where the grass wears away. Nobody is forbidden to cut across the park; the paved path simply becomes the easiest way to go. Encore’s platform teams work the same way. They do not forbid teams from leaving the road, but they measure where teams leave it and ask why, because each worn track across the grass is a sign of a need the road does not yet serve.

The analogy breaks at one point, and the break is the most useful part of it. A city can count on its citizens needing to get from home to work; the traffic arrives whether or not the road is any good. A platform enjoys no such certainty. A stream-aligned team can always choose to build its own deployment pipeline, run its own database or reach past the platform to the cloud provider directly, and it will do so the moment the road becomes slower or more frustrating than the detour. So a platform must keep earning its traffic, release after release, much as any product must keep earning its customers. The evidence that this is hard is sobering. DORA’s 2024 report found that people using an internal platform reported higher productivity, but that platforms also came with lower delivery throughput and lower change stability, and that they helped most where developers kept their independence. The road, in other words, is good for its travelers only when they remain free to leave it, and the platform teams that understand this tend to build better roads.

Operating the platform itself

If a platform is a product, then someone must run it in production, and the obvious candidate is the team that built it. Camille Fournier and Ian Nowland, in Platform Engineering, insist on exactly this: a platform team must own the operations of what it builds, on-call included. The reasoning is the same one that makes stream-aligned teams run their own services. The people who carry the pager learn what breaks, and that knowledge flows back into what they build next. Encore holds to this rule. The Reliability team carries the pager for the platform planes, and each plane has its own SLOs, just as a checkout service does. Among them are GitOps sync latency (how long a merged change takes to reach the cluster), cluster API availability (whether teams and controllers can talk to Kubernetes at all) and the time to provision a new service (how long a team waits between asking for a service and having one). Each of these measures something a stream-aligned team feels directly, which is the point: a platform’s SLOs should describe the road as its travelers experience it.

I spent years doing platform engineering inside a large bank, and the costliest lesson I took from that work was that users judge a platform by its worst week more than by its average one. A platform that is excellent 49 weeks a year and fails badly during the other three will not be remembered for the 49. Teams who were burned during a release crunch or a quarter-end close start building their own detours, quietly, and the platform loses traffic it may never win back. This is one more reason for the platform team to hold its own pager: the worst week is something you feel most sharply when it is your phone that rings.

The same reasoning applies, with even more force, to the platform’s own automation. The node-upgrade reconciler, the GitOps agents and the policy engine all act on other teams’ services, and they often do so when nobody is watching. They are, in effect, members of the road crew who work through the night. The published practice for such automation is plain. It should roll out under the same safety rules as any other change, and windows tied to the business calendar should keep it out of business-critical hours. Encore classifies its platform automation as “low-risk,” and on most days it is. A node drain at 03:00 on a Tuesday, one node at a time and within the disruption budgets, is exactly the kind of quiet maintenance that keeps a fleet healthy. But “low-risk” describes the action on an ordinary day, not on every day. As Section 3.4 showed, the reconciler is not an Argo CD application, so no sync window would hold it, and it is exempt from the on-sale freeze. Whether it should be held to the calendar on the days that matter most, the Friday mornings when a big on-sale opens, is a question the platform group has not yet asked.

Migrations are the platform’s other permanent work, and they are where a platform’s promises are tested most directly. Every time the platform changes a base image, a cluster version, a database engine or a deployment template, some number of stream-aligned teams must move from the old road to the new one, and every such move costs them attention they would rather spend elsewhere. Fournier and Nowland treat migration and sunsetting as staffed product work, planned and resourced like any feature, much as Google runs its large-scale changes centrally rather than leaving each team to find its own way. Encore adds a rule of its own: the platform team performs the first three migrations itself before it writes the self-service guide. The reasoning is practical. A guide written in advance describes the road as its designers imagine it; a guide written after three real migrations describes the road as it really is, with the surprising dependency, the undocumented setting and the step that took an afternoon instead of five minutes. The teams that follow then travel a road someone has already walked.

Recap. A platform is a product whose purpose is to lower the cognitive load of other teams. It should stay as thin as that purpose allows, because every capability is a promise that must be run and eventually retired, and it must earn its traffic without mandates, since teams are always free to take a detour. The platform’s teams own its operations and its SLOs, and its automation is owed the same rollout rules and calendar windows as any other change. Migrations are staffed product work, and the platform team does the first ones itself so that the guide describes the real road.

Exercise 3.5. Pick one capability that your platform, or your cloud provider, offers your team. Who is on call for it, and what is its SLO? Does it respect any calendar of yours, such as a release freeze or a peak trading period, or does it act on its own schedule? When was the last migration it forced on you, and how much did that migration cost your team? If you cannot answer these questions, write them down and send them to the capability’s owners.

Kata: The Lunch Rush

Your brief: “Forkline.” Forkline is a food-delivery marketplace that connects 18,000 restaurants and 25,000 couriers across six countries. It reaches them through three pieces of software: a customer app, a courier app and a tablet that sits on each restaurant’s counter. Its demand has the shape every food business knows, quiet most of the day and then a sharp rise at lunch and again at dinner, and those meal peaks are the hours that matter most. Sixty-five engineers in nine teams build the product, and a platform team of four looks after everything underneath it.

The machinery underneath has aged. Forkline runs three Kubernetes clusters, each built by hand, and all three are two minor versions behind; in five months the provider will upgrade them by force, whether Forkline is ready or not. Its infrastructure lives in a single Terraform stack of 1,700 resources whose plan takes 25 minutes to run, and because nobody wants to wait that long for a small fix, engineers make their fixes in the cloud console instead, so the code drifts further from the world each week. The order webhooks, which carry each new order to the restaurant’s point-of-sale system, run as serverless functions, and last month they were throttled when a tablet update made every restaurant reconnect at once. And every weekday at lunch there is a quiet tug-of-war over the dispatch service, which assigns couriers to orders: its HPA scales it on load, an engineer habitually scales it up by hand at 11:30, and nobody is sure which of them wins.

Each deliverable below applies one part of this chapter to Forkline, roughly in the order the chapter developed them. Read together, they should describe a single runtime in which every lever has one owner, every change has a bounded blast radius, and the four people who run the platform can still sleep.

Deliverables:

1. Controller census — List every loop and every human that changes the dispatch service’s replica count, its nodes or the infrastructure beneath it. For each, record the field it acts on, the set point it steers toward and where that set point is written down, and how quickly it acts. This is Exercise 3.1 at the scale of a company, and it should make plain whether the 11:30 scale-up and the HPA are claiming the same field, as Catalog’s engineer and Argo CD once did at Encore.
2. Upgrade plan — Show how Forkline reaches a supported Kubernetes version before the provider’s deadline arrives. The plan should cover the PodDisruptionBudgets that keep drains from taking too many pods at once, the graceful shutdown that protects the requests inside each pod, the order in which the three clusters move, and the maintenance windows that keep every upgrade clear of the lunch and dinner peaks. Aim for a plan that does more than survive this one forced upgrade: it should make the next one routine.
3. Stack decomposition — Explain how to split the 1,700-resource stack into smaller stacks without destroying anything along the way, using moved blocks or their equivalent so that the plan shows moves rather than deletions. Name the resources that get deletion protection on the provider’s side as well as in code, and write three policy-as-code rules that would stop the most dangerous plans and manifests before anyone has to catch them by eye.
4. Managed-service survival sheet — Inventory the quotas and limits on the order path, especially those the provider sets, since one of them throttled the webhooks last month. Then decide, in advance, what the webhooks degrade to during a reconnection storm or a provider outage, how Forkline will detect that degradation with its own checks, and how it will tell restaurants and customers.
5. Thinnest viable platform — Describe what a four-person platform team should offer the nine product teams, what it should decline to offer, and the SLOs it will be held to. A platform team that small cannot run many things well, so the list of refusals matters as much as the list of services.

Rubric:

Where you now stand

You can now read a runtime as a collection of governors, each a loop that watches one part of the system and pushes it toward a set point. For any loop you can name the field it owns, how fast it acts and what it cannot see. You can keep two loops from fighting over one lever by giving each field a single owner and asking anyone else to express their intent through that owner’s set point, and you can tell a loop to stand aside when a person must act in an emergency. You can run clusters, functions, infrastructure stacks and GitOps repositories as one fleet rather than a collection of special cases, and you know where the provider’s responsibility ends and yours begins. Everything in this chapter has been about the machinery that holds a system steady. Chapter 4 turns to the changes we deliberately push through that machinery, and to why change, more than anything else, is what breaks production.

References

• James Clerk Maxwell — On Governors. Proceedings of the Royal Society of London, 1868.
• Kubernetes — Horizontal Pod Autoscaling and Disruptions. Documentation — free online.
• Argo CD — Automated Sync Policy and Sync Windows. Documentation — free online.
• Amazon Web Services — Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region. 2025 — free online.
• Amazon Web Services — Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region. 2017 — free online.
• Google Cloud — Node pool upgrade strategies. Documentation — free online.
• HashiCorp — Refactoring modules. Terraform documentation — free online.
• Bilgin Ibryam, Roland Huß — Kubernetes Patterns, 2nd ed.. O’Reilly, 2023 — free online.
• Kief Morris — Infrastructure as Code, 3rd ed.. O’Reilly, 2025.
• OpenGitOps — GitOps Principles. CNCF, 2021 — free online.
• Camille Fournier, Ian Nowland — Platform Engineering. O’Reilly, 2024.
• Matthew Skelton, Manuel Pais — Team Topologies, 2nd ed.. IT Revolution, 2025.
• DORA — Accelerate State of DevOps Report 2024. Google Cloud, 2024 — free online.