Building Resilient Systems in the AI & Cloud Era

← the whole book on one page

Chapter 7: Security Operations — Secrets, Patching, and the Supply Chain

“Security is a process, not a product.” — Bruce Schneier

A system can be designed securely and still be run insecurely. The two are different achievements, and it is easy to mistake the first for the second. The Architecture book did the design work for Encore: it drew the trust boundaries, decided which parts of the system may talk to which, and gave every workload an identity of its own. That work matters, but a diagram is a statement of intent, and intent does nothing by itself. A diagram does not rotate a key when the key gets old. It does not rebuild a container image when a flaw is found in a library inside it. It does not notice that a partner’s credential, which has always been used from the same few places, is suddenly being used from a country it has never come from before. Those things happen only because someone, or some machine that someone set up and still watches, does them. This chapter is about that work. It asks what security looks like as daily practice: who does it, how often, with which instruments, and how we can know it is still being done on the days when nobody is watching. That last question is the hardest of the four, and I have come to believe it is the one that separates a security program from a security document.

The question is much older than the cloud. On the evening of 2 November 1988, a program written by Robert Tappan Morris, a graduate student at Cornell, was released onto the young internet from a machine at MIT. Within a day, thousands of computers had slowed to a crawl. What makes the episode instructive is how ordinary it was. The worm used no exotic technique and no secret weapon. It walked through doors that administrators had left open in the normal course of running their machines: conveniences left switched on, software left unpatched, trust granted between machines and never taken back. The people who fought it had no team to call and no playbook to follow, so they improvised, comparing notes over mailing lists and working out together what the program did and how to stop it. That improvised, collective response was the first real attempt at what we now call security operations: the continuing, routine work of keeping a running system’s defenses current, and of noticing and responding when they are tested. We will look closely at the worm’s four doors in Section 7.1, because nearly every lesson in this chapter is already present in them.

Encore, for once, begins a chapter in reasonable shape. Its services carry cryptographic workload identities, so each service can prove which service it is rather than being trusted because of where on the network it sits. Its service mesh enforces mutual TLS, so traffic between services is encrypted and both ends verify each other. And its golden path, the paved road that most teams use to build and ship, is able to sign container images, so that what runs in production can be traced back to a build Encore trusts. The architecture, in other words, is sound. What Encore lacks is the habit of exercising it. Nobody yet rotates, patches, revokes and rehearses as a matter of routine on an ordinary Tuesday. That habit is what this chapter sets out to build, and the reason is practical: when the extraordinary Friday arrives, with a leaked key or a critical vulnerability across the whole fleet, the response will be only as good as the routines the teams have already practiced. A drill rehearsed during a quiet week is something people can carry out under pressure; a procedure read for the first time during an emergency usually is not.

7.1 Security as an Operational Discipline

Four open doors

The worm had four ways into a machine, and each holds one of this chapter’s lessons in small and legible form, so it is worth taking them one at a time. The first was the debug mode of sendmail, the program that delivered the internet’s mail. Debug mode was a convenience for administrators, and in many installations nobody had switched it off; the worm used it to persuade the mail system to run the worm’s own code. The second was a buffer overflow in fingerd, the small service that answered questions about a machine’s users. fingerd read its input into a fixed-size space in memory without checking the input’s length, so a carefully overlong request could spill past that space and overwrite the instructions the program would run next. The third door was rsh and rexec, the remote shell services, through which one machine could let users arriving from another machine it trusted log in without a password. Once the worm held one machine, every machine that trusted it was open as well. The fourth was the plainest of all: the worm guessed passwords, trying a short dictionary of common words, and on a network of busy people that was often enough.

What turned this program into an event was a single decision about how it should behave once it arrived. Morris seems to have intended the worm to spread quietly. Before installing itself, the worm asked each host whether it was already infected, but it ignored a “yes” about one time in seven and installed another copy anyway. The arithmetic of that choice is unforgiving. A machine on a busy network would be found again and again by its neighbors, and each visit had a one-in-seven chance of adding another copy, so copies accumulated on every host faster than anyone could clear them, and the machines slowed until they were of no use to their owners. The quiet crawl Morris may have imagined became a flood across perhaps 6,000 machines. Eugene Spafford’s analysis at Purdue remains the clearest account of how it all worked. The response tells us as much as the attack does. Many sites cut themselves off from the network, which stopped the worm but also stopped the fixes, since those fixes were traveling over the same network. Within weeks DARPA funded the Computer Emergency Response Team at Carnegie Mellon, now CERT/CC, so that the next time there would be someone to call. That is the first lesson of security operations, learned in its first week: when defense depends on coordination, the people coordinating need a place to meet that the attack cannot take away.

The four doors are worth tabulating, because each one has a living descendant in the systems we run today. The table reads from left to right as a line of inheritance: what the door was in 1988, the general kind of weakness it represented, the form that same weakness takes now, and the routine that keeps it closed, with the section of this chapter that develops each routine.

Figure 7.1 — The Morris worm’s four doors and their modern descendants. Each door was opened by ordinary operation rather than by bad design, which is why each needs an operational answer.

The adversary is part of the environment

Look again at the second column of Figure 7.1, and notice what is missing from it. None of the four doors was an architectural failure in the sense of the Architecture book, where a poor boundary lets one failure spread across a system. Each door came instead from the ordinary business of running machines day after day: a convenience enabled for a good reason and never disabled, a binary installed once and never updated, a trust granted between two machines for a busy weekend and kept for years because nothing ever forced anyone to revisit it. Nobody designed those weaknesses. They accumulated, the way dust accumulates in a building nobody is assigned to clean. That is why the answers in the last column are routines rather than designs, and it is why this chapter treats security as something a team does every week rather than something an architect draws once.

There is, however, one thing that sets security apart from the reliability work of the earlier chapters, and it is the adversary. In the chapters on capacity, resilience and recovery, the forces arrayed against us were indifferent. A failing disk does not study our defenses, and a traffic spike does not change its shape because we added a cache. An attacker does both. A defense that works this month also teaches its opponent what to try next month, so every success is partly a lesson handed to the other side. This makes security a contest that never reaches a stable end, and the right posture toward it is a practice kept up indefinitely rather than a wall built once and left to stand.

The familiar image of security as a fortress, with high walls and a guarded gate, gets this wrong, because a fortress is finished when its walls are built. I have come to believe that a hospital’s infection control is the more faithful analogy. Infection control is not a building. It is a set of routines (hand washing, sterilization, isolation, surveillance of which infections are appearing on which wards) performed constantly by ordinary staff rather than by a special squad, and tightened deliberately whenever a new organism arrives. Its successes are invisible, since what it produces is the infection that did not happen, and it decays the moment people stop performing it. All of that is true of security operations. The analogy fails at intent. Bacteria evolve, but they do not choose; they do not time their attack for a hospital’s busiest night. Human adversaries do exactly that, and for a company like Encore the busiest night is precisely the one they will choose.

Encore knows this adversary personally. Bot Screening, the service that decides which sessions in the online waiting room belong to humans and which to automated buyers, tunes its rules weekly, because the people it screens rehearse. They try their tools on minor on-sales, where little is at stake and much can be learned, and bring what works to the major ones. Bot Screening is therefore never finished, in exactly the way infection control is never finished. One caution follows from this, and the Operations book’s account of the Vesper Hale on-sale makes it vivid. A rule that judges behavior is only as current as the behavior it was tuned on, and adversaries are not the only ones whose behavior changes; fans change how they behave too, and a rule tuned against last month’s bots can mistake this month’s anxious fans for an attack.

If security is a routine practiced every week, then the natural question is who practices it. Encore organizes as Google’s Building Secure and Reliable Systems suggests, treating security and reliability as one discipline rather than as two departments with separate goals. A Security Engineering team of six sits in the platform group. It owns the security planes of the paved road, the identity, secrets and signing services that every team builds on, and it runs detection; it does not act as a gate that every change must pass through. The everyday chores of security, such as keeping an inventory of what runs where, rotating credentials and applying patches, belong to the teams that own the services. Those chores sit on the same scorecard as the teams’ service level objectives (SLOs), so a stale credential is measured and discussed the same way as a missed latency target. The trade-off in this arrangement deserves to be stated plainly. Spreading security across every team puts the work where the knowledge is, since the team that runs a service knows best which credentials it holds and which libraries it depends on. But the arrangement holds only while the platform really supplies the secure defaults, so that doing the right thing is also the easiest thing. Without those defaults, security that is shared by every team tends, in practice, to end up owned by none.

Recap. The Morris worm entered through four ordinary doors: a convenience left on, an unpatched bug, trust granted by network location and a guessable password. Each has a modern descendant, and each is closed by a routine kept up over time rather than by a single design. Because the adversary adapts, security is a continuous practice, owned by the teams that run the services and made workable by a platform that ships secure defaults.

Exercise 7.1. For a system you run, fill in Figure 7.1’s last two columns: name today’s descendant of each of the four doors in your system, and the routine (with how often it runs) that keeps that door closed. Which door has no routine at all?

7.2 Identity, Secrets, and Least Privilege in Practice

Credentials that expire on their own

Of the worm’s four doors, the two that matter most here are the trusted host and the guessed password, because both were failures of identity: the machines at the far end could not tell who was really knocking, and once they had decided to trust someone, they trusted that someone indefinitely. The Architecture book answered the second half of that problem with an argument that is easy to state: a credential which expires soon is worth little to a thief, because by the time the thief has found it, copied it and worked out where to use it, it has already stopped working. Operating that idea, day after day and across an entire fleet, turns out to be much harder than stating it, and this section is about the difference.

Start with the machines. Encore’s workloads prove who they are with SPIFFE Verifiable Identity Documents (SVIDs): short-lived X.509 certificates, of the same family a web browser checks when it opens a secure site, each of which names a single workload rather than a host or a network address. The certificates are issued by SPIRE, the reference implementation of the Secure Production Identity Framework for Everyone (SPIFFE). The process works in three steps. An agent runs on every node. When a workload starts, the agent attests it, which is to say it checks the workload’s credentials with the platform it runs on, in Encore’s case by asking Kubernetes which service account the pod runs under. Having satisfied itself, the agent hands the workload a certificate naming it, for example spiffe://encore/inventory. That certificate is valid for one hour and is renewed when half its life has passed, at the 30-minute mark; both figures are SPIRE’s own defaults, which Encore has kept. When Seat Inventory calls Orders & Payments, each side presents its certificate over mutual TLS, and each knows precisely which service is on the other end, whatever pod or address the call happens to come from. This is the modern answer to rsh’s trusted hosts: trust follows a proven name, and the name expires within the hour.

A touring crew handles access to a venue in much the same way. Nobody gets a permanent pass to the building. Each night the production office issues wristbands for that night’s show, so a wristband lost in a bar after the concert opens nothing tomorrow. The analogy holds further than one might expect, because it also shows the cost: if the wristband desk fails to open one afternoon, the crew cannot get into its own venue. Where it breaks is instructive. At a real stage door, a security guard who recognizes the lighting designer can wave her through; a service checking a certificate has no such judgment, and it refuses an expired certificate as firmly as it refuses a forged one.

That is the trade every team accepts when it adopts short-lived credentials: they move risk from confidentiality to availability. We worry less about a credential being stolen and more about the issuer being able to produce new ones on time. The arithmetic for Encore is worth doing once. Because every certificate is renewed at its half-life, at any given moment the oldest certificate still in use was issued just under 30 minutes ago and has just over 30 minutes left. If SPIRE stops issuing at that moment, the first certificates expire about 30 minutes later, and services begin refusing one another’s calls; over the following half hour the remaining certificates expire in turn, until nothing in the fleet will talk to anything else. The resulting outage would be both total and self-inflicted, caused by the very mechanism meant to keep the system safe. This is no hypothetical failure class: in December 2018 an expired certificate in Ericsson software took down O2’s mobile data network in Britain for most of a day.

Encore draws the obvious consequence. A dependency on which every service depends must be more reliable than any of those services, so Encore runs the certificate issuer as a platform plane with an SLO stricter than that of any service it serves. It runs SPIRE servers in both regions, so that losing one region does not stop issuance in the other. And it alerts when any certificate passes 75% of its lifetime without having been renewed. The threshold is chosen with care: a certificate at 75% of its hour is 45 minutes old, which means its renewal at the 30-minute mark has already failed and 15 minutes have passed since; the alert therefore fires while there are still 15 minutes left before the certificate expires, long enough for a human to act before services start refusing calls.

People are the other half of identity, and here the worm’s password dictionary has a modern descendant in the phishing page. A one-time code from an authenticator app protects against a guessed or reused password, but not against a convincing fake login page: the fake page asks for the password and the code, and the attacker simply relays both to the real site within the code’s short validity window. A FIDO2 security key or passkey closes that gap because the credential is bound to the domain it was registered with. The browser will only produce a valid signature for the real site’s domain, so the fake page has nothing worth relaying. For that reason anyone at Encore with production or administrative access signs in with a FIDO2 security key or passkey, and Encore is in good company: US federal agencies have been required to do the same since 2022.

The secrets that remain

Workload identity removes most secrets, but never all of them, because some credentials are issued by parties outside Encore’s own identity system and outside its control. Encore still holds credentials from its two payment service providers (PSPs), the signing keys for the passes fans keep in their phone wallets, and the API keys that partners use to call api.encore.com. These secrets live in a secrets manager and are injected into workloads at runtime, rather than baked into images or configuration files. Each has an owner, a rotation interval and a written rotation procedure, and of these the procedure matters more than the interval.

The reason is the same one Chapter 6 taught us about backups. A rotation never performed is a restore never tested: both look reassuring on paper, and both tend to fail at the moment they are finally needed. A rotation usually fails for an ordinary reason. Some consumer read the secret once when it started and cached it in memory, so the new value sits in the secrets manager while the running process keeps presenting the old one, until the old one is revoked and the process starts failing. Only a rehearsal finds that consumer, which is why Encore cares less about how often a secret is scheduled to change than about whether the team has actually changed it and knows, step by step, what that takes.

Encore learned what one of these remaining secrets can cost when a contractor working for Rowmate, one of its resale partners, committed Rowmate’s production API key to a public repository. The key was 26 months old. Two signals arrived within minutes, from two quite different directions. The first came from the code host: its secret scanning recognized the key because Encore had registered its key prefix (enc_live_) with the scanning service, so any string with that prefix appearing in public code triggers a notice. The second came from Encore’s own systems: Bot Screening noticed the same key calling the availability endpoints at 40 times Rowmate’s usual rate, and from a cloud region Rowmate had never used. The key was revoked 38 minutes after the commit. That sounds quick, but it is slow beside the honeypot studies, in which researchers deliberately leak keys and find them tried within minutes.

Encore has since turned on push protection, which rejects a commit containing a recognizable secret before it ever reaches the repository, and it now guards all of Encore’s own repositories. Push protection could not have helped in this case, however, because the commit was made by Rowmate’s contractor in a repository Encore does not control. Detection was fast and revocation was reasonably fast, yet neither is what limited the damage. What limited it was a decision made years earlier, when the key was issued: the key’s scopes.

The key carried two permissions, listings:read and resale:write. It could not buy primary inventory, read fans’ personal data or touch payouts, so whoever held it could look at listings and post resale offers, but nothing more. The gateway’s audit log showed exactly what that amounted to: 14 minutes of scraping and two fraudulent resale listings, which were removed before anyone bought them. Had the key been issued with broad access, as convenient keys often are, the same 38 minutes would have read very differently.

Revocation was the easy part; rotation was harder, and it shows why the written procedure matters. Rowmate needed a deployment of its own software to switch to a new key, and that took time. Revoking the old key at once would have cut off a legitimate partner along with the attacker. So for an hour both keys were valid, with the old one restricted to Rowmate’s own network addresses, so that the attacker lost access while Rowmate kept it.

The deeper fix was to stop issuing partners a long-lived key at all. Partners now use OAuth client credentials bound to a client certificate (RFC 8705, which open-banking profiles also require). In practice the partner exchanges its certificate for an access token that lives 15 minutes, and the token is accepted only on a connection made with that same certificate. A token copied into a public repository is therefore useless on two counts: it is tied to a private key the thief does not have, and in a quarter of an hour it expires anyway. Like every security improvement, this one moved a cost somewhere, in this case to the partners, who must now issue, store and renew certificates of their own. For partners who cannot, Demonstrating Proof of Possession (DPoP, RFC 9449) offers a lighter route to the same property: the client holds a key pair and signs a small proof with each request, and the token is bound to that key, so a stolen token without the key is worthless.

Figure 7.2 — How long each of Encore’s credentials lives, on a logarithmic scale, set against the 38 minutes it took to revoke Rowmate’s key. A token of the kind partners use now would have expired on its own before anyone revoked it.

Least privilege drifts

Scopes limited the damage from Rowmate’s key, and scopes are one instance of least privilege: the principle that every person and workload should hold only the permissions its work requires. The principle is easy to apply on the day a role is created and hard to keep afterward, because privilege only ever drifts in one direction. During an incident someone grants a service account a broad permission to get things working, and nobody removes it afterward, because removing a permission is risky and leaving it is free. Service accounts outlive the services they were made for. Each grant is reasonable on the day it happens; it is the accumulation that is dangerous, and it ratchets upward because nothing in daily work pushes it back down. Least privilege, then, is granted once and eroded continuously.

Encore reverses the drift using evidence of actual use, as the cloud providers’ access analyzers do. Access logs record which permissions each role actually used in the last 90 days. Every quarter, the team that owns each role receives a proposed policy with the unused permissions removed, and it may accept the proposal, amend it, or defend a permission it wants to keep. The defense matters, because evidence of use has a blind spot: a permission needed only for regional failover looks unused for most of the year, and removing it would quietly break the one procedure the team most needs to work when it is finally invoked. The quarterly review puts the decision in front of the people who know which permissions are dormant and which are dead.

All of this tightening raises an honest question: what happens when an engineer, in the middle of an emergency, needs a permission the tightened policy no longer grants? The exception every organization needs, and few design in advance, is break-glass access: a pre-arranged, audited path by which a human gains elevated access to production in an emergency. The name evokes the fire-alarm box behind a pane of glass, and the analogy is apt in one important respect. Anyone may break the glass, but nobody can do it quietly, and the noise is the point. Where the analogy breaks is that a fire alarm needs no second opinion, whereas production access, in the wrong hands, is itself the emergency.

At Encore, break-glass access therefore requires a second engineer’s approval and expires after 60 minutes. Using it posts a message to the security channel and opens a review ticket, so that every use is seen at the time and examined afterward. The cloud providers’ guidance adds the condition that matters most: the path must work when normal sign-in is the thing that has failed. An emergency path that runs through the same single sign-on as everything else will be unavailable in precisely the emergency where it is needed most. So Encore’s emergency accounts sit outside single sign-on, and the team exercises the path at every quarterly game day, for the same reason it rehearses its rotations: a path nobody has walked recently is not a path anyone can count on.

Recap. Short-lived certificates remove most secrets, but they turn the issuer into a dependency that must be more reliable than anything that relies on it. Humans need sign-in that a fake login page cannot relay, and the secrets that remain need owners, rotations that have actually been rehearsed, and scopes narrow enough that a thief gains little. Least privilege drifts upward unless evidence of actual use turns it back, and break-glass access must be loud, brief and independent of the systems it exists to rescue.

Exercise 7.2. Pick one long-lived secret your system depends on and rotate it in a staging environment, timing each step. Which consumer broke, and why? Then ask what would have happened if you had done the same rotation in production at 3 a.m., under pressure, because the secret had just leaked.

7.3 Vulnerabilities, Patching, and the Fleet

Severity is not risk

Of the Morris worm’s four doors, the overflow in fingerd is the one that most resembles our daily work: a known class of bug, sitting in a binary that thousands of machines ran, waiting for someone to update it. Its modern descendant is a flaw in a library the whole fleet shares, and the story that best shows how such a flaw becomes a catastrophe is a recent one. In March 2017 the Apache Struts project published a fix for a remote code execution flaw. Roughly two months later, attackers entered Equifax through that same flaw and exposed personal data on about 147 million people. It is tempting to read this as a story about a company that did not patch, but the more useful lesson is subtler and more uncomfortable. The organization had a patching process. The process simply did not reach the system that mattered, because nobody could see clearly which systems ran what. A fix that exists, and a process that applies fixes, are worth nothing to a server the process cannot find.

Knowing where everything runs is the first problem, and Section 7.4 returns to it. The second problem is deciding which flaws deserve our urgency, because there are far more of them than any team can treat as emergencies. Public advisories, each identified by a Common Vulnerabilities and Exposures (CVE) number, arrive by the tens of thousands a year. If every one of them were urgent, none would be, and so the whole craft of vulnerability management rests on separating the few that matter now from the many that can wait. Several public instruments help with that separation, and it is worth being precise about what each one measures, because each answers a different question.

The Common Vulnerability Scoring System (CVSS) scores each flaw from 0 to 10 for how bad exploitation would be in general. It is a measure of severity: it describes the flaw considered on its own, as it might behave in a typical deployment. What it cannot say is whether anyone is actually exploiting the flaw, or whether our code ever calls the vulnerable function. A flaw scored near the top of the scale in a parsing routine we never invoke is, for us, less dangerous than a lower-scored one that sits on our checkout path and is being attacked this week.

The Exploit Prediction Scoring System (EPSS), maintained by the Forum of Incident Response and Security Teams (FIRST), answers a second question: how likely is it that this flaw will be exploited somewhere in the next 30 days? It is a probability, and like any forecast it should be read with care. EPSS knows nothing of our assets, so it cannot tell us whether the flaw is reachable in our systems. And a low score lowers the odds without ruling exploitation out, in the way that a forecast of a small chance of rain still leaves someone wet.

The Known Exploited Vulnerabilities (KEV) catalog, published by the US Cybersecurity and Infrastructure Security Agency (CISA) since November 2021, answers a third question, and the bluntest one: is this flaw already being exploited? A flaw on the KEV list is no longer a forecast. Someone, somewhere, has already walked through that door.

None of these three instruments can know two facts that only Encore can supply. The first is exposure: does the vulnerable component face the internet, or sit on the purchase path where an attacker most wants to be? The second is reachability: does Encore’s code actually reach the vulnerable function? Reachability tools, which trace call paths from our code into a library, inform that second judgment, but they do not settle it, because a path that looks dead to a static analysis can come alive through configuration, reflection or a plugin loaded at runtime.

An emergency physician would recognize this problem at once. Triage, as hospitals practice it, is the sorting of patients by how soon they will be harmed without attention and how much attention can change the outcome, rather than by how serious their condition is in general. A broken wrist waits behind a quiet patient with chest pain, although nobody would call a broken wrist trivial. Vulnerability triage works the same way: CVSS describes the disease, while exploitation evidence and exposure describe the patient in front of us. The analogy breaks at one point that matters. A patient’s condition is largely visible to the nurse who examines them, whereas a flaw’s reachability in our own code is often uncertain, so our triage must sometimes act on a judgment that may be wrong and record the reasoning, so that someone can revisit it when the evidence changes.

Encore combines these signals into a single policy, ordered first by evidence of exploitation and then by exposure:

Figure 7.3 — Encore’s triage policy, ordered by evidence of exploitation and by exposure so that urgency is spent where an attacker is most likely to arrive. The shape follows CISA’s BOD 26-04; the deadlines are deliberately tighter.

Reading the table row by row shows the reasoning behind it. The top row is the true emergency: a flaw someone is already exploiting, in a component an attacker can reach from outside. Encore’s first move there is a compensating control at the edge, such as a gateway rule that blocks the known exploit pattern, because such a rule can be in place within hours while the real fix is built and tested. It then asks whether the attacker came through before the door was closed. The second row is the same kind of flaw behind Encore’s own walls, where an attacker would first need some other way in; the fix is still expedited, but it travels through the normal pipeline rather than as an emergency change. The third row covers flaws that are severe and likely to be exploited but not yet known to be, and these ride the next scheduled rebuild of the base images. The last row is the honest answer to most advisories: the flaw is real, but our code does not reach it, so we write down why we believe that and move on. That written reasoning is what lets a later reader, or a later version of ourselves, notice when the belief stops being true.

Encore need not defend this shape alone, which helps on the day a product manager asks why a team is pausing feature work for a flaw with a middling score. In June 2026 CISA’s Binding Operational Directive 26-04 replaced the KEV directive of 2021. It now sets federal remediation deadlines by four criteria: exposure, KEV status, whether exploitation can be automated, and how much control an attacker gains. Its tightest tier allows three days and adds forensic triage, a check of whether the door was used before it was closed, and Encore borrowed that check for its own top row. Encore sets a stricter 48 hours. A deadline, however, is a promise about what the fleet can do, and Encore can make this one only because its fleet can keep it. How the fleet keeps it is the subject of the rest of this section.

Patching as a flow

The decisive move is to stop thinking of patching as an event. When patching is an event, it is something an organization does when an advisory frightens it enough: a project with a deadline, a war room and a list of servers to touch. Every such event begins by rediscovering how each service is built and deployed, and that rediscovery is where the hours go. When patching is a flow, the same work happens continuously and without ceremony, so that an emergency is simply the ordinary process run sooner.

Encore therefore never patches a running container. It rebuilds the image from an updated base and redeploys it through the ordinary pipeline, canaries included, so that a security fix travels the same road, past the same safety checks, as any other change. Base images are rebuilt weekly whether or not an advisory demands it. The service scorecard then reports the median age of each service’s running images, and the emphasis on “running” is the whole point of the measure. A fresh build that was never deployed has patched nothing, because what an attacker meets is whatever is actually serving traffic.

A touring crew understands why the weekly rhythm matters. A crew that loads out after every show and loads in at the next venue knows the rig goes together, because it put the rig together last night; a missing bolt or a frayed cable turns up on an ordinary afternoon rather than on the night of the big show. A crew that assembles its stage once a year finds all those problems at once, and under pressure. The comparison breaks in an instructive place. The touring rig is the same every night, whereas each rebuild deliberately changes its ingredients, and that is exactly why rebuilding must be rehearsed often: every change is a fresh chance for something to stop fitting, and small, frequent changes reveal the misfit while it is still small. The consequence is stark. A fleet that rebuilds every week patches in a day when it must; a fleet that rebuilds once a year discovers, on the day it must, which services no longer build.

A continuous flow carries its own danger, though, and one of Encore’s rules did not survive 2025. Dependency updates used to merge automatically whenever tests passed on a patch release, on the reasonable theory that patch releases are small and usually fix things. Then attackers showed how that theory could be turned against the people who held it. On 8 September 2025 an attacker phished the maintainer of chalk, debug and 16 other npm packages, with more than two billion weekly downloads between them, and published versions that rewrote cryptocurrency payment addresses in browsers. Days later the Shai-Hulud worm began stealing npm tokens and using them to publish infected versions of its victims’ own packages, so that each compromise seeded the next; by the end of its second wave in November, more than a thousand packages had been compromised.

The two attacks shared a shape worth naming. Each traveled as an ordinary new release, indistinguishable at first glance from the routine updates that automation exists to absorb. And most of the malicious versions were pulled within hours, once scanners and maintainers across the ecosystem noticed them. Put those two facts together and the danger becomes plain: automation that merges a release minutes after it appears is precisely the conduit such attacks need, because it acts inside the brief window before anyone has looked.

Encore’s rule is now a cooldown: a version update must have been public for three days (for patch releases) or seven (for everything else) before automation proposes and merges it, so that the ecosystem’s scanners look first. The cooldown trades a little freshness for the collective attention of everyone else who reads new releases. Security updates skip the wait, and the reason deserves spelling out. A security release fixes a flaw that is already public, so delaying it protects us from nothing unknown; it only lengthens the time during which attackers know of a weakness we have not closed. It is the one kind of release whose delay helps only the attacker. GitHub made a short cooldown Dependabot’s default in July 2026, and Encore’s configuration states its own intervals explicitly:

version: 2
updates:
  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "daily"
    cooldown:              # version updates only;
      default-days: 7      # security updates are never delayed
      semver-patch-days: 3

A cooldown and a weekly rebuild describe the flow on an ordinary week. Whether the flow survives a real emergency is a separate question, and history offers a test. Log4Shell, disclosed on 9 December 2021 with a CVSS score of 10, is the best-documented trial of patching as a flow. The hard part was not writing the fix. It was finding the library, which was often a dependency of a dependency, or bundled inside a vendor’s archive where no one had thought to look. And the first fixed release proved incomplete, so organizations that had struggled to patch once had to do it again. In 2022 the US Cyber Safety Review Board called it an endemic vulnerability, one that would linger in the world’s systems for years because so many of those systems could not easily be found or rebuilt. The lesson echoes Equifax: the speed of a patch is set less by the fix than by our knowledge of where the vulnerable component lives.

Encore’s own test came this spring: a critical remote code execution flaw in a widely used XML parsing library, added to the KEV list within a day of disclosure. That placed it squarely in the top row of Figure 7.3, with a 48-hour deadline. The first question was the one Log4Shell had made so hard, and Encore’s software bills of materials (Section 7.4) answered it in 20 minutes: 37 affected images across 23 services. A binary scan of third-party images found one more copy, inside a vendor’s monitoring agent whose supplier published no bill of materials, which is exactly the kind of hiding place Log4Shell had taught everyone to fear.

The response then followed the top row step by step. Within 6 hours the gateway’s web application firewall (WAF) blocked the known exploit pattern, buying time for the real fix, and the logs were searched for earlier use of that pattern, without result. Meanwhile the weekly-rebuild habit did its work: by hour 30, 35 images were rebuilt and redeployed, and by hour 48, all of Encore’s own. The vendor’s agent was a different matter. Encore could not rebuild software it did not build, so it confined the agent with an egress rule, limiting where it could send traffic, and the agent waited four more days for its supplier to ship a fix. The lesson Encore drew was contractual rather than technical. It now writes patch timelines into vendor contracts, because a supplier’s speed has become part of Encore’s own.

Figure 7.4 — The spring parser flaw: affected images redeployed against hours since disclosure, from the counts Encore recorded. The curve is steep where Encore owns the build and flat where it waits on a supplier.

I have come to believe that a curve of this kind is the most honest picture of a security posture we can draw. Its steep part measures what an organization has practiced; its flat part measures what it has delegated to others and must take on trust. Neither part can be changed once the flaw arrives, because both were set by the habits and contracts of the months before.

Recap. CVSS measures how bad a flaw could be in general; exploitation evidence, exposure and reachability decide how fast we must move, which is also how CISA’s directive now sorts its deadlines. Patching works best as a continuous flow of rebuilds, measured by the age of the images actually running, so that an emergency is the routine done sooner. Because attackers now hide inside ordinary releases, routine updates wait out a cooldown, while security fixes, which help us only when they arrive quickly, are never delayed.

Exercise 7.3. Take the last 20 vulnerability findings for a service you own and reclassify them with Figure 7.3. How many were treated as urgent, and how many deserved to be? Then find out how old a new release must be before your automation merges it, and decide whether that age gives the ecosystem’s scanners time to look first.

7.4 The Software Supply Chain — Provenance, SBOMs, and Signing

When the build is the target

The Architecture book told two stories that frame this section, Thompson’s warning and the xz backdoor of 2024, and there is no need to tell them again. Between them in time sits a third case, and it is the one that changed operational practice most. In December 2020 it became known that attackers had compromised the build system of SolarWinds and had inserted a backdoor into updates of its Orion software. Those updates were signed with the company’s own key and shipped to as many as 18,000 customers. Consider what each of the usual safeguards would have reported. A review of the source code would have found it clean, because the backdoor was added during the build, not in the repository. A check of the signature would have found it genuine, because the company’s real build system had applied it. And yet the product that customers installed carried the backdoor. The attack lived in the one place nobody was looking: the machinery that turns source into something shippable.

That lesson gives us the subject of this section. The software supply chain is everything that touches an artifact between a developer’s keyboard and production: the libraries we pull in, the build system that compiles and packages them, the registry that stores the result and the deployment path that runs it. SolarWinds showed that any of these links can be the one an attacker chooses, so the frameworks that grew up afterward share a single aim. Each link should produce evidence about what it did, and the next link should check that evidence before it trusts what it has been handed.

Shipping offers a faithful picture of this, and it is worth taking a little further than usual. A container leaving a port is closed with a numbered seal, and the seal number is written on the manifest. At every handover, someone compares the seal on the door with the number on the paper. If they match and the seal is intact, nobody opened the container in transit; if they do not, the container is held. The checking is cheap, it happens at every link and it needs no trust in the people who carried the box. Our build evidence works the same way. The analogy also tells us exactly where the method stops. An intact seal says nothing about whether the shipper loaded the container honestly in the first place. Signatures and provenance tell us that an artifact came, unaltered, from the build we expected; they cannot tell us that the build’s inputs were sound. SolarWinds was, in these terms, a dishonest loading behind a genuine seal, and that is why the evidence must begin as early in the chain as we can make it begin.

The pipeline that does the building is itself a chain of dependencies, as March 2025 showed. Most continuous integration today runs on platforms such as GitHub Actions, where a workflow is assembled from small reusable steps, called actions, that other people publish. Attackers compromised one of them, tj-actions/changed-files, which some 23,000 repositories used, and moved its version tags to a commit of their own. That commit printed the pipeline’s secrets into the build logs, and many of those logs were public. A tag is a movable label, so every workflow that referred to the action by tag (by a version name rather than by a fixed commit) simply ran the attacker’s code at its next build, without any change on its own side.

Encore’s golden path applies GitHub’s own hardening guidance, and each of its three rules answers one part of that attack. First, third-party actions are pinned to a full commit hash, because a hash names one exact piece of code and, unlike a tag, cannot be moved to point at another. Second, the token each workflow receives carries only the permissions its job needs, so a hijacked step can do only what that job could do. Third, the build reaches the cloud through short-lived OpenID Connect (OIDC) federation: rather than storing a cloud password in the pipeline, the build platform vouches for the running job’s identity, and the cloud exchanges that identity for credentials that expire soon. A log that leaks such credentials leaks nothing that lives long, which is the lesson of Section 7.2 applied to the build. The workflow header that carries the first and third rules is short:

permissions:
  contents: read
  id-token: write        # OIDC: cloud federation and keyless signing
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<full-commit-sha>   # pinned, never a tag

Protecting the pipeline is half the work; the other half is making the build leave evidence that later links can check. Supply-chain Levels for Software Artifacts (SLSA) is the framework that grades this evidence for the build link. Its build track, unchanged from version 1.0 to the current 1.2, climbs in three steps, and each step closes a gap that the one below it leaves open. Level 1 asks that the build produce provenance: a record of what was built, from which source, by which builder and with which inputs. Provenance at this level is useful for finding things, but anyone could have written it. Level 2 therefore asks that a hosted build platform sign the provenance, so that it can be traced to a real builder and is no longer the word of whoever ran the build. Even signed provenance is only as trustworthy as the platform that signs it, so Level 3 asks that the platform be hardened: builds must be isolated so that they cannot tamper with one another, and the signing must happen out of reach of the build’s own steps, so that a build cannot forge its own provenance. We should be honest about the standing of all this. SLSA is a consensus framework, written by practitioners and without published data on the outcomes it produces. Its practical appeal is that the lower levels are cheap to reach on hosted platforms, which already do much of the work.

Provenance is worth little unless it is signed, and for years signing was held back by the burden of long-lived keys. A key that signs releases must be generated, stored, protected, rotated and, one day, revoked, and most projects quietly decided that the burden was not worth bearing. Sigstore (Newman, Meyers and Torres-Arias, 2022), now used by npm, PyPI and Kubernetes, removed most of that burden by applying the idea of short-lived credentials to signing itself. When the build is ready to sign, it presents its OIDC identity, the same one it uses to reach the cloud, and receives a certificate valid for ten minutes that binds that identity to a freshly generated key. It signs with that ephemeral key, records the signature in a public transparency log, an append-only record that anyone can audit and nobody can quietly edit, and then the key is discarded. There is no long-lived key to steal, because none exists.

This shifts the security question from “who holds the key?” to “whose identity was in the certificate?”, and verification must ask that question precisely. The question Encore wants answered is whether this image was signed by our release workflow running from our main branch. A verifier therefore has to name both the identity issuer and the workflow. If it checks only that the signature is valid, it will accept a signature from any workflow anywhere, since anyone can obtain a ten-minute certificate for their own identity. A signature, in other words, proves who signed; only the verifier’s policy decides whether that signer was the one we trust.

The bill of materials as an index

Provenance tells us where an artifact came from. The next question, and the one Section 7.3 kept asking, is what is inside it. A software bill of materials (SBOM) is a machine-readable inventory of the components inside an artifact, together with their versions, written in a standard format such as SPDX or CycloneDX. The term comes from manufacturing, where a bill of materials lists every part that goes into a product, and the borrowing is apt: when a supplier recalls a faulty part, the manufacturer with a good bill of materials knows within the hour which products contain it.

That comparison points to where an SBOM’s value lies. It lies entirely in being queried. An SBOM filed away beside each image answers nothing on the day it is needed. Encore indexes every SBOM and joins it to the record of which image runs where, so that “where do we run this library?” becomes a query rather than a week of messages between teams. This is how the spring parser flaw of Section 7.3 was traced to 37 images across 23 services in 20 minutes.

The index, however, is only as complete as the documents in it, and an SBOM is only as complete as the tool that wrote it. Academic studies in 2023 and 2024 found that SBOM generators disagree widely with one another and miss vendored or bundled components, meaning code copied into a project or packed inside another archive rather than declared as a dependency. These are the very places where Log4j hid. That is why Encore does not rely on SBOMs alone, and why the binary scan of third-party images in Section 7.3 found the copy inside the vendor’s monitoring agent that no bill of materials had listed.

An inventory also raises the opposite problem: it will report components that are present but harmless. A vulnerable library may be included without the vulnerable function ever being called, and a team that cannot record this will rediscover it, and argue it again, with every rescan. A Vulnerability Exploitability eXchange (VEX) statement is the record of that reasoning: a machine-readable note that a given product is not affected by a given vulnerability, and why. VEX is young and its formats still compete, so Encore chose the one its tools already understand and writes OpenVEX, which its scanner reads. The last row of Figure 7.3, “record as not affected, with the reasoning”, is written in this form.

With provenance, signatures and SBOMs defined, we can see how Encore assembles them into one path. Each stage produces an artifact, a later stage checks it, and each check has a decided behavior when it fails:

Figure 7.5 — Supply-chain controls on Encore’s golden path. Every stage leaves evidence that a later stage checks automatically.

Read down the table and the container seals reappear. The lockfile records the exact hash of every dependency, so the build can check that it received what was chosen. The build signs its image and provenance, and the cluster checks that signature before anything runs. The SBOM travels with the image, so the nightly rescan and the runtime inventory can both ask what is inside. No stage takes the previous one’s word for anything.

Encore applies this path in proportion to what each service protects. Every service reaches SLSA Build Level 2. The purchase path and the identity issuer, the two places where a forged build would do most harm, are built at Level 3 on isolated builders. At the cluster’s door, the admission check rejects any image that was not signed by Encore’s release workflow. Vendor images, which Encore cannot build itself, enter through an exception path that is priced and logged, so that every exception is visible and has a cost someone agreed to pay.

Here a familiar trade-off between security and availability appears. Encore’s admission check fails closed: if the policy engine that verifies signatures cannot be reached, deployments are refused rather than allowed through unchecked. That is the right posture for security, but it means that an outage of the policy engine stops every deployment, which makes the engine as critical as the cluster’s own control plane. Encore therefore runs it replicated across zones. One further exception keeps the arrangement from trapping itself. System namespaces, where the cluster’s own components run, are exempt from the check, so that an unavailable engine cannot block the very pods that would restore it. Section 7.5 will show what happened when one of these settings had been chosen differently, long ago, and forgotten.

Dependencies, human and machine

Everything so far has been about proving that we received what we chose. It cannot tell us whether we chose well, and that is a matter of judgment about people. Each dependency means trusting its maintainers with every release they will ever publish, including the ones we accept automatically. I spent a decade maintaining a widely used open-source library, and the view from the other end is sobering: much of the world’s software rests on a few volunteers’ evenings, and on whether those volunteers stay interested, stay well and keep their accounts secure. Encore therefore asks of each new dependency whether it is maintained, and by how many people. It reads part of the answer from OpenSSF Scorecard, which checks a project’s practices automatically, and supplies the rest by looking.

Coding assistants have added a new way to choose badly. Spracklen and colleagues, at USENIX Security in 2025, found that commercial models suggested packages that do not exist at least 5% of the time, and open models about a fifth of the time. A name that a model invents can be registered by anyone. An attacker who registers such a name and fills it with malicious code, a practice now called slopsquatting, has only to wait for a developer to accept the assistant’s suggestion and install it. The risk is real: when a researcher published a harmless package under one hallucinated name, it drew tens of thousands of downloads. The defense, fortunately, is cheap. Encore’s builds resolve packages only through a curated internal proxy, as the first row of Figure 7.5 shows, and adding a package to that proxy is a reviewed change. A hallucinated name therefore fails at the build, before it can run anywhere that matters.

The assistants are dependencies themselves, and they run with more privilege than most libraries. In July 2025 an attacker used an over-scoped token in the build configuration of Amazon’s Q Developer extension for Visual Studio Code to commit a prompt to the extension’s code. The prompt told the coding agent to wipe local files and cloud resources, and the extension’s release pipeline shipped it as version 1.84.0. Only a syntax error kept it from running. The case joins the two halves of this section: it was a supply-chain attack on the build of a tool, and the tool was one that acts on a developer’s behalf. Such an extension runs with that developer’s credentials, so whatever it is told to do, it can do with the developer’s reach. Encore treats these tools as it treats any other dependency on the critical path: it pins their versions, admits them only from an allowlist and gives agents scoped credentials of their own rather than borrowing a developer’s, as Chapter 8 describes.

Recap. SolarWinds and tj-actions showed that the build itself can be the target, so each link in the chain must produce evidence that the next can check. SLSA grades how trustworthy the build’s evidence is, Sigstore makes signing cheap enough to be universal, and SBOMs pay only when they are queried and only as far as they are complete. None of this tells us whether we chose our dependencies well. Whether they are chosen by people or suggested by their assistants, they still need judgment and a curated gate.

Exercise 7.4. Pick one image running in production today. Can you show, with evidence rather than memory, which commit and which builder produced it, and which version of its largest dependency it contains? List each step you could not answer.

7.5 Detection, Response, and Security Chaos Engineering

Detection that knows what normal looks like

Everything in the sections before this one lowers the chance that an intruder gets in: credentials that expire, patches that land in hours, images that carry proof of where they came from. None of it lowers that chance to zero, and an honest program admits as much. Once we accept that some attacker will eventually succeed, the question that matters most changes. It is no longer only “how do we keep them out?” but also “how quickly do we notice that they are in?”, because every hour between entry and discovery is an hour in which the intruder works undisturbed.

Recall how the Rowmate key was caught. Nobody saw the commit happen, and nothing about the requests themselves was malformed; the key was valid and its calls were the kind Rowmate was allowed to make. What gave it away was behavior. The key was being used at 40 times Rowmate’s usual rate, from a cloud region Rowmate had never used. In other words, the detection worked because something knew what normal looked like for that particular credential, and the stolen key did not look normal. That is the general shape of good detection, and it explains why Encore writes its detections mostly against identity. Every request at Encore already carries a workload identity or a partner credential, so each identity has a history of what it usually does, and a departure from that history is a signal. Three of Encore’s detections show the pattern. One fires when a workload calls a service it has never called before, which is what an intruder exploring from a compromised pod tends to do. One fires when a break-glass session opens without a matching ticket, because the emergency path described in Section 7.2 is legitimate only when someone has declared an emergency. And one fires when an image is running that the admission controller never saw, which means something reached the cluster by a route that skipped the supply-chain checks of Section 7.4.

A stage manager calling a show offers a useful comparison. After a few weeks of performances she knows how the evening normally sounds: when the fly cues land, how long the scene change takes, which door the crew uses. She notices a stranger in the wings not because she has a list of every possible stranger, but because she knows who belongs there. Identity-based detection works the same way, by learning who belongs where and noticing the exception. The analogy breaks in one instructive place. The stage manager learned what normal looks like by being present, while a detection learns it only from the logs it can read, and if those logs reach back only a few days, the detection has no memory of normal to compare against.

That is why Encore treats its detections as engineering artifacts with the same discipline that Chapter 1 applied to burn-rate alerts. Each detection is kept as code, so it is reviewed, versioned and changed deliberately. Each is tested with synthetic events, so that we know it fires when the condition occurs and not merely that it was written. Each has an owner and a runbook, so that whoever is paged at 3 a.m. knows what the alert means and what to do next. And the logs the detections read are kept for weeks rather than days, because a baseline, the record of what an identity normally does against which unusual behavior is judged, needs history before it can tell the ordinary from the strange. A gateway that keeps three days of logs can neither establish a baseline nor answer, a week later, what a stolen credential actually did.

Detection, however, only starts the clock. Once something has been noticed, someone has to respond.

Security incidents follow the same command structure that the Operations book’s chapter on incident response describes for any incident, with an incident commander, clear roles and a single place where the response is coordinated. They differ from an ordinary outage in three ways, and each difference comes from the fact that this time there is someone on the other side.

The first difference is that the adversary may still be watching. In an outage the failing component does not read our chat; in a security incident the intruder may have access to the very mailboxes, chat channels or tickets we would normally use to coordinate. So the incident channel is restricted to the people who need to be in it, and some actions are deliberately held back until containment is ready. If we revoke one stolen credential while the intruder still holds three others, we have only told them we are coming, and given them time to use the rest before we can act on all of it at once.

The second difference concerns evidence. In an outage the natural instinct is to restart the broken thing, and usually that instinct is right. In a security incident, restarting a compromised host or wiping a container destroys exactly the evidence we need to learn how the intruder got in, what they touched and whether they are still present. NIST’s revised guidance of 2025 insists on preserving evidence before remediation for this reason. The engineer’s reflex to restore service has to wait, briefly, while memory, disks and logs are captured, and that wait feels wrong to anyone trained on outages, which is precisely why it needs to be written into the runbook rather than left to judgment in the moment.

The third difference is a legal clock. Under the General Data Protection Regulation (GDPR), a personal data breach must normally be reported to the supervisory authority within 72 hours of the organization becoming aware of it. Seventy-two hours sounds generous until we remember that it includes the time spent working out whether personal data was involved at all, what data, and how many people are affected. That is why a privacy lead belongs in the room from the first hour, not summoned on the second day once the engineers have finished their investigation.

When it comes to containment, the work of stopping an intruder from doing further harm while the investigation continues, the nature of the intrusion decides what has to be done. Where the intruder holds credentials, as in the Rowmate case, containment is mostly revocation: invalidate what was stolen and issue replacements. Where a host itself has been compromised, revocation is not enough, because we can no longer trust anything running on that machine; it must be isolated from the network and rebuilt from a known-good image. The Morris worm showed the cost of having only the second, blunter tool: in 1988 many sites contained the worm by cutting themselves off from the network entirely, which stopped the infection and also stopped the fixes from arriving. Fine-grained revocation is the modern alternative to pulling the cable.

This is also where the earlier sections of the chapter pay off. An organization that can rotate any credential within an hour can contain an incident within the hour. An organization that needs a week to rotate leaves the intruder that week to use what was stolen. The speed of containment in a crisis is decided long before the crisis, on the ordinary days when rotation was either rehearsed or put off.

Security chaos engineering

All of the controls we have described, from detections to admission policies to break-glass alerts, share an uncomfortable property. Most of the time they do nothing visible. A detection that has never fired might be working perfectly in a quiet environment, or it might be broken, and from the outside the two look exactly alike. The same is true of a policy that has never rejected anything. So how do we know our controls work?

Kelly Shortridge and Aaron Rinehart, in Security Chaos Engineering (2023), answer by borrowing the experimental method of Chapter 5’s game days and pointing it at security controls. Their central idea is that a control we have never seen trigger is not a fact about our system but a hypothesis about it. Like any hypothesis, it can be tested: we introduce the condition the control is meant to catch, deliberately and within agreed limits, and we watch whether the control does what we believe it does. The fire alarm is the familiar version of this. A building does not wait for a real fire to find out whether its alarms sound; it presses the test button and runs a fire drill. The analogy is imperfect in a way that matters, because a fire does not change its behavior after watching our drills, whereas an adversary adapts, so security experiments have to be repeated as attackers’ methods change rather than performed once and filed.

It is worth being precise about how much evidence stands behind this practice. The name “security chaos engineering” is recent, and its published evidence consists mostly of practitioner case studies, Capital One’s among them, rather than controlled research. But the practices it gathers under that name are older and well established. Canary tokens are fake credentials, files or records planted where only an intruder would find or use them, so that any use at all is a reliable alarm. Control validation is the routine of triggering a control on purpose to confirm it still responds. And purple-team exercises bring together the red team, who play the attacker, and the blue team, who defend, working side by side rather than in secret, so that every attack attempted is also a lesson in whether and how it was seen. What the new name adds is a frame: treating these older practices as experiments with stated hypotheses and limits, run on a schedule like any other game day.

Encore runs its security experiments as part of its quarterly game day, and the most recent one shows what such experiments find. It ran three. In the first, a canary key was pushed to a test repository, standing in for a leaked credential, and it was detected in four minutes. In the second, an unsigned image was deployed to a staging cell, and admission rejected it, as the supply-chain policy of Section 7.4 requires. Both controls confirmed their hypotheses. Confirmations like these are not wasted effort; they turn “we believe this works” into “we watched it work this quarter.”

The third experiment found a real gap. Its purpose was to see what happens to admission when the policy engine itself is unavailable. The admission webhook is the hook through which Kubernetes asks the policy engine whether a new pod may run, and every such hook has to decide what to do when the engine does not answer. It can fail closed, refusing anything it cannot check (the posture Section 7.4 described), which protects security at the cost of availability, or it can fail open, allowing anything it cannot check, which protects availability at the cost of security. With the engine taken down for the experiment, Encore’s webhook failed open, and an unsigned image deployed without complaint. The setting had been chosen long before, during an upgrade, to keep deployments flowing while the engine was briefly unavailable, and it had simply never been changed back.

What makes this finding instructive is how the decision was made, or rather how it was not. Nobody at Encore had ever decided that security should yield to availability on the purchase path. A setting had decided it, years earlier, for a reason that made sense for one afternoon, and some widely used policy engines still ship with that default today. No review would have caught it, because nobody reviews a value they do not know they chose. Only provoking the failure revealed it.

The fix was to make the choice deliberately. The webhook now fails closed for the purchase path, where an unverified image is the greater risk. Because failing closed makes the policy engine as critical as the control plane, the engine is replicated across zones, as Section 7.4 described, and there is a break-glass procedure for the day it fails anyway, so that a broken engine cannot freeze the purchase path with no way out. This is the trade-off the Kata’s rubric asks about: every control fails one way or the other when its dependencies fail, and the only real question is whether someone chose which way. I have come to believe that this is the most valuable thing security experiments find. A control that has never been seen to fail has only been seen to be configured.

Recap. Detection works best when it knows what normal looks like, which is why Encore writes its detections against identity and behavior, keeps them as code, tests and owns them like any alert, and keeps enough log history to form a baseline. Security incidents use the ordinary command structure with three additions: a restricted channel because the adversary may be watching, preserved evidence before remediation, and a privacy lead watching the GDPR’s 72-hour clock. Containment is mostly revocation, and its speed is decided by how well rotation was rehearsed beforehand. Security chaos engineering, a young name for older practices such as canary tokens, control validation and purple-team exercises, tests controls by provoking them, and its most useful findings are defaults that nobody remembers setting.

Exercise 7.5. Choose one security control you believe in and design an experiment that could prove you wrong about it. Write down four things: the hypothesis (what you expect the control to do), the condition you will inject to trigger it, the limit on blast radius that keeps the experiment safe, and the signal that would show the control did not work. Run the experiment outside production and record what surprised you, including any default you discovered you had never chosen.

Kata: The Spare Key

Your brief: “Hearthstay.” Hearthstay is a vacation-rental marketplace. It lists 85,000 properties from 30,000 hosts across southern Europe and takes 2.5 million guest bookings a year. Its payment flow charges guests when they book and pays hosts after the guest has checked in, so for weeks at a time Hearthstay holds money that belongs to someone else. Around the core marketplace sits a partner API, used by 40 channel managers and resellers to keep hosts’ calendars and prices in sync across the sites where a property is advertised. Each partner authenticates with a static API key, and the oldest of those keys dates from 2018. At check-in the platform also sends guests the codes that open their doors, which means a Hearthstay credential can, at the end of a long chain, unlock a real apartment.

Twenty-six engineers run about 70 services on two Kubernetes clusters. A recent audit found base images up to 500 days old, no software bills of materials (SBOMs), a single service account shared by 11 services, and a payout-admin console that anyone on the office network could reach without multi-factor authentication.

Last month a channel manager’s key turned up in a public code repository. That same week, three hosts found that the bank details for their payouts had been changed. Nobody can say whether the leaked key was the cause, because the gateway keeps its logs for only three days, so the record that could have answered the question no longer exists.

Hearthstay’s story begins where Encore’s Rowmate incident began, with a partner’s key committed to a public repository, and the comparison is the most useful way into the kata. Encore’s key was found within minutes, revoked in 38, and its scopes kept it away from payouts; its audit log then told the team exactly what the key had done. Hearthstay has none of those protections in place. Its keys are older, nothing we know of limits what they can reach, and the one record that could settle the question of the bank details has expired. The kata asks you to build, for a smaller and less prepared company, the routines this chapter built for Encore, and to decide which of them to build first. Twenty-six engineers cannot do everything at once, so the order of the work is part of the answer.

Deliverables:

1. Credential inventory and plan — list every class of credential, human and machine, with its age, its scope and its blast radius, meaning what an attacker holding it could reach. Then sequence the migration to strong sign-in for people, workload identity for services and short-lived tokens for partners, as Section 7.2 did for Encore.
2. Triage policy — Hearthstay’s own version of Figure 7.3. Include the host mobile apps, which a fix reaches only when each host chooses to update, so the deadline for closing a door there cannot be met by Hearthstay’s rebuild alone.
3. Supply-chain path — which stages of Figure 7.5 Hearthstay adopts first, the SLSA level it targets six months from now, and how its SBOMs will be queried when the next widely used library turns out to be vulnerable.
4. Detection and response pack — five detections written against identity, the log retention each one needs to have a baseline worth comparing against, and a runbook for the event that has already happened once: “a channel manager’s key is leaked.”
5. Three security chaos experiments — each with a hypothesis and the limits that keep it safe, and one of them aimed at a change to a host’s payout details, the very event Hearthstay could not explain last month.
6. Break-glass design — who may use it, how the access expires, and how it still works on the day the identity provider is the thing that has failed.

Rubric:

Three of these questions deserve a moment’s thought before you start. Rotation realism is where good plans meet partners’ release calendars: Encore had to run Rowmate’s old and new keys side by side for an hour because Rowmate needed a deploy to switch, and a channel manager that deploys once a quarter turns that hour into months unless your plan gives it a gentler path. Failure posture asks you to do deliberately what Encore discovered it had done by accident, when its admission webhook was found to fail open under a setting nobody remembered choosing; for each control, write down which way it fails and why. And proportion is a reminder that the audit’s findings are not equally urgent. The payout console without multi-factor authentication and the partner keys that never expire are the doors an attacker would try first, and a plan that starts with them will protect Hearthstay sooner than a plan that starts with whatever is easiest to fix.

Where you now stand

You can now treat security as routine work. You can run credentials that expire on their own and rotations that have been rehearsed, so that a leaked key is an inconvenience with a known procedure. You can triage vulnerabilities by evidence of exploitation and by exposure, and patch by rebuilding rather than by tinkering with what is running. You can make every link of the supply chain leave evidence that the next link checks, and you can test your controls by provoking them, because a control you have never watched fail is one you know only as configured. Every control in this chapter, though, guards a system whose parts we wrote, or at least can read, and whose behavior repeats when we ask the same question twice. The newest components Encore runs offer neither comfort. Chapter 8 turns to the models and agents Encore now runs, which are fluent, usually right and never quite the same twice, and to what it takes to operate them with the same care. The other skill this chapter leaned on, acting quickly and in concert when something does go wrong, is the subject of the Operations book.

References

• Heather Adkins, Betsy Beyer, Paul Blankinship, Piotr Lewandowski, Ana Oprea, Adam Stubblefield — Building Secure and Reliable Systems. O’Reilly/Google, 2020 — free online.
• CISA — BOD 26-04: Prioritizing Security Updates Based on Risk. CISA, 2026 — free online.
• CISA — Known Exploited Vulnerabilities Catalog. CISA, 2021 — free online.
• FIRST — Exploit Prediction Scoring System (EPSS). FIRST — free online.
• GitHub — Dependabot version updates introduce default package cooldown. GitHub Changelog, 2026 — free online.
• Kennedy Toomey — The case for dependency cooldowns in a post-axios world. Datadog Security Labs, 2026 — free online.
• CISA — Supply Chain Compromise of Third-Party GitHub Action, tj-actions/changed-files (CVE-2025-30066). CISA, 2025 — free online.
• GitHub — Security hardening for GitHub Actions. GitHub Docs — free online.
• OpenSSF — SLSA: Supply-chain Levels for Software Artifacts, v1.2. OpenSSF, 2025 — free online.
• Zachary Newman, John Speed Meyers, Santiago Torres-Arias — Sigstore: Software Signing for Everybody. ACM CCS, 2022 — free online.
• Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala — We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs. USENIX Security, 2025 — free online.
• AWS — Security bulletin AWS-2025-015 (Amazon Q Developer for VS Code). AWS, 2025 — free online.
• Kelly Shortridge, Aaron Rinehart — Security Chaos Engineering: Sustaining Resilience in Software and Systems. O’Reilly, 2023.
• CISA — Implementing Phishing-Resistant MFA. CISA, 2022 — free online.
• Eugene H. Spafford — The Internet Worm Program: An Analysis. Purdue Technical Report CSD-TR-823, 1988 — free online.