Building Resilient Systems in the AI & Cloud Era

← the whole book on one page

Chapter 6: Data, State, and Disaster Recovery

Code can be redeployed. Data can only be remembered.

Everything this book has said about change so far rests on a quiet promise: if a release goes wrong, we can take it back. Canaries, feature flags and rollbacks, the machinery of Chapter 4, all depend on that promise. A canary is safe because a bad canary can be withdrawn. A flag is safe because it can be switched off. A rollback is safe because the previous version is still in the registry, ready to run again. In every case the thing we change is code, and code is a description of behavior that we can swap for an older description whenever we like.

Data breaks the promise. A row that has been written cannot be unwritten by redeploying anything, because a redeploy replaces the program and leaves untouched everything the program has already done. Suppose a faulty release writes thousands of malformed records before anyone notices. Rolling back stops it from writing more, but the malformed records stay where they are, and the older version of the code must now live with them. Worse, the mechanism we rely on to protect data against the loss of a machine works against us here. Replication faithfully copies a corruption to every replica, so the error becomes exactly as durable as the truth it replaced. This chapter is about the part of production that remembers, and about how to make its memory survive three kinds of threat: our own mistakes, the failure of our machines, and the deliberate work of our adversaries.

The story every operations engineer eventually hears took place at Pixar in 1998, during the making of Toy Story 2. Someone ran a recursive delete, of the rm -rf kind, at the root of the film’s directory tree. Oren Jacob, then an associate technical director, later recalled watching the film disappear piece by piece, first Woody’s hat, then his boots, then Woody himself, until the team pulled the plug. By then most of the film was gone. The backups, which everyone assumed would cover exactly this case, had been failing unnoticed for some time. What saved the film was luck. Galyn Susman, the supervising technical director, had been working from home after the birth of her child, and she kept a copy on her home computer. The team drove that machine to the studio wrapped in blankets, restored from it, and then spent days finding out what the copy lacked.

The story is worth telling carefully, because it holds three lessons, and each returns later in this chapter. The first is that the deletion itself was ordinary: a routine command, run in the wrong place by someone doing normal work. That is the kind of failure Jim Gray had in mind in 1985, when he found that administration and software caused far more outages of Tandem’s systems than hardware did. We tend to build our defenses against the dramatic failure, the disk that dies or the data center that floods, while the commonest threat to our data is a person with the right permissions pointed at the wrong directory. The second lesson is that the formal plan had failed, and failed silently. Backups were being taken, as far as anyone knew, but nobody was checking that they could be read back, and a backup that cannot be read back is only a ritual. The third lesson is about what actually worked. The recovery came from a copy that was independent of the original system in every way that mattered: a different machine, in a different place, beyond the reach of the command that destroyed the original. And the people who restored it were improvising, because no one had ever practiced a restore. They succeeded, but improvisation under pressure is exactly the mode in which tired people make the next mistake. I have come to believe that a recovery we have never rehearsed is not yet a capability. It is a hope with a procedure attached.

Encore has far better machinery than Pixar had in 1998, and yet much the same exposure, because machinery protects only against the failures it was designed for. Consider four pieces of its state and practice. Its seat ledger, the event log in which Seat Inventory sells each seat exactly once, commits every write to a majority of replicas in three EU locations, so no single machine or site can lose a sale. That is strong protection against failing hardware, and none at all against a bad write, which the majority will accept as readily as a good one. Its Finance ledger, the relational books fed by every completed sale, is backed up nightly, and that backup has never been restored in full, which puts it closer to Pixar’s position than anyone at Encore would like. Its holds store is about to change shape under live traffic, the situation in which a half-finished change can leave data that neither the old code nor the new code understands. And its game days, the rehearsed failure exercises the team runs each quarter, have never failed over a whole region, so the largest recovery Encore depends on is also one it has never performed. By the end of the chapter all four will have been tested, and one of them will have failed in a way that teaches more than a success would have. (The map of Encore in Chapter 1 shows where each sits.)

6.1 Why Stateful Systems Are Different

The asymmetry of state

A stateful system is one whose correct behavior depends on what it has remembered from earlier requests, so that losing or corrupting that memory is a failure even when every process is healthy. The definition is worth reading twice, because its last clause is where the trouble lives. A stateless service is, in effect, a function: give it the same request and it computes the same answer, so we can replace it, restart it or roll it back to yesterday’s version and nothing of value is lost. A stateful service is a function plus a history. Its answer to “is seat 14 in row F still available?” depends on everything that has happened to that seat since the event went on sale, and that history runs in one direction only. Every pod can report healthy and every dashboard can be green while the system is still wrong, because what it remembers is wrong.

A touring production shows the difference well. The road crew strikes the stage after each show and builds it again in the next city, and nobody worries much, because the set can be rebuilt from its drawings as often as needed. The box office is another matter. Its record of who bought which seat cannot be rebuilt from anything; if it is lost, no amount of carpentry brings it back. Most of a modern system is stage set, and a small, precious part of it is box office. The analogy holds up to a point and then breaks in a way that teaches something: in software the drawings are themselves data, and a set we believed we could always rebuild turns out to be rebuildable only if someone kept the drawings and knows how long the build takes. We will come back to that before the section ends.

This one-directional quality shows up as an asymmetry between code and data, and the asymmetry appears in four places.

The first is rollback. A deployment can be reverted, because the old binary still exists and running it again restores the old behavior. A data change cannot be reverted in the same way. Once a migration has rewritten a million rows, and fans have bought seats against the new shape in the meantime, there is no previous version to return to; there is only a further, forward change that compensates for the first. Accountants learned this long ago, which is why a ledger is corrected by a new entry that offsets the mistaken one, never by erasure. The operational consequence is demanding: every migration must be safe if it is left half done, because a half-finished migration is a state we may have to live in for hours while we work out how to move forward. Section 6.3 turns that requirement into a technique.

The second is copying. Code is small, and moving it is nearly free. State is large, and copying it is bounded by physics: by disk throughput, network bandwidth and the time a database needs to replay its logs. Restoring 4 TB takes hours, however clever the tooling, and no orchestration makes the bytes move faster than the hardware allows. This is why recovering a stateful system is a matter of hours where recovering a stateless one is a matter of seconds, and why a recovery time objective for state has to be tested rather than assumed.

The third is corruption. When a stateless service has a bug, the bug fails requests while it is deployed, and fixing the code ends the failure. When a bug writes bad data, the damage outlives the bug. Every later reader of that data is misled, long after the faulty code has been replaced, until someone finds the bad rows and repairs them. That makes corruption the quietest kind of outage, because nothing crashes and no alert fires, and often the longest, because the clock starts with the first bad write and stops only when a person notices the consequences and traces them back.

The fourth is replication, and here the asymmetry turns a protection into a hazard. We replicate state so that losing a machine, a zone or a region does not lose data, and replication is very good at that job. But a replica copies whatever the primary writes, faithfully and within milliseconds, so it copies our mistakes too. A dropped table is dropped on every replica; a corrupting bug corrupts them all. Replication protects us from losing hardware and does nothing to protect us from ourselves, which is why the backups of Section 6.2 have to be something other than one more replica.

Taken together, these four asymmetries explain why the habits that serve us well with stateless services (deploy freely, roll back quickly, trust the replicas) become dangerous when applied to state, and why state deserves a discipline of its own.

Classifying Encore’s state

That discipline begins with knowing what we have. The first operational act for state is an inventory: a list of every place the system remembers something, what each memory is for, and what losing it would cost. Encore’s inventory is short, because most of Encore was made stateless on purpose, and most of its services can be restarted anywhere without losing anything that matters. What remains is the small, precious part, and Figure 6.1 lists it. Two of its columns use terms the Architecture book introduced: the recovery point objective (RPO) is how much recent data we can afford to lose, measured back in time from a failure, and the recovery time objective (RTO) is how long we can afford to be without the service.

Figure 6.1 — Encore’s state inventory. The third column matters most: derived data needs a tested rebuild rather than a backup, while systems of record need both a backup and a practiced restore.

Reading the table row by row shows how differently each store has to be treated. The seat ledger is the heart of the business: losing even one committed write could mean selling a seat twice, so its RPO is zero, and it is protected by the majority replication described in the chapter’s opening. Seat holds matter only briefly, since each one expires within minutes, so losing a few minutes of them inconveniences fans without corrupting anything. Orders can tolerate seconds of loss because the PSPs keep their own records of every charge, and any gap can be reconciled against them. The Finance ledger can tolerate more loss and more downtime than the purchase path, but its failure carries a different kind of cost: a failed nightly close and exposure to regulators. Each row is a separate promise with its own price, and a single company-wide “data protection policy” would overpay for some rows and underprotect others.

The third column, “System of record?”, is the one that does the most work. A system of record is the authoritative store for some fact: if it says a seat was sold, the seat was sold, and there is nowhere else to look it up. Derived data is computed from a system of record, as a cache, an index or an analytical copy, and can in principle be computed again. The distinction, which Kleppmann develops at length, takes one large and anxious question (“how do we protect our data?”) and breaks it into several small ones, each of which can actually be answered. For a system of record, the question is how we would restore it and how long the restore would take. For derived data, the question changes: where is its source, and how long does a rebuild take?

The seat-map cache is the instructive case, and it is where the stage-set analogy earns its keep. The cache needs no backup, because the seat-map geometry it holds is computed from data kept elsewhere and can always be computed again. What it does need is a rebuild whose duration is known, because while the cache is cold, requests that it would have answered fall through to slower paths beneath it, and the system is slow until the cache is warm again. “No backup needed” is easy to hear as “no risk”, and that would be a mistake; that rebuild duration matters, as the Operations book’s review of the Vesper Hale on-sale will show. I have come to believe that derived data is where teams are most often surprised, precisely because it is labeled harmless. A rebuild is safe to rely on only when two conditions hold: its source is retained, and its duration has been timed.

Recap. State moves in one direction only: we can revert code, but we can only compensate data with a further change. Copying state is slow because physics bounds it, corrupting it is quiet because nothing crashes, and replicating it spreads our mistakes as faithfully as our writes. An inventory that separates systems of record, which need a backup and a practiced restore, from derived data, which needs a retained source and a timed rebuild, is the first and cheapest act of control.

Exercise 6.1. Build Figure 6.1 for a system you know. Mark each store as a system of record or derived data, and for each derived store write down how it would be rebuilt and how long that would take. Which “derived” store turns out to have no source?

6.2 Backups That Restore — RPO, RTO, and Testing Recovery

What a backup is for

The Architecture book defined the two numbers that frame this section, and Section 6.1 used them to price each of Encore’s stores: the RPO looks back from a failure to the last data we can still recover, and the RTO looks forward from it to the moment service returns. An architect writes these numbers into a design. An operator has to treat them differently, as claims that nobody has yet tested. This section asks what has to be true for those numbers to hold on the day they are needed.

The first thing to get right is what counts as a backup. A backup is a copy of a system of record, held independently of it, from which a known past state can be restored within a stated time. Two words in that definition do most of the work. Independently rules out a replica. A replica protects us well against losing a machine, but it shares the system’s fate for logical damage: when someone drops a table or a bug writes nonsense, the replica faithfully applies the same change within milliseconds, as Section 6.1 warned. Within a stated time is what turns a copy into something we can plan around during an outage. A copy that might take a week to bring back is an archive. It may be valuable, but no one can promise service on the strength of it. The data-integrity chapter of Google’s Site Reliability Engineering puts the point plainly: nobody really wants backups; what people want are restores. A backup is only ever a means. The restore is the thing being bought.

Figure 6.2 — RPO measures back from a failure to the last good copy, RTO forward to service restored; here for the seat ledger and the Finance ledger, beside what the Finance drills of Section 6.5 measured. The promise was kept only once the team changed what had to come back first.

When we imagine needing a backup, we tend to picture a region burning down. That does happen, but the common threats are humbler, and they shape the design more. The first is a deletion in the wrong place, the ordinary mistake that nearly cost Pixar a film. The second is a bug that writes bad data quietly for a week before anyone notices. The third is an adversary who encrypts the database and then goes looking for the backups, so that the victim has nothing to restore from and must pay. Each threat asks something different of the copy. Slow corruption needs a copy from before the bug began, which means the backups must reach far enough back in time. Ransomware needs a copy the attacker cannot reach, which means the backups must sit outside the credentials the attacker has stolen. A burning region needs a copy somewhere else. No single mechanism meets all three demands, and the table below sets out the usual candidates side by side.

Reading down the “Weakness” column shows why every row needs another row to cover it, and why serious data protection is always built in layers. The SRE book starts with the cheapest layer, soft deletion. A deleted row is not removed at once but marked and kept for a grace period, so the common slip can be undone in minutes. Its operational cousin is the delayed replica, a replica deliberately held some time behind the primary (PostgreSQL’s recovery_min_apply_delay, MySQL’s SOURCE_DELAY). If someone drops a table, the delayed replica, an hour behind, still holds it, and we can copy it back before the drop reaches the replica too. The protection lasts only as long as the delay. Once the hour has passed, the replica has caught up with the mistake.

For corruption that runs longer than any delay, the workhorse is point-in-time recovery. We restore a physical base backup, a full copy of the database files taken at some moment, and then replay the write-ahead log, the database’s own record of every change, forward to an instant we choose. That might be 14:02 last Tuesday, just before the bad deploy began writing nonsense. Two limits come with this power. It reaches back only as far as the log is retained, often a week to a month, so a bug that goes unnoticed for longer than that cannot be undone this way. And it rewinds the whole database. Every good write made since 14:02 on Tuesday is discarded along with the bad ones, and someone must then reconcile them, finding the legitimate orders and edits of the intervening days and putting them back. Logical dumps, which export tables or schemas as data rather than as files, are slower but finer-grained, and that granularity can matter more than speed. At Encore each Professional tenant has its own database, so restoring one tenant’s database from a dump is far gentler than rewinding a cluster that many tenants share and making all of them pay for one tenant’s accident.

Three copies, two media, one elsewhere

Layers protect against different threats only if they do not all fail together. The oldest compact statement of that idea is the 3-2-1 rule, usually credited to the photographer Peter Krogh: keep three copies of your data, on two kinds of media, with one copy off site. It was framed for people keeping photographs on hard disks, yet it has aged remarkably well, because underneath the specifics it is a rule about independence of failure. Two kinds of media means one flaw in a disk model or a tape format cannot take every copy. Off site means one fire or flood cannot take every copy either. The numbers matter less than the question they force: what single event would destroy all my copies at once?

In the cloud the question survives but the answers move. “Medium” and “site” become accounts, regions and, above all, credentials. A backup stored in another region but deletable by the same role that administers the database is not independent of the database in the way that matters most. When an attacker steals that credential, they can destroy the database and its backups with the same hand. The most instructive real case runs the other way, showing what independence looks like when it happens by accident. When the NotPetya malware struck Maersk in 2017, recovery hinged on a domain controller in Ghana that a local power cut had kept offline. That copy survived because it was unreachable, and it was unreachable only by luck. Like Pixar’s home computer, it was the independent copy no one had planned. The lesson is to plan for one.

Encore plans for it this way. Backups of systems of record go to a separate account, into a vault under a compliance-mode retention lock. Compliance mode means that for 35 days nobody can delete a recovery point or shorten its retention. “Nobody” here includes Encore’s own administrators and the account’s root user, and no approval process can change that. The lock is deliberately beyond anyone’s reach, because any key that opens it could be stolen. Everything that must stay adjustable, from the backup plans and the vault’s access policy to the list of who may restore, sits instead in governance mode. Changes there are possible but require multi-party approval, so that no single person, and no single stolen credential, can quietly weaken the protection. The encryption key lives in the backup account as well. This detail is easy to miss and fatal to get wrong: a backup encrypted with a key that was destroyed alongside the source is a perfectly preserved copy that no one can ever read. A backup whose key died with the source is no backup.

Restore testing

Independence protects the copy from the disaster. It does not prove that the copy works. A backup has two failure modes, and only one of them is visible. In the first, the backup job fails, and an alert can catch that. In the second, the job succeeds and reports success, but what it produced cannot be restored, because it is empty, truncated, incompatible or unreadable. Only an actual restore catches the second failure, and it is the more dangerous one, because it looks exactly like health until the day it is needed.

GitLab’s outage of January 2017 is the modern case, and its public postmortem is unusually candid. Late at night, an engineer working to repair database replication wiped the data directory on the primary database instead of the lagging secondary. The team then turned to their backups and found them wanting, one after another. The daily dumps had been producing nothing, because of a version mismatch between the dump tool and the database, and the emails reporting that failure were themselves being rejected, so no one saw them. Disk snapshots had never been enabled for the database servers at all. What saved the company was a snapshot taken by hand six hours earlier, for a different purpose: refreshing a staging environment. Those six hours of data, touching some 5,000 projects, were lost, and copying the snapshot back took most of a day. GitLab’s postmortem traced the root cause not to the mistyped command but to ownership: nobody had been responsible for testing the backup procedures. Every mechanism existed on paper, and no person’s job was to find out whether any of them worked.

The echo of Pixar, nearly 20 years earlier, is hard to miss: backups failing unnoticed, a mistaken delete, and rescue by a copy made for some other reason. The past repeats because the failure is structural. Backups are written once and then trusted. Nothing in ordinary operation exercises them, so nothing reveals that they have quietly stopped working. One practice answers each of the two failure modes. The first is to alert on a backup’s absence, rather than waiting for a failure report, since a job that fails badly enough may never manage to report anything, as GitLab’s rejected emails showed. The rule below fires when the last successful backup of any system of record is more than 26 hours old, which is a daily schedule plus two hours’ grace for a slow run:

- alert: BackupStale
  expr: time() - backup_last_success_timestamp_seconds{kind="system_of_record"} > 26 * 3600
  for: 15m
  labels: {severity: page}
  annotations:
    summary: "No successful backup of {{ $labels.store }} for over 26 hours"

The second practice addresses the invisible failure mode, and it is continuous restore testing: restores run on a schedule into an isolated environment, then verified against invariants. Verification is the part teams skip, and it is the part that matters, because a database that starts up can still be missing a day of data. The checks are things that must be true of a correct copy. Row counts should be close to the source’s. Totals that must balance, such as money in against money out, should balance. The latest transactions known to exist should be present. Encore restores the Catalog and every Professional tenant database weekly, checks each against its source and records how long the restore took.

That last measurement is worth dwelling on. The duration of a scheduled restore is a service level indicator (SLI), a measured signal of how the system is actually doing, and it gives the RTO something it otherwise lacks: evidence. Data grows steadily, and restore time grows with it, so an RTO that was comfortable when it was agreed can become impossible without anyone changing a line of configuration. A weekly restore duration that creeps upward month by month is the earliest warning that a promise is drifting out of reach, and it arrives while there is still time to act, by splitting the data, provisioning faster storage or renegotiating the promise. Section 6.5 shows what happens to a store that never had this signal.

A backup that has never been restored is a rumor about your data.

Recap. A backup is an independent copy that restores a known state within a stated time. Each threat, whether a slip, slow corruption, ransomware or a lost region, needs its own kind of copy, and the most important of them is locked, kept with its encryption key, and held in another account beyond the reach of any single credential. Watch backups for freshness by alerting on their absence, and prove them with scheduled, verified and timed restores, whose duration tells us whether the RTO is still true.

Exercise 6.2. For your most important database, write down the date of the last full restore, how long it took and who did it. If you cannot answer, schedule one. If you can, compare its duration with your stated RTO and with how much the data has grown since.

6.3 Schema Changes and Data Migrations

Two deploys and a promise

Section 6.1 argued that data runs in one direction only, and nowhere is that felt more sharply than when we change its shape. A schema change is a deployment whose effects cannot be undone by redeploying. If we ship a bad build, we roll back to the previous one and the world is as it was. If we drop a column, rename a table or rewrite a million rows into a new format, rolling back the code does not bring the old data back; the previous build simply wakes up in a world it no longer recognizes.

A second fact makes the problem harder. During any rolling deployment, two versions of the code run side by side: some pods have the new build, some still run the old one, and for minutes (or, with mobile clients and long-lived workers, for weeks) both are serving real requests against the same database. Both versions must work against the data at every moment. Consider the simplest change imaginable, renaming a column. If we rename it and ship the code that uses the new name in the same release, then for the length of the rollout every old pod is querying a column that no longer exists, and every one of those requests fails. Reverse the order and the new pods fail instead. There is no instant at which schema and code can change together, because there is no instant at which only one version of the code is running.

The answer, refined by many teams over many painful releases, is expand and contract (also called parallel change): a schema change made in stages, first expanding the schema so that it supports both the old and the new code, then migrating the code and the data, and only when nothing uses the old shape any longer, contracting it away. For the column rename, that means adding the new column alongside the old one, deploying code that writes to both and reads from the old, copying the existing values across, switching reads to the new column, and finally, in a separate later deploy, removing the old column and the code that still mentions it. It takes at least two deploys, and between them stands a promise: for as long as both shapes exist, both stay correct. Each stage is small, and each, until the last, can be reversed by a flag or an ordinary rollback. What we buy with the extra ceremony is the thing data otherwise denies us, a way back.

The civil engineer’s version is replacing a bridge without closing the road. We build the new span alongside the old, shift traffic across lane by lane while watching how the new structure carries it, and demolish the old span only when the last car has crossed. At every point before demolition, a problem with the new span means sending traffic back to the old one. The analogy teaches the order of operations well, and it breaks at the end, in an instructive place. Once the old bridge is gone, no driver needs it; nobody arrives tomorrow expecting to cross a span that was demolished last week. Software has readers with long memories. Caches hold values in the old shape, stream consumers replay messages written months ago, mobile clients update when their owners please, and at Encore the event-sourced seat ledger will read its past events in their original shape forever, because replaying those events is how Seat Inventory rebuilds its state. Contraction is therefore the one step that deserves real suspicion. It is safe only when we can show, with evidence rather than hope, that no reader of the old shape remains.

The holds store, migrated under traffic

Encore’s own example is the seat holds. A hold is the short reservation a fan receives on picking a seat, which keeps it off the market for five minutes while the fan checks out. Holds lived in a relational table keyed by (event_id, seat_id), and a sweeper job deleted expired rows every few seconds. The design was simple, but at on-sale scale the sweeper had become a hot spot: a job doing bulk deletes against the very table that fans were writing to as fast as the Sale Gate admitted them. The On-Sale and Seat Inventory teams decided to move holds to a key-value store with native per-key expiry, in which each hold carries its own time to live and simply disappears when it runs out, with no sweeper at all. Holds sit on the purchase path, so the migration could not take the path down, even briefly. It ran over three weeks, kept outside on-sale windows, in five phases:

Figure 6.3 — Expand and contract for the holds store. A flag or a deploy reverses each phase until the fifth, and the fifth waits until there is nothing left to reverse to. The thresholds are Encore’s own; no standard exists.

It is worth reading the table row by row, because each phase answers a different question. In the first phase, the expand, every hold was written to the old table as before and also, in shadow, to the new store: written there, but never read by anything a fan could see. The question was whether the new store could carry the load at all, and a week of clean writes answered it without a single fan depending on the answer. The second phase, the backfill, asked whether the new store held everything the old one did. The third, verify, kept the old table as the source of every answer but also read the new store and compared the two, asking whether they agreed. Only in the fourth phase did fans’ requests start being answered from the new store, and even then behind a flag that could send reads back to the old table in seconds, with both stores still receiving every write so that the old one stayed current enough to fall back to. The exit criterion for that phase was deliberately demanding: one full on-sale, Encore’s hardest traffic, served from the new store within its SLO. The fifth phase, contract, is the demolition. Old writers were removed, and even then the table was archived and kept for another 30 days before anyone dropped it, a last margin against the reader nobody had remembered.

A backfill, the bulk copy of existing data into a new shape, is usually the riskiest phase of any migration. It runs for a long time, sometimes days; it puts heavy load on a database that is also serving production; and it races live writes, so that a row copied at 10:00 may be changed at 10:01 by a request the copy never saw. The holds backfill avoided all of that for a reason that is easy to overlook: holds live for only five minutes. Once every write path was sending every new hold to both stores, any hold older than five minutes had already expired and needed no copying. After five minutes of dual writes, the new store contained every live hold, and the backfill was complete without a single row being copied. The general lesson is that knowing the lifetime of your data can turn the hardest part of a migration into a wait. It carries a proviso, though, and the proviso is where the danger lay: every write path must really write to both stores from the first minute. A path that was missed would leave holds in the old table that the new store had never heard of, and the five-minute wait would prove nothing.

That proviso was the real hazard, and it is the general hazard of dual writes, in which application code writes the same change to two stores one after the other. Without a transaction spanning both stores, and there is none between a relational database and a separate key-value store, one write can succeed while the other fails, and the two stores quietly drift apart. Encore handled this in the way Stripe described in its account of online migrations. The old table stayed authoritative throughout: whatever it said was the truth. A failed write to the shadow store was logged and counted but never failed the fan’s request, because the store being tested must never be allowed to hurt the store being trusted. And phase 3 existed to measure the drift directly, by comparing what each store returned for the same reads. The comparison found exactly one class of mismatch, and it was a good one to find before fans did. When a fan pressed “back” during checkout, the hold was released: deleted from the old table, but in the new store left to expire on its own. Had reads switched over with that bug in place, every seat a fan had let go would have stayed held, invisible to every other fan, for up to five minutes. On an ordinary day that is a nuisance; in the middle of a sold-out on-sale it is a great many seats briefly and wrongly unavailable. No test written before the migration had thought to look for this case, which is precisely why the verify phase is a phase and not a formality.

There is an alternative to dual writes, with a different set of costs. Change data capture (CDC) reads each committed change from the old store’s own log, the record a database already keeps of every write, and applies it to the new store. Because the application writes only once and the copy is derived from what actually committed, CDC removes the race between two independent writes. It has a price, and it is worth stating plainly. The new store lags behind the old one by however long the capture pipeline takes, so a read from the new store may not yet reflect a write a moment old. Changes are delivered at least once, which means that after a crash or a retry the same change may arrive twice, and so the consumer must apply them idempotently: applying a change a second time must leave the store exactly as applying it once did. And there is now a connector, one more running component, to operate, watch and recover when it falls over. That last cost is the one small teams most often underestimate, which is why Encore’s platform offers CDC on the paved road, so that no single team carries it alone. The choice between the two approaches is a genuine trade-off: dual writes are simple to start and drift silently, while CDC is consistent by construction and adds lag and machinery.

The same reasoning, keep the old shape serving while the new one is built beside it, also governs a humbler and more frequent kind of change, altering a large table in place. On many databases an ALTER TABLE on a big table takes a lock that blocks writes for as long as the change runs, which on a table of hundreds of millions of rows can be minutes, and minutes of blocked writes on the purchase path is an outage by another name. For large MySQL tables, tools such as gh-ost apply the expand-and-contract idea at the level of a single table: they build a shadow copy with the new structure, keep it in sync with ongoing writes, and swap it in at the end, so that no ALTER TABLE holds a lock longer than an SLO allows. PostgreSQL leans instead on operations designed to avoid long locks, such as building indexes concurrently, and on a short lock_timeout set on every migration, so that a statement which cannot get its lock quickly gives up and can be retried, rather than waiting in line and blocking every query that queues up behind it. Because the dangerous statements are easy to write and hard to spot, linters such as strong_migrations and squawk read migrations during review and flag the ones that would take a locking path. Each of these tools encodes, in a narrow form, the same lesson as the holds migration: change the shape of data only in steps that the system can survive while it keeps serving.

Recap. Schema changes cannot be reverted by redeploying, and old and new code always overlap during a rollout, so we expand before we contract and keep both shapes correct in between; that discipline is what gives us back the way back that data otherwise denies. A backfill is usually the riskiest phase, unless the data’s own lifetime makes it a matter of waiting. Dual writes drift silently, so we measure the drift before trusting the new store, or we trade it for the lag and upkeep of change data capture. Contraction comes last and deserves the most suspicion, because it is the one step with no way back.

Exercise 6.3. Take the next schema change on your team’s roadmap and write it out as a table like Figure 6.3. For each phase, name the exit criterion and the way back if that phase goes wrong. Which phase has no way back, and what must be proven, and by what evidence, before it starts?

6.4 Replication, Multi-Region, and Failover

Lag and what it means

Backups protect us against losing data slowly, through corruption or a careless delete; replication protects us against losing a machine, a zone or a region suddenly. The two are easy to confuse, and the quantity that tells them apart is lag. Replication lag is the delay between the moment the leader acknowledges a write and the moment a replica applies it. Under asynchronous replication, where the leader answers the client without waiting for any replica, the lag is not merely a performance figure. It is exactly the data a failover would lose: every write that the leader has confirmed but the replica has not yet received vanishes when the replica is promoted. If a replica runs two seconds behind, then two seconds of acknowledged writes are the price of losing the leader at that instant, and the store’s real RPO is whatever the lag happens to be at the moment of failure, whatever the documents say.

Lag has an unkind habit of growing at the worst time. It is largest when write volume peaks, because the replica must apply everything the leader accepted, and peaks are when systems are most stressed and trouble most likely to arrive. GitLab’s outage of 2017, which we met in Section 6.2, began in just this way, with a replica that had fallen too far behind to catch up and an engineer working late to repair it. For that reason lag deserves a place among the very few signals that should wake someone up. Most causes do not earn a page, because the symptoms they produce will be caught by an SLO alert anyway; lag approaching an RPO is different, because when it finally shows up as a symptom, the data is already gone.

Zero RPO needs a third vote

For most of Encore’s stores some lag is tolerable. The seat ledger is the exception. Its RPO is zero, because a lost write there is not a minor inconvenience: it is a seat sold twice, with two fans holding tickets for one chair. And the only way to lose nothing when a region fails is to acknowledge nothing until it is durable somewhere outside the region that wrote it.

The obvious design, and the one many teams reach for first, is a synchronous standby in the second region: the leader in the home region waits for the standby to confirm each write before telling the fan it succeeded. Following that design through to its failure shows why it is not enough. Suppose the network link between the two regions breaks. The home region must now choose, and both choices are bad. It can keep waiting for a peer it cannot reach, in which case writes stop and nobody can buy a seat. Or it can stop waiting and carry on alone, in which case the next write exists in one place only, and the zero RPO is abandoned at exactly the moment it was bought for. The standby faces its own dilemma on the other side of the break. From where it sits, a dead primary and a severed link look identical, since in both cases the primary has simply gone silent. If it promotes itself, it is guessing, and if the primary is in fact alive and still accepting writes, the guess produces the split brain described in the next subsection. Two sites cannot settle the question, because each sees only that the other has gone quiet. Telling a dead partner from a lost connection takes a third site with a vote.

This is an old problem in distributed computing, and the field’s answer is the majority. Each seat-ledger shard is therefore a consensus group: a set of replicas that commit a write only once a majority of them hold it durably, and that elect their leader by majority vote. Encore’s groups are Raft-style, in the manner Diego Ongaro and John Ousterhout described in 2014. The majority rule matters because two majorities of the same group always share at least one member, so two leaders can never both be elected, and any committed write is known to at least one member of whatever majority comes next.

Each group has five voting members spread across three EU locations:

• two in the event’s home region, EU-1 or EU-2, placed in different zones, where the leader normally sits;
• two in the other serving region;
• and a witness in a third region, EU-3: a replica that stores the log and votes but serves no traffic.

A write is acknowledged once three of the five hold it. In practice those three are the two home replicas and whichever remote member answers first, so every seat write pays for one cross-region round trip, about 18 ms between nearby EU regions. That sounds like a cost a checkout cannot afford, but against the 3 s checkout budget it is a small line. The harder question is throughput. If each write waited for its own 18 ms round trip before the next could begin, a shard that must keep writes in order could manage only about 55 of them a second. Group commit removes that ceiling: the leader gathers all the writes that arrive during one round trip into a single replicated batch, so the round trip is paid once per batch rather than once per write. That is how a shard sustains its 1,500 ordered writes a second while every one of them is durable in two regions.

With this arrangement in place, the failure cases resolve themselves by arithmetic rather than by judgment, which is exactly what we want of a mechanism that must work when nobody is watching. Take them in order of severity:

• A node or a zone is lost. Leadership moves to the other home replica, in the other zone, and the shard stays in its home region.
• The home region loses contact with one peer location, either the other serving region or the witness. The home region still holds its own two votes plus those of the location it can still reach, which is at least three of five. It keeps its majority, and fans notice nothing.
• The home region is cut off from both. Now it holds only two votes of five. Its leader governs under a lease, authority that lapses unless the majority keeps renewing it; the lease runs out and the leader stops accepting writes. Meanwhile the other region’s two members and the witness, together three of five, elect a new leader in the other region.

Consider what that last case delivers. Every acknowledged write was already on a majority, and every possible majority of the old configuration overlaps with the three survivors, so the new leader has every write the fans were told succeeded: the RPO stays zero. The RTO is the election plus re-routing the Gate and Orders to the new leader, a matter of minutes. And because a minority can never commit anything, the isolated home region cannot keep selling seats on its own, so there is no split brain to repair afterwards. The witness is a modest machine doing a modest job, but it is the one that lets the system tell “my partner is dead” from “I am alone”.

None of this is novel, and I find that reassuring rather than disappointing. Google’s Spanner, Amazon’s DynamoDB multi-Region strong consistency and Aurora DSQL all offer the same shape, two regions plus a witness, and CockroachDB states the same trade as a survival goal, letting the operator declare which failure the data must survive. When several independent teams arrive at the same design, it usually means the design follows from the problem rather than from anyone’s taste. The price is the same everywhere too: added latency on every write, and a third site to operate, patch and watch. Encore pays it only for the seat ledger, the one store where a lost write is a broken promise to a fan. Its other stores’ RPOs allow simpler asynchronous arrangements, and spending consensus on them would buy complexity without buying anything the business needs.

One limit, first met in Section 6.1, deserves to be restated plainly here. A majority protects Encore from the failure of machines and regions, but it cannot protect Encore from itself. If a bad deploy writes a wrong value, the consensus group faithfully replicates that write to all three locations in 18 ms, exactly as it would a correct one. Replication preserves whatever it is given, mistakes included. So the ledger still has point-in-time backups, which can return it to the moment before the mistake, and any corrupted projections, the read models derived from the ledger, can be rebuilt by replaying its event log, which only ever grows and so still holds the history the corruption overwrote.

Split brain and fencing

The consensus group prevents two leaders from being elected. It does not, by itself, prevent an old leader from acting as one. Split brain, failover’s characteristic failure, is the condition in which two nodes both believe they lead and both accept writes, so that the data diverges in ways that may never be reconciled. It happens more easily than one might hope. A leader cut off from its peers, or one frozen for many seconds by a long garbage-collection pause, cannot know that it has been replaced in the meantime. When it wakes or reconnects, it carries on with complete confidence, issuing writes on authority it no longer holds. It resembles a stage manager who steps out during the interval, misses the announcement that someone else is now calling the show, and comes back to call cues into a headset nobody is supposed to obey. The analogy breaks in one useful place: a theater crew would recognize the wrong voice, while storage, left to itself, has no way to tell a current leader’s write from a deposed one’s. Since the old leader may never hear of the handover, the protection must sit with the storage it writes to, which is the one party every write must pass through.

That protection is fencing. Every leadership term carries a number that only ever increases, called a fencing token, and storage rejects any write bearing a term older than the newest it has seen. The idea draws on two lines of work. One is the leases Cary Gray and David Cheriton described in 1989, grants of authority that expire unless renewed, which bound how long a leader can believe in itself without confirmation. The other is the numbered terms of consensus protocols, which give each reign of leadership a place in a strict order. On Encore’s Seat Inventory shards, a new leader’s first act is to record its term with the storage. From that instant, any write from the old leader carries a smaller number and fails, loudly and harmlessly: loudly, because the old leader receives an error it cannot ignore, and harmlessly, because nothing it attempted was applied.

It is worth separating what fencing costs from what a takeover costs, because they are easy to blur. Fencing itself costs almost nothing, a single comparison of two numbers on each write. The expensive part of a takeover is the wait for the old leader’s lease to run out, tens of seconds in which the shard accepts no writes at all. That wait exists so that the old leader has had time to notice it has lost its authority and step down. But it relies on clocks, and clocks on a stalled or overloaded machine can misjudge how much time has passed. Fencing is what makes the wait safe: if a deposed leader’s clock tells it the lease is still good when it is not, its writes are refused anyway.

Figure 6.4 — Fencing on a Seat Inventory shard. The deposed leader is never asked whether it still leads; the storage simply refuses its stale term, and the refusal is how it finds out.

Failing over a region

The seat ledger is only one part of what must move when a region fails. Here an earlier decision pays off. Because Encore pins each event’s writes to a home region, a regional failure is not one enormous event but many small ones: each affected event must have its home moved, one at a time and independently. The ledger’s part, as we have seen, needs no human action, since the consensus groups re-elect their leaders on their own. The rest of the work is moving each event’s home and redirecting the Gate and Orders to the new leaders. Encore automated all of that as a single operation, so nobody has to type a sequence of commands under pressure at 3 a.m., but it deliberately kept the trigger human: a person decides that a region has failed, and the machinery then carries out the decision.

That may look like timidity, given how much else is automated. The evidence for it is GitHub’s outage of 21 October 2018. During routine maintenance, connectivity between GitHub’s US East Coast network hub and its primary data center broke for 43 seconds. In that short window its failover tooling, Orchestrator, did exactly what it was built to do: seeing the primaries unreachable, it promoted database primaries on the West Coast. When the link returned, some writes existed only in the East and others only in the West, two histories that had diverged in under a minute, and the site ran degraded for just over 24 hours while engineers reconciled them. Every component had behaved as designed. What failed was the assumption that a 43-second interruption and a lost data center should be met with the same response. GitHub’s remediation was to stop Orchestrator promoting primaries across regions.

The lesson I draw from it is about matching the reflex to the stakes. Within a region, automated failover is right: the failures are frequent, the distances short, and a fast mechanical response beats a slow human one. Across regions the arithmetic flips. Real regional failures are rare, brief network flaps are not, and a failover fired by a flapping link turns a momentary blip into a self-inflicted outage with the largest blast radius a company has. A person who takes a few minutes to confirm that a region is truly gone costs little against that risk, provided the steps that follow are rehearsed and automatic, which is what Section 6.5 sets out to test.

Recap. Under asynchronous replication, lag is the RPO: whatever has not yet reached the replica is what a failover loses. A zero RPO needs a majority across three sites, because two sites can only guess whether the other is dead or merely unreachable, and even a majority faithfully copies our own mistakes, so backups and replay remain necessary. Fencing, enforced by storage rather than trusted to leaders, is the cure for split brain. Region failover should be automated in its steps and human in its trigger.

Exercise 6.4. For one replicated store you run, find out what prevents an old leader from writing after a failover. If the answer is “it will notice it has been replaced,” write down how long that noticing takes and what the old leader could write in the meantime.

6.5 Disaster Recovery as Practice, Not Plan

The plan on the shelf

Most organizations have a disaster recovery (DR) plan; far fewer have a DR capability, and the gap between the two is the subject of this section. A DR plan is a document that says how an organization will restore service after losing a system, a site or a region. A DR capability is the demonstrated ability of particular people, with particular tools and permissions, to actually do it within the time promised. The difference is the one between an evacuation map and a fire drill. The map is drawn once, correctly, and then the building changes around it: walls move, a stairwell is locked for renovation, the people who knew where the second exit was leave. Nothing on the map announces that it has gone stale. Only a drill, with real people walking real corridors against a clock, reveals that the route now ends at a locked door. The analogy holds better than most, though it breaks in one place that matters to us. A building changes slowly and visibly, while a software system changes every week and mostly out of sight, so a DR plan goes stale far faster than any evacuation map.

Regulators understood part of this long ago. Banking, where I spent much of my career, requires DR tests by rule, and the rule does its job of ensuring that a test happens on schedule. Whether the test teaches anything is a separate matter, and it depends on something no regulation can require: whether people are allowed to fail. A test that must pass will be designed to pass, rehearsed along the easy path and declared a success, and the organization learns nothing it did not already believe. Only a test that is allowed to fail can find the locked door.

In the vocabulary of this book, a DR drill is a game day of the kind Chapter 5 described, aimed at a narrower and sharper question: can these people, with this tooling and these permissions, rebuild this system within the RTO someone promised? Section 6.2 treated the RTO as a claim awaiting a test; a drill is that test. Drills also counter one of Bainbridge’s ironies of automation, an irony that bites hardest in DR. The better the failover machinery, the more rarely anyone has to recover by hand, and so the less practiced anyone is on the day the machinery itself fails, which is precisely the day a manual recovery is needed. Encore’s seat ledger fails over so reliably that hardly anyone remembers how to do it otherwise. Good automation, left alone, quietly deskills the people who stand behind it, and a drill is how we keep them in practice.

All of which leads to a conviction I hold firmly. An RTO is a hypothesis until a stopwatch has run against it.

The Finance ledger drill

Encore put that conviction to work in the spring, when the Reliability team ran the company’s first full restore of the Finance ledger. Recall from Section 6.1 that the Finance ledger is the relational database of Encore’s books, fed by every SaleCompleted event, reconciled daily and closed nightly. It had been backed up every night and never once restored in full. By the spring it held about 4.2 TB, and its RTO of two hours had been agreed when the database was a tenth of that size: a promise made about a smaller system, never revisited as the system grew. The scenario was realistic and unglamorous. A data corruption is discovered at 09:00, and the repair is to restore the ledger to its state at 02:00, just after the nightly close completed. The team wrote down the planned duration of each step before starting, then recorded what actually happened.

Figure 6.5 — The Finance ledger restore drill. No delay came from the database engine.

The drill missed its RTO more than fourfold, and it was the most useful day of Encore’s quarter. That is less of a paradox than it sounds. Every hour of overrun was an hour Encore would otherwise have discovered during a real corruption, with Finance unable to close its books and the regulatory exposure that brings; the drill bought the same knowledge on a calm spring day, at the price of one team’s time. The caption of Figure 6.5 carries the first lesson. Not one minute of delay came from the database engine, which was the part everyone had worried about. The delays came from the people, permissions, capacity and preparation that surround the engine, and it is worth taking them one at a time.

The first delay was in reaching the backups at all. As Section 6.2 described, Encore keeps backups of its systems of record in a separate account, where no single credential can weaken them. Opening that account for a restore needs two approvers. The rule was right; it had simply never been tried. One approver was on leave, and once access was finally granted, the policy on the encryption key that protects the backups turned out not to grant the account the drill was using. Neither problem was exotic, and neither could have been found by reading the plan. The fixes were equally plain: a standby approver rota, so that two approvers can always be reached, and a key policy that grants the recovery account, so that the people restoring the data can also decrypt it.

The second delay was physics, and physics does not yield to runbooks. The restore was limited by the provisioned I/O of the volume it was writing to, a volume sized for the ledger’s normal daily running and never for the sustained bulk write of a full restore. A short calculation from Figure 6.5 shows how firm that limit was. Restoring 4.2 TB took 3 h 10 min, about 11,400 seconds, which works out to roughly 370 MB a second. At that rate two hours of restoring brings back only about 2.6 TB, and that is before log replay, verification and repointing have had a single minute. Whatever the runbook said, the whole ledger could not come back in two hours. A faster volume would have helped at the margin, but the deeper problem was that the RTO treated years of accounting history and the current year’s sales as one undifferentiated mass that all had to return at once.

So the team changed the shape of the problem. It split the ledger so that the current financial year, about 600 GB, restores first and becomes writable on its own while the history follows behind, and for that current-year slice it added a warm standby, a copy kept continuously up to date and ready to take over. Then someone asked Finance the question that should have been asked when the original RTO was set: what do you truly need, and by when? Finance’s answer was precise. It needed to run the next nightly close, which means the current year back within two hours; the history could wait a day. The two-hour promise had never been wrong in spirit. It had been attached to the wrong unit of data, and decomposing the ledger by what each part was for turned an impossible promise into two achievable ones. Three months later a second drill had the current year writable in 1 h 50 min. Figure 6.2, in Section 6.2, tells the same story in terms of RPO and RTO: the promise was kept only once the team changed what had to come back first.

Verification taught the subtlest lesson, the one Pixar learned in the days after its rescue, when the team had a running copy of Toy Story 2 and still had to discover, file by file, what that copy lacked. A restore is finished when someone with the authority to say so can declare that the data is right, and a database that has started and answers queries proves very little about that. For the Finance ledger, “right” means that its records agree with the settlement reports of Encore’s two PSPs and with the stream of SaleCompleted events up to 02:00. Before the drill nobody had written down how to check either, so engineers spent two hours and twenty minutes composing verification queries under time pressure, which is the worst possible moment to be writing careful queries against a company’s books. Afterwards the team kept those queries, refined them and began running them every week against the scheduled restores of Section 6.2. The work did not disappear; it moved out of the emergency and into an ordinary week, where it could be done calmly and checked. Written in advance and exercised weekly, the queries turned two hours of improvisation into 10 minutes of reading a report.

The last delay, a reporting job whose connection string had been hard-coded to the old database address, is the smallest in the table and perhaps the most typical. Every system accumulates quiet assumptions of this kind, each one harmless until the day the thing it assumes stops being true. A drill is how we find them while finding them is still cheap, and Encore’s next exercise found a whole family of them.

The region failover game day

The Finance drill tested one store. The largest test of the year tested a region. It came on a quiet October Tuesday, chosen because no on-sale was scheduled within a week, so that any surprise would land far from a night when fans were queuing. The Reliability team cut EU-1 off from both EU-2 and the EU-3 witness, isolating it as a severe network failure would, and an engineer then triggered the failover of every event whose home region was EU-1. Section 6.4 explained why Encore automates the steps of a region failover but keeps the trigger human; here a human pulled it deliberately. To make the test honest without putting a major sale at risk, small events carried real traffic, real fans buying real seats, alongside synthetic buyers that exercised the paths no real fan happened to take.

The core held, and it held exactly as Section 6.4’s arithmetic said it would (Figure 6.6). In each seat-ledger shard, the two replicas in EU-1 now commanded only two votes of five, so their leaders lost their majority, let their leases lapse and stopped accepting writes. In EU-2, the two replicas there gathered the witness’s vote to make three of five and elected new leaders. Fencing then rejected a handful of late writes from the deposed EU-1 leaders, loudly and harmlessly, as designed. The whole failover took about four minutes, and no hold or sale was lost. It is worth pausing on how satisfying, and how uncommon, such a result is: a mechanism designed on paper, exercised under a realistic failure, behaving precisely as its design predicted.

The surprises were at the edges, in services around the ledger that had never been asked to live anywhere but EU-1. The first was in the Sale Gate, the virtual waiting room in which fans queue for an on-sale. Its pre-warm, the preparation that raises provisioned capacity before a sale so that the first fans do not wait for new instances to start, had only ever been done in EU-1, because every on-sale until then had been homed there. So the first buyers in EU-2 met cold starts, the same slow opening seconds that a skipped pre-warm step had once caused, as the Operations book recounts. The second was in Bot Screening, whose risk data the Gate uses to judge which buyers are bots. That data reached EU-2 only by a daily copy, so the Gate in EU-2 screened fans against yesterday’s signals, blind to anything learned since. The third was an Orders reconciliation job whose schedule named EU-1 explicitly; with EU-1 cut off, it simply did nothing for six hours.

None of the three broke the failover, and each would have made a real one worse: slower for the first fans, weaker against bots, quietly behind on reconciling orders. They share a single cause, which Figure 6.6’s caption names. Each was a thing that had assumed EU-1 would always be home. The consensus design had been built from the start to survive losing a region, while the services around it had grown up in a world where that region never went away, and so the system was only as ready for a failover as its least-examined assumption. Each finding became a ticket with an owner, following the discipline of Chapter 5 that a game day is worth what is fixed after it.

Figure 6.6 — The October game day: EU-1 isolated, and EU-2 taking over with the votes of the EU-3 witness. The core held; the three findings sat at the edges, in things that had assumed EU-1 would always be home.

One more choice made these drills worth more than their findings, and it concerns who leads them. A DR runbook is good when someone who has never performed the procedure can follow it at 3 a.m., because that is very likely the person who will be holding it when a real disaster comes. A runbook led every time by the engineer who wrote it is never really tested, since its author fills each gap from memory without noticing that the gap is there. So Encore’s own rule, a step beyond the rotation of Google’s DiRT exercises that Chapter 5 introduced, is that each drill is led by someone who has not led one before, with an experienced engineer shadowing to keep the exercise safe. The theater has long worked this way: an understudy is not truly ready until they have played the role on the stage, under the lights, with the cast around them, and a company that only ever lets the lead perform has a single point of failure in its cast. The analogy is imperfect, because an understudy performs a fixed script while a DR lead must improvise whenever the system departs from the runbook, but that is all the more reason for the rehearsal to be real. How a genuine disaster is commanded, once it is no longer a drill, belongs to incident response, which the Operations book takes up.

Recap. Disaster recovery is a capability that exists only where it is practiced: a plan on a shelf records intentions, while a drill measures what people can actually do. Drills time real recoveries against the RTOs that were promised, and their findings concern permissions, throughput, verification and the assumptions at the system’s edges far more often than the database engine. A restore is finished only when someone with authority can say the data is right, and a runbook is proven only when someone new to it can follow it.

Exercise 6.5. Plan a restore drill for one system of record. Write down the scenario, the target state you will restore to and the RTO you have stated, then list every permission, key and person the restore needs, including who stands in when each person is away. Before you run the drill, predict the duration of each step and write down how you will verify that the restored data is right; afterwards, compare your predictions with what happened, and give each gap a ticket and an owner.

Kata: The Stored-Value Card

Your brief: “Tidewater Transit.” Tidewater runs fare payment for a regional transport network. Its riders carry 2.4 million stored-value cards, each holding a balance that the network spends down one fare at a time. Some 3,000 validators on buses and trams read those cards. A bus is not always within signal, so each validator buffers its taps while offline and uploads them when it reconnects. Riders add money through a top-up app.

Every balance lives in one 9 TB PostgreSQL primary, with an asynchronous replica in a second region. In the morning rush, when the most taps and top-ups arrive at once, that replica lags several minutes behind. Backups are a nightly logical dump that takes seven hours to produce and has never been restored in full, so no one knows how long a restore would take. The regulator now asks for an RTO of four hours and an RPO of five minutes for balances, and neither figure has yet been checked against anything Tidewater has measured.

Tidewater has already had one warning. Last spring an engineer dropped a partition of tap history, and the team rebuilt it from the validators’ own logs over two weeks. Next year brings a larger change: account-based ticketing, in which fares are charged to a rider’s account rather than deducted from the card, will reshape the balance model under live traffic. Tidewater has 14 engineers to do all of this, and one database specialist.

The brief resembles Encore’s story in shape and differs from it in detail. Tidewater has a single relational store where Encore has a consensus-replicated ledger, a nightly logical dump where Encore has layered backups, and a restore no one has tried where Encore now has two drills behind it. Work through it in the order the Reliability team followed at Encore: the inventory first, then the backups, then the migration, then failover, and finally a drill that tests the rest.

Deliverables:

1. State inventory — classify every store as a system of record or as derived data, in the shape of Figure 6.1, with its RPO, its RTO and the path by which it would be restored or rebuilt. Decide in particular where the validators’ buffered taps belong, since last spring they were the source from which lost history was rebuilt.
2. Backup and restore design — the layered mechanisms, each chosen against a threat that actually occurs (a dropped partition, slow corruption, ransomware, the loss of a region), and where each copy lives. For every copy, state which credentials could delete it. Then give the restore-testing schedule and the SLI that times each restore, so that the RTO becomes a measured figure rather than a hopeful one.
3. Migration plan — expand and contract for account-based ticketing, as Section 6.3 did for Encore’s holds store. Give the phases and the exit criterion for each, the backfill strategy for the 2.4 million existing balances, and how any drift between the old and new models during dual writing will be measured or avoided.
4. Failover design — lag alerting set against the five-minute RPO, so that someone is paged before the lag consumes the whole budget; how promotion of the replica is fenced, so that the old primary cannot go on writing; and who decides to pull the trigger.
5. First DR drill — the scenario, a predicted duration for every step written down before the drill begins, the verification queries that will prove the restored balances are right, and who leads it.

Rubric:

Three of the rubric’s questions repay a moment’s thought before you begin. The restore arithmetic is where Encore’s Finance ledger drill came apart, and nothing in Tidewater’s brief suggests its numbers will be kinder: a dump that takes seven hours to produce gives little reason to expect a restore in four. The fencing question asks for a mechanism that lives with the storage, because, as Section 6.4 showed, a deposed primary cannot be trusted to notice that it has been deposed. And the practice question expects failure on purpose. A first drill that passes cleanly has probably tested too little, while one that misses its target by hours, as Encore’s first Finance drill did, shows exactly where the work lies.

Where you now stand

You can now take an inventory of a system’s state and say which stores are systems of record and which can be rebuilt from something else. You can design backups for the threats that actually occur rather than only the dramatic ones, and migrate data under live traffic without losing the way back. You can reason about replication lag, quorums and fencing well enough to say what a failover will lose and what stops two leaders from writing at once. And you can turn a disaster recovery plan into a capability that is timed, practiced and led by people new to it. One threat in this chapter differed from all the others. The misplaced delete, the slow corruption and the severed link were accidents; the adversary who hunts for your backups before encrypting your data is not. Chapter 7 turns to that adversary, and to security as an operational discipline.

References

• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering, ch. 26. O’Reilly/Google, 2016 — free online.
• Martin Kleppmann, Chris Riccomini — Designing Data-Intensive Applications, 2nd ed.. O’Reilly, 2026.
• GitLab — Postmortem of database outage of January 31. GitLab blog, 2017 — free online.
• GitHub — October 21 post-incident analysis. GitHub blog, 2018 — free online.
• Diego Ongaro, John Ousterhout — In Search of an Understandable Consensus Algorithm. USENIX ATC, 2014 — free online.
• Cary G. Gray, David R. Cheriton — Leases: An Efficient Fault-Tolerant Mechanism for Distributed File Cache Consistency. SOSP, 1989 — free online.
• Stripe — Online migrations at scale. Stripe blog, 2017 — free online.
• Kripa Krishnan — Weathering the Unexpected. ACM Queue, 2012 — free online.
• Jim Gray — Why Do Computers Stop and What Can Be Done About It?. Tandem Technical Report 85.7, 1985 — free online.