Building Software at Scale in the AI & Cloud Era

← the whole book on one page

Chapter 3: Testing Strategy

“Software is an expression of a highly detailed design and does not fail in the same sense as a mechanical system.” — Ariane 5 Flight 501 Inquiry Board, 1996

What does a passing test tell us? The answer seems obvious: the code does what the test says it should. But a test says more than its assertions. Every test carries a picture of the world in which the code will run, the inputs it will see, the neighbors it will call, the order in which things will happen, and a green result means only that the code behaves well inside that picture. When the picture is accurate, the suite is among the most valuable things an engineering organization owns, a record of what the system should do, checked thousands of times a day. When the picture is wrong, the suite is worse than useless, because it lends the confidence of evidence to a guess about the world that nobody remembers making.

This chapter is about choosing what our tests should assume, and about spending a finite budget of effort on the tests that repay it. It begins with the suite that cleared the inertial reference software of Ariane 5 for its first flight in June 1996, and it ends with a kind of testing the Ariane engineers never had to imagine: evaluating a language model whose output differs every time we ask. Between them lie the ordinary disciplines of deciding what a test is worth, growing software in small steps, replacing the neighbors we cannot run with simulators we can trust, and keeping a few end-to-end journeys alive without letting them rule us.

Encore enters the chapter with tens of thousands of tests across forty teams, a developer survey in which flaky end-to-end tests ranked among the loudest complaints (Section 1.2), and a new product that will stress every layer of the suite. Barcode issuance, written by the original nine engineers in the company’s first year and owned today by the Inventory team, was built on the assumption that a ticket is issued once. Its tests, such as they are, encode that assumption. Verified Resale will issue a ticket, revoke it and issue it again under a new identity, as often as fans resell it, with partners whose code Encore cannot run, a payment provider whose marketplace product splits the money four ways, and an organizer rule assistant whose answers a model drafts. Each is a test of the testing strategy itself.

3.1 What a Test Is Worth

The world a test assumes

On 4 June 1996 the first Ariane 5 lifted off from Kourou, in French Guiana, carrying the European Space Agency’s four Cluster satellites. About 37 seconds after main-engine ignition, half a minute into flight, both of the launcher’s inertial reference systems failed within 72 milliseconds of each other. The on-board computer, now receiving a diagnostic bit pattern from the failed unit, read it as flight data and commanded full deflection of the nozzles. The launcher veered, began to break up under aerodynamic load, and destroyed itself at an altitude of about four kilometers. The inquiry board, chaired by the mathematician Jacques-Louis Lions, reported about six weeks later, in one of the clearest documents ever written about how software fails.

The mechanism was small. Software in the inertial reference system converted a 64-bit floating-point value, the horizontal bias, into a 16-bit signed integer. On Ariane 4 the value always fit. On Ariane 5, which builds horizontal velocity much faster in the early part of its trajectory, it did not, and the conversion raised an operand error. Of seven variables at risk of exactly this error, the designers had protected four and left three unprotected, the horizontal bias among them, to stay within a target of 80% maximum load on the computer. The system’s specification said that on any exception the processor should shut down, because exception handling had been designed for random hardware failure, and under that assumption a shutdown hands control to the backup. The backup ran identical software on the same inputs, and it had already failed in the same way one data cycle earlier. The deepest irony is the one the board took care to record. The function that failed had no purpose after lift-off. It kept running for about 40 seconds into flight because Ariane 4 needed to be able to recover quickly from a hold late in the countdown, and the board noted that this behavior was not required for Ariane 5 at all.

What makes Ariane 501 a story about testing, more than about integer overflow, is the board’s account of why nobody found the fault. The inertial reference system was reused from Ariane 4, “practically the same”, in the report’s words, “particularly as regards the software”. It had flown successfully. When the Ariane 5 requirements were written, the report says, “it was jointly agreed not to include the Ariane 5 trajectory data in the SRI requirements and specification.” In 1992 the project had also decided not to put the real inertial units in the loop during the closed-loop system simulation, using simulated outputs instead. So no test anywhere in the program ever fed the reused software the trajectory it would actually fly. The suite was not careless. It was faithful, with great rigor, to a world that no longer existed. The board concluded that had such a test been performed, by the supplier or as part of the acceptance test, “the failure mechanism would have been exposed,” and two of its recommendations are worth quoting because they read like a testing strategy in miniature. The second: “Prepare a test facility including as much real equipment as technically feasible”, with realistic inputs and complete closed-loop testing. The tenth: “Include trajectory data in specifications and test requirements.” Figure 3.1 sets the flight that happened beside the world the tests assumed.

Figure 3.1 — Ariane 501, schematic. In flight, both inertial reference systems met a trajectory their specification had left out and failed one data cycle apart; in test, simulated outputs stood in for the real units, and every test passed.

Later writers drew different lessons. Jean-Marc Jézéquel and Bertrand Meyer argued in 1997 that a stated precondition on the reused routine would have exposed the error at the moment of reuse; Bashar Nuseibeh answered in the same year that the failure belonged to the system and its process more than to any one piece of code. Both are interpretations, and both are useful. The lesson I take for this chapter is narrower. A test suite is a model of the world, and reused code arrives with the old world’s model intact, its tests still passing. Encore has paid for this lesson once already. The admission-rate change behind the Vesper Hale morning was approved on the strength of a load test run against a warm seat-map cache, a test that was perfectly sound for a world the on-sale did not live in (the Resilience book, Section 2.3, and the Operations book’s Chapter 6 tell that story). Barcode issuance is Encore’s own inertial reference system: old, trusted, reused for a purpose its authors never imagined, and tested, where it is tested at all, against a world in which a ticket is issued once.

The four pillars

If a test is a model of the world, we need a way to judge whether a particular model is worth keeping. Vladimir Khorikov’s Unit Testing Principles, Practices, and Patterns (2020) gives the most useful framework I know. He starts from the purpose of testing, which he states as enabling “sustainable growth of the software project”, and he judges every test against four attributes, which he calls pillars. Protection against regressions is the test’s chance of failing when a real bug is introduced. Resistance to refactoring is its chance of staying green when the code is restructured without changing its behavior. Fast feedback is how quickly it runs. Maintainability is how cheap it is to read, understand and keep running. A test’s value is roughly the product of the four, so a test that scores zero on any one of them is worth nothing, however well it does on the others.

Figure 3.2 — The four pillars, after Khorikov. A test’s worth is roughly their product, so one pillar at zero makes the test worthless.

The pillars trade against one another. Protection and speed pull apart, because the tests that exercise the most real code are the slowest. Resistance to refactoring is the one Khorikov refuses to trade, and his reason is worth dwelling on. A test that fails when nothing is wrong is a false alarm, and false alarms teach the people who receive them that a red build means nothing. Google’s Software Engineering at Google (2020) arrives at the same priority by a different road. It calls such a test brittle, “one that fails in the face of an unrelated change to production code that does not introduce any real bugs”, and it opens its unit-testing chapter with an engineer whose brittle tests broke on a harmless refactor. DORA, the DevOps Research and Assessment program, puts it more plainly: “test failures should indicate a real defect”. The pillars are a reasoned framework rather than a measured result, and their strength is that three independent sources converged on the same priority. A fire alarm that goes off whenever someone makes toast is soon unplugged, and the analogy holds closely here, with one difference: a building has one alarm, while a suite has thousands, and people learn to ignore all of them from the few that cry wolf.

Behavior, coverage and mutants

The practical consequence is a rule easy to state and harder to keep: test what a caller can observe, and leave the implementation free to change. A test of the barcode issuance code should ask whether the issued ticket is valid at the gate, whether its identity is unique and whether a second request with the same idempotency key returns the same ticket. It should not ask which internal methods were called in which order, because that is precisely what a refactoring changes, and the Inventory team is about to refactor that code harder than it has ever been refactored (Section 2.4).

Coverage is where well-meaning organizations go astray, because numbers invite targets. Khorikov’s verdict is exact: “coverage metrics are a good negative indicator but a bad positive one.” Low coverage is a reliable sign of trouble, since code no test executes is code no test protects. High coverage proves very little, because, as Google’s chapter on testing notes, coverage “only measures that a line was invoked, not what happened as a result.” A test can execute every line of the fee-split code and assert nothing about the money. Google also observed what happens when the number becomes a target: “instead of treating 80% like a floor, engineers treat it like a ceiling.” Encore’s scorecards report coverage for each service for exactly this reason, as a warning light below a floor and nothing more.

A better question than “was this line executed?” is “would any test notice if this line were wrong?” Mutation testing asks it directly: a tool makes a small deliberate change to the code, flipping a comparison or deleting a statement, and checks whether some test fails. A surviving mutant marks a place where the suite would not notice a bug. Long dismissed as too expensive for real codebases, it was made practical at Google, as Goran Petrović and Marko Ivanković described in 2018, and the conditions of their success matter more than the technique. Google mutates only the lines changed in a code review, suppresses mutants in code where they are known to be unproductive, and shows the survivors to the author as review comments, with no score attached. By 2018 some 6,000 engineers had used it. A later study by the same group with Gordon Fraser and René Just, covering almost 15 million mutants, found that developers who were shown mutants went on to write more tests and stronger ones. Encore turned diff-based mutation on for two codebases where a quiet gap would be most expensive, the barcode core and the Resale service’s fee arithmetic, and it reports nothing about mutants on any scorecard. A mutation score that became a target would repeat the coverage mistake one level up.

Sizes and shapes

The Architecture book (Section 5.3) gave Encore’s fleet its test pyramid, from many fast unit tests up to a handful of end-to-end journeys. It leaves open a practical question: when an engineer sits down to write the next test, how should they decide what kind it is? Google’s answer separates two things that the pyramid blurs. A test’s size is the resources it may use: a small test runs in a single process, with no sleeping, no disk and no network; a medium test is confined to a single machine and may talk to a real database on localhost; a large test may span machines. A test’s scope is how much code it is meant to verify, from one function to a whole journey. Size and scope are, in Google’s words, “two distinct dimensions for every test case”. The distinction earns its keep because size, unlike scope, can be enforced: a build system can refuse a small test that opens a socket, which does more for determinism and speed than any exhortation. Google suggests a mix of roughly 80% narrow, 15% integration and 5% end-to-end by test count, which it explicitly calls its own version of Mike Cohn’s pyramid, and it adds that every team’s mix will differ.

Martin Fowler, surveying the arguments between pyramid, honeycomb and trophy in 2021, found that most came from people who meant different things by “unit test”, some a solitary test of one class, others a sociable test of a cluster of real objects. Once the words are settled the shapes converge, and what matters is the quality Fowler borrows from Justin Searls, that tests should “run quickly & reliably, and only fail for useful reasons.” The pyramid survives this scrutiny as a cost argument. Large tests cost more to run, more to debug and more to keep green, so we should buy as few as will do the job. It does not survive as a shape to hit: counting tests to check the proportions measures the shape and says nothing about the cost.

Doubles, and who owns them

Every test that does not run the whole world must replace part of it, with what Gerard Meszaros named test doubles in xUnit Test Patterns (2007). Mock objects first appeared in print in 2000, in “Endo-Testing: Unit Testing with Mock Objects” by Tim Mackinnon, Steve Freeman and Philip Craig. The idea was a design technique as much as a testing one: by specifying the messages an object should send to its collaborators, the test helped discover the collaborators’ interfaces. Fowler’s essay “Mocks Aren’t Stubs” later named the two schools that grew up around the technique. The classical school replaces only what it must and checks the final state; the London or mockist school replaces every collaborator and checks the interactions.

Both schools have produced good software, and long practice shows where they go wrong. Google’s chapter on test doubles reports that the company learned “the hard way” that tests built heavily on mocking frameworks “required constant effort to maintain while rarely finding bugs”, and its guidance is blunt: “A real implementation should be preferred over a test double,” and where that is impractical, “the best option is often to use a fake”, a lightweight working implementation of the same interface. Khorikov gives the most useful line to draw. A managed dependency is one only our service talks to, such as its own database; an unmanaged dependency is one whose interactions other systems can observe, such as a payment provider, a message bus or an email gateway. Use the real managed dependencies in integration tests, and reserve mocks for the unmanaged ones, where the interaction itself is the behavior a caller cares about. Charging a card twice is a bug the payment provider observes, so the test should check the call; the SQL Orders & Payments writes to its own database is a detail, so the test should run it against a real database and check the outcome. That is no longer expensive: libraries such as Testcontainers start a disposable database in a container for each test run, and at Encore the service template wires one in by default.

I created MockK, a widely used mocking library for Kotlin, and have maintained it for a decade, so I hold this view from the toolmaker’s side. Much of what a mocking library for Kotlin must offer lets a test reach further into the code under test than the mock objects of 2000 ever did: classes are final by default in Kotlin, and MockK can mock them anyway, along with singleton objects, top-level functions, constructors and private functions. Each of those capabilities exists for a good reason, usually code we cannot yet change. Used routinely, they let a team test around a design instead of listening to it (Section 3.2 returns to that idea), and the result is a suite that pins the code’s present structure in place and fails on every attempt to improve it. A mock is a precise statement about a conversation between two objects. It is valuable when that conversation is the thing we mean to specify, and expensive everywhere else.

The last question about doubles is who owns them. Google’s answer is that “the team that owns the real implementation should write and maintain a fake”, because that team knows how the real thing behaves and will notice when it changes. At forty teams the alternative is a dozen hand-rolled fakes of Seat Inventory, each a consumer’s guess, each drifting at its own pace. Encore’s rule is that a service’s fake ships with the service, from the same repository, and that consumers use it or explain why not. Section 3.3 extends the rule to the outside world.

Sorting code by what it deserves

Not every line of code deserves the same testing, and Khorikov offers a sorting tool for deciding which deserves what. Plot code along two axes, its complexity or domain significance and the number of collaborators it talks to, and four quadrants appear (Figure 3.3). Domain logic and algorithms, complex and with few collaborators, deserve thorough unit tests, because that is where the bugs that matter live and where tests are cheapest. Controllers, simple but talking to many collaborators, deserve a few integration tests that check the wiring. Trivial code deserves no tests. And overcomplicated code, complex and entangled with many collaborators, deserves a refactoring before it deserves a test, because any test written against it now will be large, slow and brittle. The refactoring that moves code out of the fourth quadrant is Meszaros’s Humble Object pattern: pull the decisions out into a plain object that is easy to test, and leave behind a thin shell that does the input and output and is too simple to get wrong.

The quadrants are one author’s heuristic, built on long practice, with no measured study behind them, and they need one condition attached. “Trivial code deserves no tests” is a statement about getters and data classes. It does not license untested glue on a path where money or seats move. The resale purchase saga is mostly wiring, and every link of that wiring is covered by an integration test, because a mistake there costs a fan a ticket. For barcode issuance, which sits squarely in the fourth quadrant, the sorting tool says what Section 2.4 says by another route: characterize first, then pull the logic out of the shell and test it where it lands.

Figure 3.3 — Sorting code by what it deserves, after Khorikov. Tests are cheapest and most valuable top left; code top right earns a refactoring first, and the Humble Object moves its decisions to where a unit test can reach them.

Recap. A test is a model of the world, and Ariane 5 showed what happens when the model is faithful to a world that is gone. Khorikov’s four pillars give us a way to judge a test, and resistance to refactoring is the one we cannot trade, because false alarms teach people to ignore the suite. Test observable behavior, treat coverage as a warning light and mutation as a question asked in review, choose tests by size as well as scope, and let the owners of a dependency own its fake.

Exercise 3.1. Take ten tests from your own suite, chosen at random, and score each from zero to three on the four pillars. Delete or rewrite the two with the lowest product. For each, write one sentence about the world the test assumed, and whether that world still exists.

3.2 Growing Software with Tests

The walking skeleton

The Resale team formed in month 4 of the bet, and the first thing it built was not a feature. It was a walking skeleton: the thinnest possible implementation of the whole system that runs end to end, built, deployed and tested automatically, before any feature is fleshed out. Alistair Cockburn coined the term in Crystal Clear (2004). Steve Freeman and Nat Pryce made it test-driven in Growing Object-Oriented Software, Guided by Tests (2009), insisting that the skeleton’s first job is to prove that the deployment pipeline and the automated tests can reach every part of the system, so that every later feature is born into a working harness.

For Verified Resale the skeleton is the same slice the Products book drew in its Section 3.4, now made of code: one organizer, one partner, a face-value cap and nothing else, entry at one venue. A test fan who bought a ticket through the Gate lists it through a stub of Rowmate’s integration; a second test fan buys it; Inventory reissues the ticket under a new identity; the seller’s barcode lands on the revocation list held by one venue’s scanners; and the buyer’s new ticket reaches the app. Each step was as crude as it could be while still being real. The rules engine knew one rule, the fee split was hard-coded, and the reissue call entered the barcode core through a narrow new entry point the Inventory team wrote behind characterization tests. But every step ran in Encore’s real pipeline, on the golden path, under one automated test.

The skeleton cost several weeks in which no organizer saw anything new, and it forced the cross-team questions to arrive early, while they were cheap: who owns revocation (the Inventory team, as Section 9.5 confirms), how a revocation reaches an offline scanner, which payment product carries the split. Civil engineers do something similar when they throw a pilot line across a gorge before building the bridge, a thin cable that proves the two sides can be joined and then carries the heavier cables across. The analogy breaks in a way that favors software. The pilot line is discarded once the bridge stands, while the walking skeleton becomes the bridge, every later feature added to bones that have carried traffic since the first week.

Two loops

Freeman and Pryce describe the rhythm of work after the skeleton as two nested loops. The outer loop begins with an acceptance test: a test of a whole feature, written from the outside in the language of the domain, which fails because the feature does not exist yet. The inner loop is Kent Beck’s cycle from Test-Driven Development: By Example (2002): write a small failing unit test, make it pass with the simplest code that works, then refactor while the tests stay green. The engineer goes round the inner loop many times, each turn adding one small, tested piece, until the acceptance test passes and the outer loop turns once (Figure 3.4).

Figure 3.4 — Two loops, after Freeman and Pryce. The acceptance test is the executable goal; Beck’s cycle turns many times inside it, and the outer loop turns once per feature.

Here is the acceptance test that drove the skeleton, essentially as the Resale team wrote it:

@Test
fun `a resold ticket moves to the buyer and the old barcode dies`() {
    val seller = resale.fanWithTicket(faceValueCents = 6000, boughtVia = GATE)
    val listing = resale.partner("rowmate").lists(seller.ticket, priceCents = 6000)
    val buyer = resale.screenedFan()

    buyer.buys(listing, via = "rowmate")

    assertTrue(buyer.app.showsTicketFor(seller.ticket.seat))
    assertTrue(resale.venueScanner().rejects(seller.ticket.barcode))
    assertEquals(6600, resale.charged(buyer))
    assertEquals(mapOf("partner" to 240, "organizer" to 180, "encore" to 180), resale.ledger.split(listing))
    assertEquals(6000, resale.payoutDue(seller))
}

The test names no service, endpoint or table, and money is in integer cents. The resale object is an application driver, a layer that translates domain actions into calls on the real system, and Section 3.4 returns to why that layer matters. It would survive a complete rewrite of everything beneath it.

The double loop is a way of working in small steps against an executable goal, and that is the claim it can carry; whether it beats other ways of working in small steps is a question the end of this section takes up. Chapter 5 takes the outer loop one step further, because the acceptance test written before the code is also the most precise specification we can hand a coding agent.

Listening to the tests

The most useful idea in Freeman and Pryce’s book may be the one in its title for Chapter 20: listen to the tests. When a test is hard to write, the difficulty is usually information about the design. A test that needs 40 lines of setup is reporting that the code under test depends on too much. A test that needs five mocks is reporting that the object has too many collaborators, or that it is doing work that belongs elsewhere. A test that cannot be named without the word “and” is reporting that the code does two things. Treating these as testing problems, to be solved with better fixtures or a stronger mocking tool, silences the design’s request for help.

David Farley’s Modern Software Engineering (2021) builds a whole approach to design on this observation, treating testability as a proxy for the qualities we actually want, modularity, cohesion, separation of concerns and loose coupling, because code that has them is easy to test and code that lacks them is not. DORA’s research on test automation adds a finding about who hears the message: when tests are written by a separate group, developers “write code that is hard to test”, because the feedback never reaches the people who could act on it. This is practitioner consensus, with no controlled study that isolates the effect, and it is a consensus I share. The first time the Resale team tried to test the fee split, the test needed a running rules engine, an event, an organizer and a PSP fake. The team listened, and the split became a pure function of a price, a channel and a jurisdiction, tested with a table of cases in milliseconds.

Wrap what you do not own

Freeman and Pryce add a rule that sounds fussy until it saves a team a month: only mock types that you own. When our code depends directly on a third-party library or a vendor’s API, a mock of that interface encodes our guess about how the vendor behaves, and the guess goes untested. The remedy is an adapter, a thin layer that speaks the vendor’s language on one side and our domain’s language on the other. The adapter is tested against the real vendor, or its sandbox, by integration tests; everything else in our code depends on the adapter’s interface, which we own, and is tested against a fake of it.

Encore already has the structure this rule needs. Orders & Payments talks to its two payment service providers (PSPs) through an anti-corruption layer, the Architecture book’s term for exactly this kind of translating boundary, owned by the PSP Integrations team. The testing consequence is a division of labor. PSP Integrations runs integration tests of the layer against each provider’s sandbox, including, now, the primary provider’s marketplace split-payment product, through which resale money moves so that Encore never holds it. Every other team tests against the layer’s fake, which PSP Integrations ships and maintains, and Section 3.3 turns that fake into something more ambitious.

What the evidence says about TDD

Test-driven development (TDD), in which the test is written before the code, has been argued about for two decades with more heat than data. The most careful study I know is by Davide Fucci, Hakan Erdogmus, Burak Turhan, Markku Oivo and Natalia Juristo, published in 2017. They dissected the process into four characteristics: sequencing, whether the test or the code came first; granularity, how small each cycle was; uniformity, how steady the rhythm of cycles was; and refactoring effort. Across 82 data points from 39 professionals, granularity and uniformity were associated with better external quality and productivity. Sequencing, the defining feature of test-first, had no important influence. Refactoring effort, surprisingly, was associated with worse outcomes on both. Their tasks were short and their participants came from one company’s workshops, so these are associations, not causal effects on long-lived systems, and the authors say so. Earlier work points in a compatible direction. Nachiappan Nagappan and colleagues studied one IBM team and three Microsoft teams in 2008 and found pre-release defect density 40 to 90% lower than in comparable projects without TDD, with the teams estimating that initial development took 15 to 35% longer; those were case studies, without randomization. A 2013 meta-analysis by Yahya Rafique and Vojislav Mišić found a small improvement in external quality, mostly against waterfall comparisons, and no clear effect on productivity. The honest summary is that controlled studies do not show that writing the test first matters, and they do support working in small, steady steps with fast feedback. Encore asks for small steps, a test with every change and an acceptance test for every feature, and leaves the order of the inner loop to the engineer.

Recap. Build the walking skeleton first, so that every feature is born into a pipeline and a harness that reach the whole system. Work in two loops, an acceptance test outside and small test-backed steps within, and treat a hard-to-write test as a report about the design. Wrap what you do not own and test the wrapper against the real thing. The evidence favors small, steady steps; the order of test and code within a step matters less than its advocates claim.

Exercise 3.2. Before you write any code for your next feature, write its acceptance test in the language of the domain, using no names of services, endpoints or tables. Note which design decision the test forced you to make, and which question it forced you to ask another team.

3.3 Contracts and Simulators

Consumer-driven contracts across forty teams

The Architecture book (Sections 5.2 and 7.2) made the argument for consumer-driven contracts and extended them to the public API’s edge. Ian Robinson set out the pattern in 2006: each consumer of a service publishes the expectations it actually relies on, and the provider runs all of them as tests, so that it learns before shipping which of its consumers a change would break. Pact, the tool most teams now use, came seven years later, written in 2013 at realestate.com.au while that company was building Ruby microservices. What this section adds is the operational side: what it takes to run the pattern across forty teams without its becoming the shared bottleneck it was meant to remove.

Four pieces of machinery do most of the work. A broker stores every consumer’s contract, versioned, with the results of every provider’s verification, so that a pipeline can ask whether two versions have been checked against each other. Provider states let a contract say what must be true before an interaction (“given a ticket that has been listed for resale”), and the provider’s test harness sets that state up, so that consumers do not need to know how. A deployment check, Pact’s can-i-deploy command, asks the broker whether the version about to ship has been verified against every consumer version currently deployed in the target environment, and the pipeline refuses to deploy if not. And a rule about ownership settles the argument a failed verification would otherwise start: at Encore the provider fixes a broken verification, or negotiates with the consumer before merging, and never merges first. Within a month the Inventory team’s reissue endpoint had three consumers, Resale, Orders & Payments and Venue Entry, all checked on every pull request.

Contracts check shape. They cannot check meaning, as the Architecture book’s example of euros and cents showed, and they need one extension for a company built on events. Most of what Verified Resale publishes is asynchronous: TicketReissued, TicketRevoked, the listing and sale events that feed the finance ledger. Request-response contracts say nothing about those. Encore covers them in two ways. Pact’s message contracts let consumers of an event state the fields they rely on, verified like any other contract, and the Event Backbone team’s schema registry enforces compatibility at publish time, so a producer cannot register a new version of TicketReissued that would break consumers reading the old one. The registry catches the structural break on every publish; the message contract catches the consumer who relied on a field the schema allowed to be optional.

When the consumer cannot run your tests

Consumer-driven contracts assume something easy to miss: that the consumers are known, that they can write tests, and that the provider can run those tests in its own pipeline. Inside Encore all three hold. At the edge they fail. Rowmate is an independent company with its own release schedule and no obligation to publish its expectations, and Tessera will be another. The partners Encore most needs to protect, the small ones who integrate once and touch the code again only when it breaks, are precisely the ones least likely to maintain a contract.

When the consumer cannot hand the provider its tests, the provider must hand the consumer something to test against instead, and the best-known public example shows how. Stripe’s test mode gives every integrator a sandbox that behaves like the real API, and its documented test card numbers produce specific outcomes on demand: a decline, a card flagged for fraud, a payment that requires 3-D Secure authentication. An integrator can exercise every failure path without waiting for it to happen in production. The sandbox is, in effect, the provider’s contract made executable, published for anyone to run.

Encore publishes two things for its partners. The partner sandbox is a hosted copy of the partner API, populated with synthetic events, in which each partner gets an isolated account with its own test fans, tickets and organizer rules; partners receive sandbox credentials as soon as they sign. The partner simulator is a container image of the same API that a partner can run inside its own continuous integration (CI) pipeline, versioned with the API. Both can produce the awkward cases on demand: an organizer tightening a rule while a listing is live, a reissue that takes 40 seconds, a webhook that arrives twice or late, a listing rejected because it breaks the cap. The testing discipline that makes them trustworthy lies on Encore’s side. The Partner APIs team runs the same suite of behavior tests against the production API, the hosted sandbox and the simulator image on every release, so that a partner who passes against the simulator has passed against behavior Encore has pinned. How partners are certified on the sandbox before they get production credentials belongs to Section 8.3.

One property of the sandbox was designed in from the start, and it came out of a painful memory. Section 2.5 told the story: Rowmate’s integration with the old Partner Listings product had read the event-code prefix embedded in the legacy barcode, a detail Encore never promised and Rowmate could nonetheless observe, and the integration broke against the new opaque ticket identity the first time Rowmate ran it in the sandbox. That is Hyrum’s law, which the Architecture book’s Section 7.2 states for public interfaces. Verified Resale gives partners an opaque ticket_id through the API and never exposes a barcode, which rotates in any case, and the sandbox goes further: it issues ticket_id values in a deliberately different format from production, so that any partner whose code tries to parse them fails in the sandbox, on the partner’s own schedule, instead of on Encore’s next release.

Simulators as products

The same reasoning applies to the vendors Encore depends on. Each outside service on the purchase and resale paths has a simulator, a fake in Google’s sense, owned by the team that owns the integration and shipped with it. Each does what the vendor does on a good day and, on request, what it does on a bad one (Figure 3.5).

Figure 3.5 — Encore’s simulators. Each belongs to the team that owns the integration, and each can produce the vendor’s bad days on demand.

The “slow to 8 s” mode replays the first big outage in Encore’s history, when the PSP slowed from 200 ms to 8 s and a retry storm, amplified by autoscaling, took the site down (the Architecture book, Chapter 4). Every team on the purchase path now has a test that sets the primary PSP simulator to that mode and checks that the breaker opens, new attempts move to the secondary provider and nothing retries in two layers. The Resilience book’s game day (Section 5.5) injected a similar delay into production-like conditions; the simulator is its pre-production twin, run on every build. Resale payments run only through the primary provider’s marketplace product, a documented limitation, so the secondary simulator deliberately has no split mode, and a test asserts that a resale purchase with the primary breaker open fails cleanly with an honest message to the fan, never silently routing to a provider that cannot split the money.

A simulator is a product in Chapter 6’s sense, with other teams as its users: the ones that get used have documentation, a changelog and an owner who answers questions. Chapter 4 shows the same simulators in the inner loop, where an engineer runs the purchase path on a laptop without touching any vendor.

Keeping a fake honest

A fake is a model of a vendor, and Section 3.1 has already shown what happens to models nobody checks against the world. Google’s chapter on test doubles states the requirement plainly: “A fake must have its own tests to ensure that it conforms to the API of its corresponding real implementation”, and it suggests running the same contract tests against both the real implementation and the fake. Encore does exactly this. Each simulator’s owning team keeps one suite of behavior tests and runs it on a schedule against the simulator and against the vendor’s own sandbox; when the two disagree, the simulator is wrong until proven otherwise, and the owning team fixes it before the drift reaches anyone’s build.

That check has a limit, and it is Ariane’s lesson in miniature. A vendor’s sandbox is itself a simulator, written by the vendor, and it can differ from the vendor’s production in ways the vendor never documents. Running our suite against the sandbox tells us that our fake agrees with their fake, and nothing about whether either agrees with the world. The remaining distance can be closed only in production, and Encore closes it there with the two instruments other books describe: the synthetic buyers that run the real purchase path every hour against a hidden test event, paying through the providers’ sandbox (the Architecture book, Section 5.3), and the canaries that expose a few percent of real traffic to each new version (the Resilience book, Sections 4.3 and 4.4). For Verified Resale the synthetic buyers gained a hidden test listing, so that every hour a scripted fan resells a ticket to another scripted fan, and a scripted scanner confirms that the old barcode is refused. The Ariane board’s second recommendation, as much real equipment as technically feasible, is what this arrangement reaches for, accepting that some of the real equipment can only be met in flight.

Properties over examples

Most tests are examples: given this input, expect that output. They test only what their author thought of. A property-based test states instead a rule that must hold for every input, lets a generator produce hundreds of random inputs, and, when one fails, shrinks it to the smallest input that still fails, so that the report reads like a minimal bug report. Koen Claessen and John Hughes introduced the idea with QuickCheck for Haskell in 2000, and it has since spread to nearly every language. Its most powerful form is stateful or model-based: the generator produces random sequences of operations, and the test checks after every step that the system agrees with a simple model of what it should be doing.

The best public industrial account is from Amazon. In 2021 James Bornholt and colleagues described how the team building ShardStore, a storage node for Amazon S3, wrote an executable reference model and used property-based testing to check that random sequences of operations, including crashes and concurrent requests, gave the same results on the model and on the real implementation. The paper reports that the approach had prevented 16 issues from reaching production, including crash-consistency and concurrency bugs, and that engineers who were not formal-methods specialists extended the checks themselves. It is the right precedent for barcode issuance, because issuance has exactly the shape the technique handles best: a small model (“each seat has one live ticket”) and an implementation full of retries and ordering.

Verified Resale adds two facts that example tests handle badly. A ticket may be resold, and so reissued, more than once. And a retried reissue request carrying the same idempotency key must be a no-op, whatever happened to the first attempt. The Inventory team’s property test runs the real issuance code, with its storage in memory, against random sequences of reissues and retries. A small harness holds the service and the set of keys used so far, which is the whole of the model:

class ReissueProperties {
    class Harness {
        val issuance = IssuanceService(InMemoryStore())   // real code, small test
        val keys = mutableSetOf<String>()
        init { issuance.issue(seat = "B-14") }
    }

    @Property
    fun reissuesKeepOneLiveTicket(@ForAll("reissues") chain: ActionChain<Harness>) {
        chain.withInvariant { h ->
            assertEquals(1, h.issuance.liveTickets(seat = "B-14").size)
            assertEquals(h.keys.size, h.issuance.reissueCount(seat = "B-14"))
        }.run()
    }

    @Provide
    fun reissues(): Arbitrary<ActionChain<Harness>> =
        ActionChain.startWith { Harness() }.withAction(Reissue)

    object Reissue : Action.Independent<Harness> {
        override fun transformer(): Arbitrary<Transformer<Harness>> =
            Arbitraries.of("k1", "k2", "k3", "k4").map { key ->   // repeats model retries
                Transformer.mutate("reissue($key)") { h: Harness ->
                    h.issuance.reissue(seat = "B-14", idempotencyKey = key)
                    h.keys.add(key)
                }
            }
    }
}

The barcode core is Kotlin, so the team used jqwik, a property-testing engine for the JVM that runs on JUnit 5 and supports stateful testing through action chains. It generates sequences such as “reissue with k1, reissue with k1 again, reissue with k3”, checks both properties after every step, and on a failure shrinks the sequence to the shortest one that still breaks them. The test found a real fault in its first week: under a particular interleaving the old code wrote the new ticket before revoking the old one, so for one step two tickets were live. The five fairness invariants of Section 1.3 are written the same way, as properties in CI (Section 11.3 shows them in the capstone), and the money has one of its own: for every generated price, channel and jurisdiction, the buyer’s payment equals the seller’s payout plus the three shares, to the cent. How such spec-level tests are guarded against being edited away by an author, human or machine, is the Resilience book’s subject (Section 4.5); this section is only about how to write them.

A close relative of property testing deserves a paragraph, because Verified Resale hands it two ideal targets. Fuzzing feeds a program large volumes of generated or mutated input, guided by which inputs reach new code, and watches for crashes, hangs and broken invariants. It is the established defense for parsers and anything that reads untrusted input. Google’s OSS-Fuzz service reports that as of May 2025 it had “helped identify and fix over 13,000 vulnerabilities and 50,000 bugs across 1,000 projects”. Partner payloads and the barcodes venue scanners read are both untrusted input to parsers, and both now run under a fuzzer nightly.

Recap. Consumer-driven contracts work across forty teams when a broker, provider states, a deployment check and a clear rule about who fixes a broken verification stand behind them, with message contracts and schema compatibility for events. When consumers cannot run our tests, we publish a sandbox and a simulator pinned by our own suite. Fakes of vendors belong to the teams that own the integrations, and they stay honest only if the same tests run against them and the vendor’s sandbox, with production checks closing the last gap. Properties, model-based and shrunk, test the rules examples cannot reach.

Exercise 3.3. Pick one external dependency of your system. List the five failure modes your tests can produce on demand today, then the five you most fear in production. Every fear without a matching mode is a gap; choose the one you would add to a simulator first.

3.4 End-to-End Without Despair

A few journeys, each with an owner

End-to-end tests are the most persuasive tests we can write and among the most expensive to keep, and the Architecture book (Section 5.3) gave the reason: they are slow, they fail for reasons unrelated to the change under test, and when they fail they rarely say whose code is at fault. Its advice was to keep only a handful, written as journeys a business would recognize. The clearest public account of keeping that handful healthy comes from Lyft.

In a series of posts on scaling productivity across its microservices, Lyft’s engineers Ben Radler, Garrett Heel and Shu Zheng described a fleet of about 900 services whose integration suites had grown until they took upwards of an hour to run and gave a very poor ratio of signal to noise. Lyft kept a small set of end-to-end scenarios for its critical flows, logging in, requesting a ride and paying for it, and made them acceptance tests that run in staging after each deploy and gate the promotion to production. InfoQ’s report on the change describes thousands of the old integration tests being removed or rewritten as unit tests, with no measurable rise in production bugs. The flows were well chosen: Lyft classes a failure in any of them as a SEV0, its most severe kind of incident, so nobody needed convincing that each deserved a test.

Encore keeps eleven end-to-end journeys, and each has a named owning team, the one whose pager would ring if the journey broke in production: a fan buys a ticket in an on-sale, a fan completes a pending payment, an organizer publishes an event, a venue scans a ticket offline, and so on. The owner fixes the journey when it fails, keeps its runtime within budget and decides when it should be retired, which happens when a lower-level test has come to cover everything the journey catches. DORA’s finding that performance improves when “developers are primarily responsible for creating and maintaining suites of automated tests” is the general form of the rule; an end-to-end journey without an owner becomes everyone’s flaky test and nobody’s job. Verified Resale added one journey, the resale of Section 3.2’s acceptance test running against the real system with the Rowmate stub replaced by the partner simulator, owned by the Resale team. Everything else the feature needs to prove lives lower down, where a failure points at one team.

Test data as a product

Much of the cost of an integration or end-to-end test is the data it needs. A resale test needs an organizer with rules, an event with an on-sale schedule, a fan who bought through the Gate within the per-fan limit and a partner with credentials, and building all of that by hand in each test is how test suites become unmaintainable. Freeman and Pryce’s answer is the test data builder: an object that constructs a valid domain object with sensible defaults and lets each test override only the fields it cares about, so that a test reads anEvent().withResaleDisabled() and the reader sees at once which fact matters. Humble and Farley’s Continuous Delivery (2010) adds the discipline at the environment level: every test creates the data it needs, or starts from a known baseline that resets, and no test depends on data another test left behind.

At Encore’s scale test data becomes a product with an owner, like the simulators. The Resale team’s builders live beside the service and are used by every team that tests against it. Synthetic organizers and venues are created through the same onboarding workflow that real ones go through (Section 7.1), so a change to onboarding cannot quietly break every test that needs a tenant.

What each environment can tell you

Humble and Farley devote a chapter to managing environments, and their underlying point is that an environment is an instrument with a known range, answering some questions and silent on others. A developer’s machine with simulators can tell us whether our code behaves correctly against the documented behavior of its neighbors. A shared staging environment can tell us whether services integrate, whether configuration is right and whether a deployment works. Neither can tell us how the system behaves under production load, against a vendor’s real production behavior or with the data real fans create. Those questions can be asked only in production, carefully, as the previous section described.

The interesting choice is between shared staging and an environment per change. Lyft’s move from Onebox, a full copy of its stack per developer, to request-level isolation inside shared staging is the clearest public evidence, and Section 4.1 tells it along with how Encore builds and pays for its own environments. The short version is that full per-change environments buy isolation at a price that grows with the number of services, and at some fleet size isolating requests becomes cheaper than duplicating everything. For a testing strategy the discipline is to write down, beside each test, which question it asks of which environment, much as Encore now requires every load test to state the cache state it assumed.

Why tests flake

A flaky test is one that can pass and fail on the same code. Flakiness kills end-to-end suites by the mechanism of Section 3.1: each false alarm teaches people that red means nothing, until a real failure is rerun and ignored like the others. John Micco reported Google’s numbers in 2016. About 1.5% of all test runs reported a flaky result, almost 16% of tests had some level of flakiness, and about 84% of the transitions from pass to fail that Google observed involved a flaky test. Google’s 2020 book adds a judgment about thresholds: “as you approach 1% flakiness, the tests begin to lose value”. These are two measurements made in different years by different methods, and they should be read separately.

The causes are better understood than the remedies are practiced. Qingzhou Luo, Farah Hariri, Lamyaa Eloussi and Darko Marinov studied flaky tests in 51 Apache Software Foundation projects in 2014, inspecting commits that fixed them and classifying the root cause of 161. Figure 3.6 gives their three largest categories, with the design that prevents each.

Figure 3.6 — The three largest causes of flaky tests in Luo et al.’s study, and the test design that prevents each.

A further finding from the same study shapes how Encore fights flakiness. Some 78% of the flaky tests were flaky from the moment they were written, so flakiness is mostly a design defect present at birth, which argues for catching it at review time more than for repairing it years later. Two disciplines prevent most of what remains. Control time and randomness: inject the clock and the random seed into the code under test, so that a test about a resale window that closes two hours before doors can set the clock instead of waiting for it. And keep large tests as hermetic as possible, because, as Google’s chapter on larger tests puts it, “if a large test is nonhermetic, it is almost impossible to guarantee determinism”. Encore measures flakiness per test and per journey, by rerunning failures on unchanged code. When Verified Resale began, the end-to-end suite failed spuriously on about one run in 40, far past the point where Google’s book says tests lose value. How CI detects flaky tests, quarantines them and decides when to rerun them is a property of the pipeline, and Section 4.4 owns it; this section owns the reasons tests flake and the designs that keep them from doing so.

Acceptance tests in domain language

The acceptance test in Section 3.2 spoke of fans, listings and scanners and never of pages, buttons or endpoints, and that was deliberate. Humble and Farley describe the structure in their chapter on automated acceptance testing: the test is written against an application driver, a layer that knows how to perform domain actions on the real system, and only the driver knows about the user interface or the API. When the interface changes, one driver changes, and the tests, which express what the business needs, stay as they are. Without such a layer, every visual change rewrites dozens of tests that were never about the visuals.

Encore’s drivers are owned like any other shared code. The resale driver has two implementations, one that acts through the partner API and one that acts through the fan app’s backend, and the same acceptance tests run against both, which is how Encore knows that a fan reselling through the app and a fan reselling through Rowmate experience the same rules. DORA puts the ownership plainly: “Primary authors and maintainers of acceptance tests are developers.” At Encore the trio decides what the acceptance tests should say, and the engineers write and keep them.

Recap. Keep a few end-to-end journeys, each owned by the team whose pager it would wake, and retire them when lower-level tests cover what they catch. Treat test data and environments as instruments with known ranges, and write down which question each test is asking of which environment. Flaky tests are mostly born flaky, from asynchronous waits, races and shared state, and the designs that prevent them are cheaper than the reruns that hide them. Write acceptance tests in the domain’s language against a driver, so that a redesign changes one layer.

Exercise 3.4. Measure the flake rate of your end-to-end suite for one month by rerunning every failure on unchanged code. Name the owner of the worst journey, classify its flakiness by Figure 3.6’s causes, and estimate what fixing it would cost against what its reruns cost now.

3.5 Evals as Product Practice

Everything so far assumes that a correct output exists and can be stated in advance. The organizer rule assistant breaks that assumption. An organizer types a request in plain language, such as “no resale until a week after the on-sale, two per fan”, and the assistant, a large language model (LLM) behind Encore’s AI gateway, drafts a typed rule set and a plain-language summary, which the organizer confirms before anything applies (the Products book designs it in Section 6.5). Part of its correctness is guaranteed by construction. The assistant can express only rules the type system allows, so the third fairness invariant, “Organizer caps and rules can only tighten”, holds whatever the model says, and an example test proves it. The rest is a matter of judgment. Did the draft capture what the organizer meant? Is the summary clear? Did it ask when the request was ambiguous? Those questions have answers but no single correct output, and the same request produces different drafts on different days. An eval is a test for this kind of system: a set of inputs with criteria for judging the outputs, run repeatedly, whose result is a rate of success by category. The practice is young, documented mostly by practitioners with a small body of research, and the subsections below say which is which.

Who decides what good means

The first question about an eval is not technical. Someone must decide what a good draft is, and the people best placed to decide are rarely the engineers who wrote the prompt. Shreya Shankar and Hamel Husain, whose Evals for AI Engineers (2026) collects the clearest practitioner guidance, give advice, in their published answers to common questions about evals, that surprises teams used to consensus: “For most teams, the most effective strategy is to appoint a single, internal domain expert as the final decision-maker on quality.” They call this person a benevolent dictator. A committee averages away the judgments that matter, while one expert, consulted often, produces consistent labels quickly and can explain them. This is practitioner experience, with no controlled study behind it, and it fits what Encore knew from its support assistant.

For the rule assistant, the Resale trio asked the organizer-success lead to decide, and she accepted. She had spent years helping organizers configure events, and knew what “a week after the on-sale” means to a festival promoter whose on-sale runs in waves. She labels; the product manager decides what the assistant should attempt; the engineers build the machinery. How to check that her labels agree with a second labeler, and how to calibrate an automated judge against her, is taught in the Resilience book (Section 8.3), and Encore follows it.

Criteria drift

The second surprise is that the expert does not arrive with her criteria ready-made. Shankar and colleagues named the phenomenon in a 2024 paper, “Who Validates the Validators?”, which studied nine industry practitioners building evaluators for LLM pipelines. They called it criteria drift: “users need criteria to grade outputs, but grading outputs helps users define criteria.” People discover what they mean by “good” while they grade, and their criteria change as they see more outputs. The study is small and qualitative, and the phenomenon it names will be familiar to anyone who has coded interview transcripts, where categories always shift during the first pass.

The rule assistant’s first labeling session showed it within an hour. The original criterion said a draft was good if its rules matched the request. Then the lead met “two per fan”, graded three drafts that had each chosen differently between the listing limit and the buying limit, and realized that none of them was right, because the request was ambiguous and a good assistant should ask. The criterion changed: a good draft either matches an unambiguous request or asks one clarifying question about an ambiguous one. A criterion written in advance and frozen would have graded those drafts on a question the organizer never answered.

Criteria before prompts

The analogy with Section 3.2 is close, and it needs care at one point. An acceptance test written before the code states a goal precisely and then holds still. For the rule assistant, the equivalent is to write down what good looks like before anyone writes the first prompt: a first set of real requests, drawn from the design-partner interviews and from support cases about resale settings, each with a provisional criterion. Then the prompts and the evals are iterated together, and after each reading of real outputs, the criteria are revised where the reading taught the lead something. Writing the criteria first keeps the team from tuning the prompt to whatever the model happens to produce; revising them keeps the team honest about criteria drift. Shankar and Husain’s method begins from reading real traces before building automated evaluators, and Chip Huyen’s AI Engineering (2025) gives evaluation a central place in building AI applications; the Resale team did both, in that order. Here is one case from the set, after its third revision:

# rule-assistant/cases/two-per-fan-ambiguous.yaml
request: "No resale until a week after the on-sale, two per fan."
event: {on_sale: "2027-03-05T10:00+01:00", doors: "2027-06-12T18:30+02:00"}
expect:
  window_opens: "on_sale + 7d"
  window_closes: "doors - 2h"          # the default, unchanged
  clarifying_question: about listing limit vs buying limit
  must_not: [apply_any_rule_before_confirmation]
summary_must_say: ["one week after the on-sale"]
criterion_version: 3                     # v1 required a limit; v2 allowed either
labelled_by: organizer-success lead

The first set held 60 cases across the categories in Figure 3.7. By launch it had grown to about 250, most of the growth coming from failures found while reading outputs, each turned into a case.

Figure 3.7 — The rule assistant’s first labeled set, by category. The categories came from the organizer-success lead’s first reading of outputs, and two of them did not exist before it.

A screen for reading

An eval’s quality is bounded by how many outputs the expert actually reads, and that depends heavily on how painful reading is. Shankar and Husain give a whole chapter to interfaces for human review, and their advice is that “the best way to empower the domain expert is to give them custom annotation tools that display system outcomes alongside traces.” Their supporting examples come from research prototypes and consulting practice, so the claim that a custom screen beats a generic tool is practitioner judgment, without a measured comparison. It is also cheap to try. In two days the Resale team built a page showing, for each case, the request, the event’s dates, the drafted rules as the console would render them, the summary, and pass, fail and “criterion unclear” buttons. The lead’s reading rate in an afternoon went from a few dozen cases in a spreadsheet to well over a hundred, and the “criterion unclear” button produced most of the criteria revisions.

Evals in the build

Once the criteria have settled enough to automate, the eval joins the pipeline like any other test, and the series has already taught how: the Architecture book (Section 12.4) runs evals in CI as fitness functions, and the Resilience book (Sections 8.2 and 8.3) runs each case several times, gates per category and grows the golden set from production failures. What this chapter adds is the path into the build. Every failure the lead marks becomes a case in its category, with the criterion version that judged it. A change to the prompt, the model or the rule schema runs the whole set, through the same AI gateway the assistant uses in production, before it can merge. And the typed-rule examples that prove invariant 3 run beside the eval in the same stage, so the guarantee made by construction and the judgment made by the expert are checked together. Repeated runs are needed because the model is not deterministic, and a gate that judged each case once would be a flaky test by another name.

Recap. Evals are tests for systems with no single correct output, and the practice is young. One domain expert, given authority and a good screen, decides what good means; her criteria will drift as she reads, so criteria are written before prompts and revised after reading outputs. Failures become cases, and the cases run in the build beside the conventional tests that prove what the type system guarantees.

Exercise 3.5. For one AI feature in your product, write ten labeled examples and a provisional pass criterion before anyone changes its prompt. Then read twenty real outputs and record every place where the criterion did not tell you what to decide.

Kata: One Run in Twelve

Your brief: “Repwell.” Repwell sells class-booking software as a service to 900 gyms and fitness studios, from single-room yoga studios to chains with 40 clubs. Sixty engineers in seven teams build it. Members book classes through a web widget and a mobile app, and studios manage timetables, waitlists, reminders, memberships and payments in a back office. The test suite has grown for nine years. It takes 70 minutes on the CI cluster, and about one run in twelve fails for reasons nobody can reproduce, so engineers rerun failures by habit and merge on the second green. Coverage is 84%, a target leadership set three years ago. Most unit tests mock every collaborator, including the company’s own database. Last quarter a reminder-scheduling bug shipped to every studio, although 31 tests touched the code, because none of them checked a reminder across a daylight-saving change. This quarter Repwell must integrate a corporate-wellness vendor, which confirms before a booking is accepted whether a member’s employer-funded pass covers the class. The vendor offers a sandbox, publishes no simulator, has an uptime commitment of 99.5% and answers slowly at the start of each month.

Deliverables:

1. The suite’s worth — Score a random sample of 20 tests on the four pillars (Figure 3.2), and propose what to delete, rewrite or keep, with the evidence for each group. Say what to do about the coverage target.
2. The integration’s test strategy — Design the adapter around the eligibility vendor, the tests at the adapter, and the simulator every other team will use, with its failure modes on demand and the suite that keeps it honest against the vendor’s sandbox. Say what only production can tell you, and how you will learn it.
3. Properties — Write, in prose or code, two properties for the booking core: one about overbooking a class under concurrent requests, one about reminders across time zones and daylight-saving changes, with the clock injected.
4. The flaky-test plan — Describe how you will measure flakiness per test, classify the worst offenders by cause, and redesign them, with a target rate and a date. Say which part belongs to test design and which to the CI pipeline.
5. End-to-end journeys — Choose at most five journeys to keep, name an owning team for each, and explain what happens to the rest.
6. A 70-minute budget — Show how the suite’s runtime falls, by test size and by what runs on each change versus nightly, and what you will measure to know the trade has not cost protection.

Rubric:

Where you now stand

You can now judge what a test is worth and spend testing effort where it pays: grow a system from a walking skeleton, replace the neighbors you cannot run with simulators you keep honest, state the rules that matter as properties, keep a few owned journeys at the top and design flakiness out. You can also build evals where no single correct output exists. What a suite cannot do on its own is decide how quickly its verdict reaches the engineer who needs it. A test that takes an hour to report invites the same habits as a flaky one: engineers stop waiting for it, and then stop reading it. Chapter 4 follows a change from a laptop to main, through the inner loop, the build, code review and the CI pipeline that runs all of these tests, and asks how to make that journey short enough that engineers still listen.

References

• J. L. Lions et al. — ARIANE 5 Flight 501 Failure: Report by the Inquiry Board. ESA and CNES, Paris, 1996 — free online.
• Vladimir Khorikov — Unit Testing Principles, Practices, and Patterns. Manning, 2020.
• Titus Winters, Tom Manshreck, Hyrum Wright (eds.) — Software Engineering at Google (chapters 11–14). O’Reilly, 2020 — free online.
• Steve Freeman, Nat Pryce — Growing Object-Oriented Software, Guided by Tests. Addison-Wesley, 2009.
• Davide Fucci, Hakan Erdogmus, Burak Turhan, Markku Oivo, Natalia Juristo — A Dissection of the Test-Driven Development Process: Does It Really Matter to Test-First or to Test-Last?. IEEE Transactions on Software Engineering 43(7), 2017 — free online.
• Ian Robinson — Consumer-Driven Contracts: A Service Evolution Pattern. martinfowler.com, 2006 — free online.
• Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, Darko Marinov — An Empirical Analysis of Flaky Tests. FSE, 2014.
• John Micco — Flaky Tests at Google and How We Mitigate Them. Google Testing Blog, 2016 — free online.
• Goran Petrović, Marko Ivanković — State of Mutation Testing at Google. ICSE-SEIP, 2018 — free online.
• Koen Claessen, John Hughes — QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. ICFP, 2000.
• James Bornholt et al. — Using Lightweight Formal Methods to Validate a Key-Value Storage Node in Amazon S3. SOSP, 2021 — free online.
• Hamel Husain, Shreya Shankar — Frequently Asked Questions (And Answers) About AI Evals. hamel.dev, 2026 — free online.
• Shreya Shankar, J.D. Zamfirescu-Pereira, Björn Hartmann, Aditya G. Parameswaran, Ian Arawjo — Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. UIST, 2024 — free online.
• Jez Humble, David Farley — Continuous Delivery. Addison-Wesley, 2010.
• Nachiappan Nagappan, E. Michael Maximilien, Thirumalesh Bhat, Laurie Williams — Realizing Quality Improvement Through Test Driven Development: Results and Experiences of Four Industrial Teams. Empirical Software Engineering 13(3), 2008.
• Chip Huyen — AI Engineering. O’Reilly, 2025.