Building Products in the AI & Cloud Era

← the whole book on one page

Chapter 10: Capstone — Verified Resale from Idea to Launch

A bet written down before we knew how it would end is the only kind we can learn from.

On 1 April 2004 Google issued a press release headed “Google Gets the Message, Launches Gmail.” It offered free web mail with “1,000 megabytes (1 gigabyte) of free storage”, at a time when Hotmail’s free accounts held 2 MB and Yahoo Mail’s 4 MB, so that Gmail’s gigabyte was about 500 times Hotmail’s free allowance. Given the date and the size of the offer, many readers took it for an April Fools’ joke, and the same day Google did announce a real hoax, a research center on the Moon. Brian Rakowski, Gmail’s first product manager, later told Harry McCracken of TIME that “the ultimate April Fools’ joke was to launch something kind of crazy on April 1st and have it still exist on April 2nd.” The service behind the joke had been nearly three years in the making. Paul Buchheit, Google’s 23rd employee, began it in August 2001 under the code name Caribou, and built a first searchable mail prototype in about a day by adapting the search code of Google Groups. About a dozen people were working on it by launch. They ran it on about 300 old Pentium III machines that nobody else at Google wanted, and Buchheit described the trap they were in with some precision: “We couldn’t get many machines because people thought we couldn’t launch.”

What Google did with those machines is the part that concerns us. It did not open Gmail to everyone. It gave accounts to about 1,000 people outside the company, let each of them invite a couple of friends, and widened slowly from there. McCracken is careful about the motive: “The limited rollout had been born of necessity, but ‘it had a side effect,’ says Harik”, meaning Georges Harik, who oversaw most of Google’s new products at the time. The side effect was demand. Invitations became a prized possession and sold on eBay for $150 and more. The gate turned out to do a third job as well, one nobody had designed it for. On 6 April, five days after launch, 31 organizations and advocates wrote to Larry Page and Sergey Brin asking Google to suspend Gmail, because it scanned the text of messages to place advertisements beside them, and California State Senator Liz Figueroa drafted a bill to require consent before mail could be scanned for ads. Google met that objection while its users numbered in thousands. The gate stayed in place for nearly three years, until sign-up opened to all on 14 February 2007, and the “beta” label stayed on the logo until 7 July 2009, when Google removed it from Gmail, Calendar, Docs and Talk together. By then the label carried a cost nobody had counted at the start: business customers did not want to adopt something that sounded unfinished, and Google said as much when it took the label down. Gmail grew all the same, to 425 million active users by 2012, although nothing in the record suggests that the long beta caused the growth, only that it did not prevent it. Figure 10.1 sets those dates on one line, with the gate and the label drawn as the spans they were.

Figure 10.1 — Gmail from Caribou to 425 million users. The invitation gate lasted under three years, the beta label more than five; below, the storage offer of April 2004 to scale, where Hotmail’s and Yahoo’s allowances are hairlines.

I tell this story at the start of the last chapter because one launch shows, in small, what this book has argued in parts. A launch gated by who may enter is a product decision as much as a capacity one, and it does more jobs than its designers intend. Google meant to ration machines, and found that it had also manufactured demand, bought time to answer a public objection about privacy, and, years later, put off the customers who would pay the most. Some of that was luck, good and bad. The lesson I draw is modest and practical. Write down which jobs a launch is meant to do before it is done, so that afterward we can tell what we planned from what merely happened. That is also the method of this chapter. The Resilience book’s capstone attended one review, and the Operations book’s lived through one day. A product fits neither shape, because it is a bet that is made, tested, built, shipped and judged over many months. So this capstone follows one document, the bet memo for Verified Resale, through the five checkpoints at which Encore read it again.

Encore, at about 400 engineers, wrote that memo in the late winter of its ninth year. The Vesper Hale on-sale and the added stadium dates were behind it, and so was the Rowmate key leak, which had shown how little Encore controlled the resale that already ran through its partner application programming interface (API). Chapters 1 to 9 followed Verified Resale through one practice at a time: strategy, discovery, measurement, pricing, design, the design system, the phone and the partner API. Here we follow it through time, and Figure 10.2 lays out the route. Each section that follows is one checkpoint, and each names the chapters whose practices it draws together. The two capstones divide the work. This one asks why Encore made the bet and whether it paid, up to the review at month 18. The Software book’s Chapter 11 asks how the code and the engineering system changed to deliver it, and reviews that change at month 24. When this chapter reaches engineering, it states the outcome and points there.

Figure 10.2 — One decision through 18 months. Each checkpoint rereads the same memo; the two lower bands show where the practices it draws on are taught, in this book and in the Software book.

10.1 The Bet

The diagnosis

The memo’s first page is a diagnosis, written in the shape Richard Rumelt gave good strategy and Section 2.1 taught: a diagnosis of the challenge, a guiding policy for meeting it, and coherent actions that carry the policy out. The challenge was not new, and that was part of the difficulty. Since its sixth year Encore had run Partner Listings, which let authorized partners read its event listings and post resale offers that appeared as “resale available at Rowmate” links on sold-out event pages. The ticket itself changed hands outside Encore. The seller forwarded the original pass, and Encore checked nothing beyond the partner’s word. The memo named four consequences of that arrangement. The same barcode could be sold to several buyers, and only the first through the door got in. Organizers had no control over price or timing. Organizers took no share of the resale fee. And buyers were never screened, so the bots that Bot Screening turned away from the primary sale could buy freely on the secondary one. The Rowmate leak had made the last point concrete, because the leaked key, scoped only to listings:read and resale:write, was enough to create two fraudulent listings, removed before anyone bought them. The rest of the evidence was in Encore’s own records. In the 12 months before the memo, Support had handled about 3,400 cases of dead or fake resold tickets, and within a day of the Vesper Hale on-sale more than 9,000 of its tickets were on unofficial sites.

A diagnosis has to say what the harm comes from, and Section 2.1 had already settled that: the harm comes from resale that nobody controls, and resale as such is something fans need. That sentence did a great deal of work. It turned the objection that had hung over the idea from the beginning (“we fight scalpers; now we host resale?”) into a design constraint. Encore would not host resale on anyone’s terms. It would host it on terms it could enforce at the barcode, and the guiding policy said so in one line: bring resale inside Encore, where every rule can be enforced on the ticket itself. The coherent actions followed from the policy. Every resold ticket would be reissued under a new barcode, every organizer’s rules would be applied, only vetted partners would be admitted, and the same bot controls would guard the resale checkout as guarded the primary one. Partner Listings would be retired, with a year’s notice to partners from the day Verified Resale launched (Section 9.4).

The jobs behind the diagnosis came from Section 2.2, and they explain why the product has three sides and four parties. The fan who cannot go wants the money back safely and quickly, and does not want to let down the friends in the seats beside them. The fan who wants in wants a real ticket at a price that does not feel like extortion, and fears being turned away at the door more than paying too much. The partner wants a legitimate market and inventory it can trust. The organizer wants control over price and timing, real fans in the seats, and a share of whatever the resale earns. These jobs partly conflict, and Section 2.3 showed why that conflict sits at the center of the product, and why a market with three sides needs its rules designed as carefully as its screens.

Encore was not alone in reaching this diagnosis, and the memo said so: Ticketmaster’s closure of its European secondary sites in 2018, in favor of a face-value exchange, which Section 2.1 told, had followed scrutiny of the secondary market by the UK Competition and Markets Authority (CMA). A competitor’s choice proves nothing about whether Encore’s bet would pay. What it does show is that the harm in the diagnosis is real, that it has been acted on in Encore’s own market, and that regulators have noticed it, which is the kind of evidence a diagnosis should rest on.

Where to play

Section 2.4 drew the Wardley map that answered the next question: which parts of Verified Resale should Encore build, and which should it leave to others? Anchored on the need “a fan holds a guaranteed ticket”, the map put reissue and revocation at the gate in the custom middle of the chain, possible for nobody else because only Encore issues the barcodes, and organizer rules near genesis. The memo recorded the map’s output in two lines: build reissue and rule enforcement, and partner for listings. Rowmate already ran a marketplace with fans on it, and it would have been both expensive and hostile for Encore to build a rival storefront for resale when it could instead make every partner’s storefront trustworthy. That Rowmate was a competitor as well as a partner did not change the choice; it changed the terms, which Section 2.4 worked through and the Software book’s Section 8.3 turns into a partner program.

Wardley maps are a thinking tool, widely used by practitioners and thin in published outcome evidence, and the memo used the map only to make an argument visible, never as proof. The proof it would accept was the objectives that Section 2.5 drew from the strategy. The Resale trio would own one product outcome, “verified resales completed per eligible event”, rather than a revenue figure it could only influence. The memo also named the one date that could become a high-integrity commitment, the kind Section 2.5 reserves for real deadlines: the resale clause had to be in the contracts organizers signed in the autumn contract season, because that season decided which tours of the following year could carry resale at all. As Section 2.5 recounts, Encore made that commitment only at month 4, once discovery had shown what it would be promising.

Discovery before code

The memo did not fund a build. Its first tranche bought four months of discovery for the trio, a product manager, a designer and a tech lead, and within it an appetite of six weeks for two small teams, lent from Orders & Payments and Partner APIs, to run the prototypes (Section 3.4). The trio interviewed fans, organizers and Rowmate every week, in the manner of Section 3.1: 22 organizers, 31 fans and six people at Rowmate over the four months. It kept an opportunity solution tree whose single outcome was the north star described below. The design partners were the six Section 3.3 recruited, all mid-size festival and tour promoters: three in France, two in Belgium and one in the Netherlands. One of the French three, a promoter of arena tours, was openly hostile to resale. Rowmate took part in a separate, smaller partner program of its own.

Two assumptions in the memo were marked as riskiest, because the bet failed if either was false and the only evidence for either was what people had said in interviews. The first was that organizers would set rules and keep resale switched on. The second was that buyers would trust a ticket that had been reissued, believing both that the new one would scan and that the old one would not. Section 3.2 described how each was tested before any service existed. A Wizard-of-Oz pilot ran resale on four real shows, two in Lyon and two in Antwerp, about 11,200 tickets in all, with Support agents doing each reissue by hand through their old lost-ticket screen behind a real fan interface. It completed 191 resales from 268 listings, at a median of 14 Support minutes each, and no fan holding a reissued ticket was refused at the door. Four shows could not measure demand, and the pilot’s pre-registration said so in advance: demand was read qualitatively. A clickable rules screen tested with all six promoters showed that organizers could express their rules. And when the trio asked what organizers wanted from resale, control came up in 19 of the 22 organizer interviews and the fee share in eight. By control they meant when resale opens, how many tickets one fan may list or buy, and whether it happens at all. Both riskiest assumptions came back supported. What discovery could not show was how organizers would actually behave once resale was live on their own events. We will see at month 18 what that left out.

Price before product

Madhavan Ramanujam and Georg Tacke’s argument, which Section 5.1 developed, is that willingness to pay should be tested while the product is still a concept, because the answer changes what is built. Encore held those conversations inside discovery, with the six design-partner promoters and with Rowmate, and set the price before the first screen was designed. The resale price is capped at face value, and organizers can only tighten the rules, never loosen them. The buyer pays a 10% service fee, included in the all-in price shown from the first screen. The fee is split 4 points to the partner through which the buyer purchases, 3 to the organizer and 3 to Encore, whose share carries the payment service provider (PSP) fees, fraud losses and support. When the buyer purchases in Encore’s own app, Encore keeps the partner’s 4 points as well. The seller receives the resale price in full, five days after the event, once the reissued ticket is beyond the reach of a dispute. A €60 ticket resold at €60 costs the buyer €66, of which the partner receives €2.40, the organizer €1.80 and Encore €1.80.

Section 5.2 compared the models Encore considered, among them a flat fee and a fee that fell as the price neared face value, and explained why a fixed percentage won. Two of the reasons matter here. The first is that the case was honest about its weakest term. Encore’s variable cost of serving one more ticket is about €0.12 (the Resilience book, Section 2.5), but the pilot’s hand-done reissues cost several times what Encore’s 3 points could pay in Support time. So the memo recorded the business case as conditional: once reissue was automated, Support time per resale had to fall to a fraction of the pilot’s figure, or Encore would lose money on every resale. That condition became one of the memo’s guardrails. The second reason is that a split that organizers could see and predict was itself part of what Section 5.1 called communicating value: the organizer’s message was control and a share, and a share that moved with the price would have been harder to explain than one that did not.

The memo

All of this lived in one document. The diagnosis, the map and the questions for discovery and pricing were in it at month 0, and the answers went into its later revisions. Its form deserves attention, because the form is what makes it possible to reread the document later. Writing a bet down before building it is an old discipline with a well-documented public variant. At Amazon the variant is the PR/FAQ (press release and frequently asked questions): a press release of about a page written as if the product had already launched, followed by the questions a skeptic would ask, with answers. Colin Bryar and Bill Carr, in Working Backwards, date the practice to Amazon’s move into digital media in the early 2000s. Werner Vogels described the reasoning in 2006: “Writing a press release up front clarifies how the world will see the product — not just how we think about it internally.” Encore’s bet memo is an adaptation of that form, and the difference is worth stating. It is a document that states a product decision as a wager: what we believe, what we will spend, how we will know, and when we will stop. The press release opens it because writing from the customer’s side first keeps the memo honest about value. The rest exists so that the bet can be judged later against what it said.

The FAQ is where the riskiest assumptions go, and it is the part most often written badly, because it is tempting to answer the questions we can answer well. Amazon’s own history carries the warning. The Fire Phone launched in July 2014 at $199 on contract with AT&T, was cut to $0.99 on contract within weeks, and in October Amazon took a charge of $170 million, mostly for Fire Phone inventory and supplier commitments. A company that had made writing its bets down a habit for a decade still made this one. A memo guarantees nothing about the quality of the decision; it is only as good as the honesty with which its FAQ answers the questions nobody wants to ask.

Three further ideas gave the memo its teeth. The first is appetite, which Ryan Singer’s Shape Up defines as “the amount of time we want to spend on a project, as opposed to an estimate.” Encore’s appetite was 12 months from the memo to the first resold ticket scanned at a gate, and it was a statement of what the bet was worth, set before anyone had forecast what it would cost. The second is the kill criterion: a condition, written in advance, under which we stop. Annie Duke, in Quit, recommends setting kill criteria in advance, best expressed as a state and a date, because a criterion set before the work begins is set by people who are not yet invested in continuing it. The third is the distinction Jeff Bezos drew in his 2015 letter to shareholders between decisions that are hard to reverse, one-way doors, and decisions that are easy to undo, two-way doors, which the Architecture book used for data migrations. The memo marked its doors. The fee split and the partner terms, which promise partners 12 months’ notice of any breaking change, were close to one-way, and deserved the slow deliberation the memo gave them. How the fee was displayed, or the order in which listings appeared, were two-way doors, to be decided quickly and changed on evidence.

Here is the skeleton of revision 0, as the trio presented it:

VERIFIED RESALE: bet memo, revision 0 (month 0, late winter, year 9)
Press release  One page, dated launch day, written as if it had happened
Diagnosis      Uncontrolled resale harms fans and organizers (Section 2.1)
Policy         Bring resale inside Encore; enforce every rule at the barcode
FAQ            Riskiest first: organizers set rules and keep resale on;
               buyers trust a reissued ticket
Appetite       12 months to the first resold ticket scanned at a gate
North star     Verified resales completed per eligible event (scanned at the door)
Guardrails     Primary sell-through; bot rate on resale checkouts;
               organizer opt-out rate; Support minutes per resale
Kill K1        Month 4: fewer than 2 of the first 3 organizers we recruit commit
               a real event, or a voided pass still scans offline in the prototype
Kill K2        Month 18: below 10 completed resales per eligible event, or primary
               sell-through more than 2 points under its synthetic control
Doors          One-way: fee split, partner terms. Two-way: display, ordering
Tranche 1      Discovery, months 0–4: the trio; six weeks of two lent teams
Left open      Listing in Encore's app; seller payout timing; default window

Two things in that skeleton are easy to miss. The north star joined Section 2.5’s outcome to Section 4.1’s sharper version: a resale counts as completed only when the reissued ticket scans at the door, which keeps the number honest about the one thing the product promises. And the “left open” line is a strength of the memo. A memo that pretends to have settled everything invites people to believe it, and the open questions are where discovery’s next weeks should go. All three were closed by month 4.

The memo ended with a pre-mortem. Gary Klein’s method, published in the Harvard Business Review in 2007, asks a team to imagine that the project has already failed and to write down why. It works because it gives permission to say aloud what people otherwise keep to themselves. Encore’s trio collected about 20 stories, and three recurred. In the most common, a stadium night arrives on which the old and new barcodes both admit someone, and the press reports that Encore’s verified tickets are no safer than a screenshot. In the next, organizers refuse to enable resale because they see Encore taking a cut from their fans. In the third, Rowmate leaves, and the product has a pool of listings and no storefront that fans already use. Each of the three became a line in the FAQ, and the first became a test that would run on every build.

The memo works under four conditions, and Encore held to all of them. It was written before the build was funded. Its metrics and kill criteria were concrete enough to be checked by someone who had not written them. It was read and challenged in a meeting of the product, engineering and finance leads, which began with the strongest objection read aloud: that an anti-scalping company hosting resale would look, to fans and artists, like a company hosting touts. And it was reread at every checkpoint, which is the structure of this chapter. The mechanics of the tranches belong to the Software book (Section 10.4), and so do the formation of the Resale team by isolation (Section 9.2) and the trio’s charter (Section 9.3). What belongs here is the decision those mechanics serve.

Recap. The bet began as a diagnosis: the harm lay in uncontrolled resale, and only Encore could control resale at the barcode. Discovery and pricing came before design, and the memo gathered their questions into one document with an appetite, a north star, guardrails, kill criteria written as a state and a date, and its doors marked. The form owes much to Amazon’s PR/FAQ, and the Fire Phone is a reminder that the form guarantees nothing; what makes a memo useful is that it is honest about what is risky, and that it is read again.

Exercise 10.1. Write a one-page bet memo for an initiative of your own: a diagnosis in Rumelt’s three parts, an appetite, a north star and two guardrails, and two kill criteria, each a state and a date. Mark every commitment as a one-way or a two-way door. Then hold a 30-minute pre-mortem with two colleagues and add the top three failure stories to the FAQ.

10.2 Designing It

At month 4 the trio reread the memo against what discovery had found and wrote revision 1. Both conditions in K1 had been tested and passed. The first three organizers recruited, who became the three French design partners, had all committed a real event for the launch, the hostile arena-tour promoter only one, and on the tightest rules the console allowed. The voided pass in the prototype did not scan offline, and in the pilot three revoked Apple Wallet passes, forwarded by sellers before the sale, were refused at the gate. The open questions had closed. Fans could list in Encore’s own app as well as through partners, with Encore keeping the partner’s 4 points; sellers would be paid five days after the event; and the default resale window would run from purchase until two hours before doors open, which organizers could shorten. Encore decided to build, and made the contract season its high-integrity commitment. The Resale team formed from people in Orders & Payments and Partner APIs, with the trio and a staff engineer, and the first quarterly plan showed 12 teams with work to do and 23 dependencies among them. How the team was formed and the plan drawn is the Software book’s subject (Sections 9.2 and 10.2).

The experience

Verified Resale has three faces, and Section 6.2 gave each the posture it needs. The organizer’s rules console is sovereign: it lives inside a console used for hours at a time, and it can be dense. The fan’s purchase is transient, a few screens and one decision each. The wallet pass is nearly daemonic, consulted at the door and otherwise ignored. Because organizers set resale rules only a few times a year, the console opened on presets with an advanced path, as Section 6.2 prescribed for perpetual intermediates.

The monthly usability tests of Section 6.3 found the problem that most changed the design. Sellers saw “Sold” and did not believe their old ticket was void, so some of them kept the pass in their wallet “just in case”, and a few forwarded it to a friend. Section 6.1 named it a gulf of evaluation: the system knew the ticket was dead, and nothing on the screen said so. The fix was to make the ticket itself change. On sale, the seller’s ticket card turns to a cancelled state, its barcode greyed and struck through, with the line “Sold and reissued to the buyer. This ticket will not get you in.” The next round of tests found that every participant now understood the old ticket was dead.

Accessibility was built in from the first component, as Section 6.4 requires under the European Accessibility Act, and one rule went beyond the screen. Accessible-seat tickets can be resold, but only to buyers who attest that they or someone in their party need the seat’s features; Encore asks for no proof, because proof of disability is health data it has no lawful need to hold. The rule was drafted with disability organizations. The design system took two new patterns, VerifiedBadge and ReissueStatus, and extended a third, PriceBreakdown, to show the cap. Following the contribution path of Section 7.2, both new patterns began as candidates in the Resale team’s code and were promoted to the core once Mobile App and Fan Messaging needed them.

The ticket in the phone

Section 8.3 described the ticket that makes reissue meaningful. Every mobile ticket at Encore has carried a rotating barcode since the eighth year, a time-based code generated on the device from a per-ticket secret, so a screenshot stops working within moments. (The barcode is the same entry credential the Architecture book called a QR code.) Reissue gives the buyer a new ticket identity with its own secret, voids the seller’s pass, and adds the old barcode to the revocation lists that gate scanners hold for offline validation. Whether the old ticket is dead at a gate with no signal therefore depends on how fresh the scanner’s list is, and Venue Entry’s work was to make sure it was fresh before doors opened.

The release train set the schedule. Encore ships the fan app weekly, so a missed train costs a week, but the reissue-aware ticket view had to reach the phones of fans who might sell or buy, and old versions linger for years. Section 8.2 described the decision. The soft floor on app versions went up three months before launch, so that by launch day nearly every active phone could show a reissued ticket correctly, and an app below the reissue-aware version cannot list or buy at all. The hard floor did not move, because no fan who merely holds a ticket needed to be stranded for resale to be safe. Resale became the fan app’s 15th feature module, owned by the Resale team. How revocation works in the code, and how it was made safe, is the Software book’s Section 11.1.

The partner API

The partner API was designed from the outside in, as Section 9.1 taught, starting from Rowmate’s job story: when a fan lists on our site, we want to know at once whether the organizer allows it, so that the listing never fails later. That story produced a validateOnly request that answers “would this listing be accepted?” before anything is created. Listing and purchase are idempotent. Withdrawal is a custom method, POST /v2/listings/{id}:withdraw, and reissue is a long-running operation with its own resource, because voiding a pass, issuing a barcode and delivering a wallet pass take longer than a request should wait. Partners learn that a listing sold through a thin, signed listing.sold webhook and fetch the details, because rules and prices may have changed since the listing was made, and they mirror the pool through an inventory feed that Rowmate reconciles hourly and on every rule change (Section 9.5). Rowmate built against the contract’s mock server for three weeks before the service existed.

One engineering question had to be settled before the product could be built: which team owns reissue? The Software book tells the argument (Sections 9.5 and 11.1); its outcome was that the Inventory team, which owns the barcode core, owns issuance and revocation, and Resale calls it. For the product the verdict mattered for one reason. The question “is this barcode valid?” would have exactly one owner, the same team that answers it for every primary ticket, and the fan’s guarantee would rest on it.

The price in the system

A price lives in a system long after the meeting that set it, and Chapter 5 showed where. Organizer resale rules are typed configuration owned by the Resale service, and the controls available to each kind of organizer are entitlements held as versioned data (Section 5.3). The Software book’s Section 7.3 builds that service. Every completed resale produces one ResaleCompleted event, emitted only after the reissue has succeeded, metered once, rated by the price book in force when it happened, and split into partner, organizer and Encore shares that the finance ledger reconciles daily (Section 5.4). The cap at face value is a product rule and also a compliance capability. Ireland’s Sale of Tickets Act 2021 already forbids resale above face value for designated events, and rules of this kind arrive country by country, as Section 5.5 described.

Verified Resale’s one artificial intelligence (AI) feature belongs here, because it sits on top of the rules. An organizer writes a request in plain language, such as “no resale until a week after the on-sale, two per fan”, and the organizer rule assistant drafts the typed rule set and a plain summary, which the organizer confirms before anything applies. The assistant can express only rules the type system allows, so the rule that organizers can only tighten holds whatever the model says. Section 6.5 designed the assistant, and the Software book’s Section 3.5 describes its evals.

The marketplace rules

The last design artifact was the rulebook for the market itself. Partners are vetted before they receive sandbox credentials, certified in the sandbox before they receive production credentials, and bound by terms that say what they may and may not do; the Software book (Sections 8.2 and 8.3) treats those programs as a business. Some rules are as old as Partner Listings: partners have never been able to buy primary tickets, since the original key scopes. Others are new. Every resale buyer passes Bot Screening and the per-buyer limits, and Bot Screening’s existing commitment, no more than one false block per 1,000 admitted sessions, covers the resale checkout as it covers the primary one. Partners read the same coarsened availability everyone gets and never see sales velocity, which Encore keeps confidential because scalpers would pay for it. And only fans resell: Encore admits no professional resellers, which also keeps it clear of the trader-traceability duties that the European Union’s (EU) Digital Services Act places on marketplaces that host traders. Whether the market would reach enough listings and buyers to be worth visiting, the question of liquidity and ignition, is the Software book’s Section 8.2.

Recap. At month 4 discovery had closed the memo’s open questions and the decision to build was made. The design then spread across artifacts with different owners: flows for three postures, a wallet pass that visibly dies, a partner API built from partners’ jobs, rules and fees as data, and a rulebook for the market. Each artifact traces back to a line in the memo, which is what kept 12 teams’ work one product.

Exercise 10.2. For one feature you are building, list every artifact that defines it: flows, components, API operations, entitlements, configuration and policies. Name the owner of each, and mark the ones whose owner is not on the team that proposed the feature. Those are where the product’s intent is most likely to be lost.

10.3 Building It

By month 9, in the autumn of the ninth year, the memo’s second revision recorded a different kind of progress. The product side needed only the state of each promise, and the revision recorded it briefly. A walking skeleton ran from the partner API through Resale and Seat Inventory to a test wallet. The barcode core, written in Encore’s first year and understood by one remaining author, was under tests that made it safe to change. The partner API was in the sandbox and Rowmate was certifying against it. Twenty-three dependencies had become 14. And in the autumn contract season, organizers signed contracts with the resale clause in them, which honored the memo’s high-integrity commitment a season before any fan could resell a ticket.

What the product asked of engineering

The trio’s job during delivery was to keep the product’s intent intact while 12 teams built it, and the most effective tool they had was to write that intent in a form engineering could test. The memo’s promises became four commitments: reissue under a new barcode, with the old one dead at every gate; organizer rules as typed configuration that can only tighten; a partner API with a sandbox and certification; and five fairness invariants, which Encore states in exactly these words:

1. Partners never buy primary tickets.
2. Only tickets bought through the Gate, within the per-fan limit, can be listed.
3. Organizer caps and rules can only tighten.
4. Every resale buyer passes Bot Screening and the per-buyer limits.
5. Partners never see sales-velocity data.

Each invariant is a sentence a fan or an organizer could understand, and each runs as a test on every build (the Software book, Section 11.3, tells how). That pairing is the point. A product promise that exists only in a memo can be eroded one reasonable change at a time, with no single change appearing to break it. An invariant under test fails the build instead, and someone has to decide in the open what to do about it. The first of the pre-mortem’s failure stories, an old barcode and a new one both admitting someone on a stadium night, is now a test that runs thousands of times before any fan could meet it.

Scope held, scope cut

At month 4 the first forecast, a Monte Carlo simulation of the kind the Software book’s Section 10.3 teaches, put the launch about two months beyond the appetite. The trio did what Shape Up recommends and treated the appetite as fixed and the scope as the variable. Singer calls the principle “fixed time, variable scope”, and his circuit breaker states its consequence bluntly: “If they don’t finish, by default the project doesn’t get an extension.” Encore did not adopt the whole method. It does not run six-week cycles across 40 teams, and Section 3.4 explained why it keeps its quarterly planning. It borrowed the two ideas that mattered, and they work under one condition, which Encore met: the people who set the appetite had the authority to cut scope, and the work had been shaped before it was forecast.

The cuts were chosen by one test: after the cut, could the bet still be judged? Tessera moved from launch to the months after it, because a second partner adds reach but changes nothing the north star or the guardrails needed to measure. Festival passes and other multi-day tickets were excluded from the first release, since they would have doubled the rules engine’s cases for a small share of resales. The organizer’s resale dashboard became a monthly statement, which told organizers what they had earned without the build of a new reporting surface. Rule templates that an organizer could apply across a whole tour were cut as well, on the forecast’s evidence (the Software book, Section 10.3), so organizers would set rules event by event, with the console’s bulk edits to help. What the trio held was everything the memo’s kill criteria depended on: reissue, the invariants, organizer rules, and the measurement of completed resales at the door. A cut that removed any of those would have saved time only by leaving the bet with no way to be judged.

Discovery that continued during delivery

The weekly interviews did not stop when the build began, because, as Teresa Torres insists, discovery is a habit that continues through delivery. Two findings changed the build. The first came from fans: people who had bought a pair of seats wanted to sell them together, and buyers wanted to buy them together, because nobody wants to sit beside a stranger at a concert they planned with a friend. Listings of a pair became one listing by default, which the seller could split, and the cut of festival passes paid for the work. The second came from the organizers’ box offices, which handle the fan whose phone has died at the door. They needed to see that a ticket had been resold, and to whom it now belonged, before they could help, so the box office’s ticket lookup learned to show a reissued ticket’s history. Neither finding was in the memo, and both would have reached Support as complaints in the first month had the trio stopped listening.

The engineering story is another book

Everything else that happened between month 4 and month 12 is engineering, and the Software book’s Chapter 11 tells it, from the first test written against the barcode core to the flags removed after launch. Two of its outcomes touched the product directly. Knowledge of the barcode core spread from one engineer to four, so the fan’s guarantee no longer rested on one person. And the change to an opaque ticket identity, announced to partners at month 2, kept the promise of 12 months’ notice: the old barcode format stays accepted on the partner API until month 14. What the product side needed from the rest was smaller: whether each commitment above was true, and on what date.

Recap. Building began with the product’s promises turned into tests, so that intent survived 12 teams and many reasonable changes. When the forecast outran the appetite, the trio cut scope by asking whether the bet could still be judged after each cut, and kept listening to customers, which changed the build twice. The engineering itself belongs to the Software book; the product’s part was to know which promises were true.

Exercise 10.3. For one initiative of yours, list the product commitments engineering must make true, and write each as a sentence a customer would understand. Mark which ones a test already protects. Then name the one you would cut first if the date held and the work grew, and say whether the bet could still be judged without it.

10.4 Launching It

Revision 3 of the memo, at month 12, was mostly a launch plan, and the launch is where Gmail’s lesson applies most directly. Verified Resale opened in the winter of the tenth year to three organizers, one partner and one market.

The launch plan

A launch staged by audience widens access by who may enter: a cohort of customers, then a market, then the next. Gmail did it with invitations, and Facebook did it in 2004 with college email addresses, opening at Harvard in February and at Columbia, Stanford and Yale in March, so that each new cohort arrived already dense with people who knew one another. It is easy to confuse this with progressive delivery, and the two should be kept apart. Progressive delivery releases code to a growing share of traffic so that a fault reaches few users, and it belongs to the Resilience book (Section 4.3); every Verified Resale change went through it. A launch staged by audience decides which customers the product serves, and in what order, and that is a product decision.

Encore’s first cohort was the three design-partner organizers, chosen because they had shaped the product and would forgive its first mistakes. Each cohort had an entry criterion and an exit criterion written into the memo before launch, so that widening and retreating were both decisions made in advance. This is the product counterpart of the launch checklists in Google’s Site Reliability Engineering, and Figure 10.3 shows Encore’s version. The thresholds are Encore’s own judgments, set by the trio from the pilot and the design partners’ volumes; they come from no published standard.

Figure 10.3 — Verified Resale’s launch cohorts, each with conditions to proceed and to retreat. Belgium joined at month 16 on cohort 2’s criteria; the thresholds are Encore’s.

The plan also said who would be told before the first fan could resell. Gmail met its privacy objection five days after launch; Encore preferred to meet its own before launch. It briefed national consumer bodies and the organizers’ associations in France on the product, the cap and the reissue guarantee, because the question “we fight scalpers; now we host resale?” would be asked in public, and it was better to answer it to people who had read the rules. And the plan avoided one of Gmail’s costs deliberately. The product was never labelled a pilot or a beta to organizers. It was “Verified Resale in France”, because organizers sign contracts a season ahead, and a cautious organizer, like Google’s business customers, will not build next year’s tour on something that sounds unfinished.

The experiment that could not be clean

Section 4.5 promised that Encore would judge Verified Resale without a clean A/B test, and explained why it could not have one. Fans, sellers and organizers share one pool of seats, so a treatment that helps one fan can take a resold seat from another, and the comparison between arms measures transfers more than effects. Airbnb’s fee meta-experiment, which Section 4.5 tells, put a size on that bias: at least 20% of the effect an individual-level test estimated for a fee increase was interference. What remained for the launch was to put Section 4.5’s plan into use, and to keep to it once results began to arrive.

The plan used three designs, each answering a different question. For the question the memo cared about most, whether resale eats the primary sale, Encore used a synthetic control: an estimate of what a treated market would have done without the treatment, built as a weighted combination of untreated markets that tracked it closely beforehand. France had been chosen first for business reasons, its design partners and its contract season, so no randomized comparison of markets was available. Synthetic France was built from nine EU markets without resale, using 104 weeks of primary sell-through per eligible event. Alberto Abadie’s conditions for trusting the method, a long pre-period, a pool of comparable donors, no spillover into them and no shock that strikes one market alone, were argued in the plan one by one, and Belgium was removed from the donors before the analysis began, because Belgian fans cross the border for French dates. When Belgium itself went live at month 16, the plan already said which estimator would handle a rollout that reaches countries in different months, because the ordinary regression gives biased answers there (Section 4.5).

For the narrower question of how to show the fee, Encore randomized by event. When the launch widened to all consenting French organizers, 38 of them let about 520 of their events be assigned at random to one of two presentations of the same all-in price: a single figure, or the figure with the split among partner, organizer and Encore itemized beneath it. Every fan of a given event saw one presentation, and every fan paid the same 10%, because Encore never varies the price a given fan pays (Section 5.5). Section 4.5 records why no switchback was used. The third design was a holdout: nine French organizers who agreed to go without resale for six months, were told what the holdout was for, and were free to leave it.

All of it was pre-registered before any fan had resold a ticket: the metrics, the donors, the checks and the decision rule. The overall evaluation criterion (OEC) of Section 4.1 was the memo’s north star.

PRE-REGISTRATION VR-FR-01 (registered month 11)
Question     Does Verified Resale change primary sell-through in France?
Design       Synthetic control; unit: market-week; treated: France
Donors       Nine markets without resale; Belgium excluded (cross-border fans)
Pre-period   104 weeks of primary sell-through per eligible event
OEC          Completed verified resales per eligible event
Guardrails   Primary sell-through: no worse than −2.0 pts vs synthetic France
             Bot rate on resale checkouts: no higher than on primary
             Reissued tickets refused at gates: fewer than 1 in 10,000
Fit checks   Pre-period prediction error reported; placebo run on each donor
Decision     Widen beyond France only if all guardrails hold; review month 18
Not claimed  Displacement of grey-market resale: no data Encore controls

The last line matters as much as the rest. The diagnosis was about the grey market, and the grey market is the one thing Encore cannot observe, because it happens on sites and in messages Encore does not see. The closest measurement Encore owns is at its own doors: duplicate-ticket rejections at the gate, which on Partner Listings were mostly copied passes. The memo treated a fall in those rejections as supporting evidence, and any claim about displacement as qualitative. The placebo runs are the method’s own honesty check: if a donor market, treated as if it had launched, showed an effect as large as France’s, France’s effect would mean little.

The first partners

Rowmate was the first partner, and the irony was not lost on anyone. It was Rowmate’s 26-month-old key, committed to a public repository by a contractor, that Encore had revoked in 38 minutes in the eighth year, after it had been used to post two fraudulent listings. Encore kept Rowmate and changed the structure around it. Partners received sandbox credentials at once and production credentials only after certification, the checklist of Section 9.4: handling rule changes, verifying webhook signatures, retrying safely. In production they authenticate with 15-minute tokens bound to a client certificate, the arrangement the Resilience book (Section 7.2) put in place after the leak, so a key committed to a repository today would expire before anyone could use it. Rowmate passed certification at month 11.

Tessera followed at about month 15, after its own certification, and was live in 11 days, a speed the Software book’s Section 8.3 credits to a partner program built with the second partner in mind from the start. Both partners hold terms that promise 12 months’ notice of any breaking change to the API, the lesson Section 9.4 took from Twitter’s treatment of its developers in 2012. At the month-18 review there were two partners, and no third had been signed.

The first resold ticket

The first resold ticket belonged to a fan in Lyon who could not go to a sold-out Saturday concert there, promoted by one of the three French design partners. Its face value was €80. Figure 10.4 traces it through the product, step by step, with the team that owns each step and the section of this book that taught it. The Software book’s Section 11.4 traces the same ticket through the code.

Figure 10.4 — The first resold ticket, through the product. Ten steps, nine teams, and the north star counts it only at step 9.

Read the table’s columns separately and two things appear. The “where taught” column covers nearly the whole of this book, which is the reason for tracing one ticket at all: a reader who has followed the chapters one at a time can see here that they describe one transaction. The “owner” column names nine teams for a single sale, and not one of them could have delivered the fan’s guarantee alone. That is the plain meaning of the claim, made in Chapter 1, that a product at 400 engineers belongs to the whole organization, whatever team’s name is on it. Notice too where the north star counts. Steps 1 to 8 are activity, and only step 9 is the outcome the memo promised.

Who decides on the day

The decision to open each cohort was a go or no-go call made by the Resale trio and its staff engineer, against the entry criteria in Figure 10.3 and nothing else. A decision against written criteria is long-standing practice; Google’s launch coordination engineers, whose work the Site Reliability Engineering book describes, worked from checklists through as many as 70 launches a week. The condition that makes it work is the one Encore wrote down: the criteria come from the memo and the readiness evidence, and they are fixed weeks before launch morning, the moment when everyone is invested and nobody wants to be the one who says wait.

This meeting is easy to confuse with the readiness review the Resilience book describes, and they answer different questions. The readiness review, held on the Wednesday before a big on-sale, asks whether the system can carry a known peak. The launch decision asks whether a product should reach a new group of customers. Each of the four people in the decision could stop the launch alone, and so could the incident commander rotation if an incident was running. Running the day itself, from the first listing to the first scan, belongs to the Operations book (Chapters 5 and 7).

Recap. The launch was staged by who may enter, with each cohort’s conditions to proceed and to retreat written before launch, and the public objection answered before the first fan could resell. Because a marketplace defeats a clean A/B test, Encore put Chapter 4’s plan into use: a synthetic control, an event-level test of the fee’s display and a consenting holdout, pre-registered and honest about what none of them could show. One ticket traced end to end showed that nine teams were needed to keep a single promise.

Exercise 10.4. Trace one transaction of your product end to end. For each step, name the team that owns it, the test that protects it, and the point at which your north star counts it. If the north star counts the transaction before the customer has received what was promised, rewrite the metric.

10.5 What Encore Learned

In the summer of the tenth year, about six months after launch, the trio, the staff engineer and the leads who had funded the bet met for the review the memo had promised at month 0. Verified Resale was live in France and Belgium, with Rowmate and Tessera. No kill criterion had triggered.

The review of the bet

The review followed the shape of the after-action review that the Operations book (Section 6.4) borrowed from the US Army: what was supposed to happen, what happened, why the two differed, and what to keep or change. Its value depends on one condition above all, that the prediction was written down before the outcome was known. Baruch Fischhoff showed in 1975 that people who know an outcome judge it to have been more predictable than it was, a bias the Operations book (Section 6.2) meets in incident reviews. Without the memo, the review would have been a story told by people who already knew the ending. Figure 10.5 sets the memo’s revisions side by side.

Figure 10.5 — The memo across five revisions. The cell that matters most is the riskiest assumption at month 18: it held, for a reason discovery had not seen.

The numbers came first. The north star, measured across all eligible events in France, stood at an average of 34 completed resales per event against the memo’s month-18 target of 30 (a different measure from the Resale team’s first live-quarter key result of 40, which counted only the pilot organizers’ events; Section 2.5), but the median was 11, because a handful of arena nights produced most of the resales; the trio recorded both, since an average alone would have flattered the product’s reach. The synthetic control estimated the change in primary sell-through on resale-enabled events at a fall of 0.4 points, with an interval running from a fall of 1.3 points to a rise of 0.5. The guardrail held, and the review said plainly that the method could not rule out a small harm. The fee-display test found no detectable difference in completed purchases, and fewer support contacts asking where the money went when the split was shown, so the split stayed. Bot rates on resale checkouts stayed below those on primary checkouts. Scans refused at French doors because a barcode had already been used fell sharply on resale-enabled events, which supported the diagnosis without proving that the grey market had shrunk. Figure 10.6 sets the two headline numbers beside the lines the memo had drawn for them.

Figure 10.6 — The review’s two numbers. The average cleared its target while the median sat just above the kill line; the sell-through estimate stayed well clear of its guardrail, but its interval still reaches below zero.

Then the harder part. Annie Duke, in Thinking in Bets, calls the habit of judging a decision by its outcome “resulting”, and a review that only asks whether the kill criteria fired falls into it. Encore’s review judged the decisions with the information available when each was made, and recorded the outcomes separately. It found one thing the team had got wrong, one that had gone well by luck, and one that had gone badly despite a good decision.

What the team got wrong was organizers, and the error was subtler than a failed assumption. The memo’s riskiest assumption had held: organizers set rules and kept resale on. Discovery had also been right about what they meant by control, which was timing and limits far more than price. Where the trio went wrong was in how organizers would divide. Section 5.1 had found two groups, control-seeking promoters who wanted resale closed until well after the on-sale and revenue-seeking ones who wanted it as wide as possible, and the console’s three presets served both: “Fan-first” for the first group, “Standard”, open from purchase, for the second and shown first, and “Resale off”, which Irish organizers are never offered. The trio expected the two groups to be of comparable size. In practice the revenue-seeking group barely appeared in behavior. Most organizers who enabled resale kept it closed for the first days after the on-sale, set per-fan limits below the event’s own, and almost never lowered the cap. Several who had sounded revenue-seeking in interviews did the same. The follow-up interviews found the reason, and it was not money. Organizers feared resale that was visible in the first hours after an on-sale, because to their fans and their artists that looks like scalping whatever the price. The hostile arena-tour promoter had become the heaviest user of the rules, setting exactly that one on every date he put on resale: resale a week after the on-sale, one ticket per fan, close to the very request the rule assistant had been designed around. The interviews had recorded what organizers wanted to control. They had not shown how widely the fear behind it was shared, because people describe their interests more readily than their fears.

What went well by luck was the calendar. No stadium-sized on-sale fell in France during the first two cohorts, so the resale page never met the traffic that follows a large sell-out before the product widened. The guardrails held partly because they were never stressed, and the review wrote that down, because a guardrail that was never stressed has not yet been tested (Section 7.4 describes the review that peak pages need). What went badly despite a good decision was the holdout. Encore was right to require consent and to let organizers leave, and two of the nine did leave early, after their own fans asked why they could not resell. The holdout that remained was too small to say much about lasting effects on primary sales, and the next one will need more organizers to begin with.

The product at month 18

By month 18, 61% of eligible French organizers had enabled resale, and Belgium, which joined at month 16, was following the same curve more slowly. Tessera brought fans whom Rowmate did not reach, and the pool showed the effect Section 2.3 predicted: buyers came where the listings were, and listings came where the buyers were. Discovery continued at its weekly rhythm, and the finding about organizers had already changed the product. The console now shows the “Fan-first” preset first, the pitch to organizers leads with when resale opens instead of with the cap, and the lower cap stays on the advanced path for the few who want it. The platform’s own default window, from purchase until two hours before doors, did not change, because the review wanted a season of evidence before changing what every organizer gets without asking.

Deciding again

The review ended where the memo had begun, with a decision, and it had three options in front of it: widen, change the fee, or stop. Stopping was not supported by anything in the evidence, and the memo’s own criteria said so. Changing the fee was a serious proposal. A buyer fee of 8% would have made the product easier to sell to fans, and several organizers asked for it. The review declined it for now, for reasons the memo had prepared. The split was a one-way door, partners held terms with 12 months’ notice, and the fee test had varied only how the fee was shown, by design, so Encore had no evidence about price sensitivity and would not gather it by charging fans different prices. If the fee ever changes, it will change for everyone on an announced date, with existing organizer contracts protected by grandfathering and a sunset date, as Section 5.5 describes. Widening was the decision. The second part of the third tranche, held back until this review, funds further markets on cohort 2’s criteria, with festival passes added to the rules engine, and with a larger holdout designed from the start.

The system and the organization

The engineering has its own review, the Software book’s Section 11.5, held at month 24, a year after launch, and the two reviews point at each other without repeating each other’s lessons. What it found about the codebase, the platform and the barcode core belongs there. One of its outcomes matters to the product side: partner credentials and shared webhook delivery now belong to the platform group, so the next partner program will not have to build them again.

The organization is this review’s business, because it decides who carries the bet forward. The trio stayed together through all five revisions of the memo, and the review judged that continuity to be one of the decisions that had gone well: the people who wrote the riskiest assumption were the people who read the evidence against it. The Resale team, grown well beyond a trio and a staff engineer, will split after the review, along the line between the fan and partner flows and the organizer’s rules, so that the organizer side, where the review’s main finding lies, has a trio of its own (the Software book, Section 9.2, covers how teams are split). The next tranche was released on the evidence in Figure 10.5 and on flow data from delivery (the Software book, Sections 10.4 and 10.5). And one change reached beyond Verified Resale. Encore’s quarterly planning now asks that every bet above a set size arrive with a memo that has kill criteria and a review date, and the review is held whether or not the bet succeeds.

What did not change

Section 1.1 opened this book with Fred Brooks’s sentence from “No Silver Bullet” (1986, published in IEEE Computer in 1987): “The hardest single part of building a software system is deciding precisely what to build.” Eighteen months of Verified Resale confirm it from an unexpected direction. Almost everything that cost Encore effort in this chapter was hard in the way the Architecture book called accidental: dependencies, release trains, credentials, a barcode core that one person understood. The platform, the golden path and, in places, AI assistance made much of that cheaper, and the Software book’s Chapter 11 measures how much. None of them touched the question the review found the team had answered imperfectly, which was what organizers feared when they asked for control. That question was essential in Brooks’s sense. It came from the problem, from what organizers feared and how fans behave, and no tool could have removed it, because no tool knew it was there. Only weekly conversations, a written bet and an honest review found it, and only once organizers’ choices on their own events could be counted.

So the book ends where it began, and I do not think that is a failure of progress. Construction will go on getting cheaper, and as it does, the share of our effort spent deciding what to build will grow, and so will the cost of deciding it badly, because a cheap wrong product still has to be owned, supported and eventually retired. At 400 engineers, as at nine, the essential difficulty lies in choosing what to build, writing the choice down, and having the discipline to read it again. The Software book takes up the building, the Resilience book what it takes to keep the result standing under load, and the Operations book what it takes to run it on the morning that matters.

Recap. The review read the memo against the outcome, judged the decisions separately from their results, and found that the riskiest assumption had held for a reason the team had not understood. Luck and good judgment were recorded apart, and the review ended in a new decision made by the memo’s own criteria. What the platform and the tools removed was accidental difficulty; the essential one, knowing what customers mean, remained.

Exercise 10.5. Take a bet your organization made at least six months ago and find what was written before it was made. Judge the decision and the outcome separately: list one result that came from a good decision, one that came from luck, and one that went badly despite a sound decision. If nothing was written before the bet, record that as the first finding, and write the memo for your next bet before anyone funds it.

Kata: The Second Night

Your brief: “Stayover.” Stayover is a hotel-booking software-as-a-service company with 300 engineers in 30 teams. About 6,000 independent hotels across eight EU countries run their reservations on it, and guests book either on the hotels’ own sites or through 40 partner travel agencies that call Stayover’s API. A third of all bookings are non-refundable, and each year guests who cannot travel write off about €90 million in rooms they paid for; many try to pass the booking to someone else by email, which hotels refuse because the name does not match the identity check at the desk. The board wants “verified transfer”: a guest who cannot travel passes a non-refundable booking to another guest through a partner agency, and the hotel reissues it under the new guest’s name. Hotels are split. Some want a share of a transfer fee, some want to forbid transfers on peak nights, and the largest chain says transfers will become a channel for touts who book cheap rooms in advance and resell them before trade fairs. The reservation core was written in the company’s first year and has two engineers who understand it. Two partner agencies compete with Stayover’s own booking site. The busiest week of the year, a trade fair in one city, is seven months away.

Deliverables:

1. The bet memo — One page: a press release written as if the product had launched, a diagnosis in Rumelt’s three parts, the FAQ with the riskiest assumptions first, an appetite, a north star that counts only completed stays, three guardrails, two kill criteria each written as a state and a date, the one-way and two-way doors marked, and a “left open” list.
2. The discovery plan — Six weeks with six design-partner hotels from one segment, one of them owned by the hostile chain, and one partner agency in a program of its own: the weekly interview rhythm, the opportunity solution tree’s outcome, and the cheapest test of each riskiest assumption, including one Wizard-of-Oz transfer.
3. The price and the terms — The transfer fee, its split among agency, hotel and Stayover, any cap, the rules hotels may tighten, and the partner terms with their notice period; show one worked example in euros.
4. The partner API sketch — Job stories for the agencies, the operations they imply (including a validation-only check and a long-running reissue), and the webhook that tells an agency a transfer completed.
5. The launch plan and its evaluation — Cohorts by who may enter, each with entry and exit criteria; who is briefed before the first transfer; and an evaluation that does not rely on a guest-level A/B test, pre-registered, with what it cannot show stated plainly.
6. The review template — The questions the month-18 review will ask, a table that sets the memo’s revisions side by side, and the place where decision and outcome are judged separately.

Rubric:

Where you now stand

You can now follow a product decision through its whole life: diagnose a problem, write the bet down with an appetite, metrics and kill criteria, test the riskiest assumptions and set the price before design, turn the bet’s promises into commitments engineering can test, launch it to one audience at a time with the conditions written in advance, judge it without a clean experiment when the market will not allow one, and review it honestly against what was written before anyone knew the outcome. That is the product half of the work. The other half is the engineering system that turns such a decision into running software across 40 teams, and its capstone, the Software book’s Chapter 11, follows the same 18 months and the same resold ticket from the first test written against the oldest code in Encore to the scan at the door in Lyon.

References

• Harry McCracken — How Gmail Happened: The Inside Story of Its Launch 10 Years Ago. TIME, 2014 — free online.
• Colin Bryar, Bill Carr — Working Backwards: Insights, Stories, and Secrets from Inside Amazon. St. Martin’s Press, 2021.
• Werner Vogels — Working Backwards. All Things Distributed, 2006 — free online.
• David Holtz, Felipe Lobel, Ruben Lobel, Inessa Liskovich, Sinan Aral — Reducing Interference Bias in Online Marketplace Experiments Using Cluster Randomization: Evidence from a Pricing Meta-Experiment on Airbnb. Management Science 71(1), 2025.
• Alberto Abadie — Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects. Journal of Economic Literature 59(2), 2021.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering, ch. 27, “Reliable Product Launches at Scale”. O’Reilly, 2016 — free online.
• Gary Klein — Performing a Project Premortem. Harvard Business Review 85(9), 2007.
• Annie Duke — Quit: The Power of Knowing When to Walk Away. Portfolio, 2022.
• Ryan Singer — Shape Up: Stop Running in Circles and Ship Work that Matters. Basecamp, 2019 — free online.
• Baruch Fischhoff — Hindsight ≠ Foresight: The Effect of Outcome Knowledge on Judgment Under Uncertainty. Journal of Experimental Psychology: Human Perception and Performance 1(3), 1975.
• Annie Duke — Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts. Portfolio, 2018.
• Teresa Torres — Continuous Discovery Habits. Product Talk, 2021.
• Frederick P. Brooks Jr. — No Silver Bullet: Essence and Accidents of Software Engineering. IEEE Computer 20(4), 1987 (first presented at IFIP, 1986).