Building Products in the AI & Cloud Era

← the whole book on one page

Chapter 4: Measuring Products — Analytics and Experimentation

“Every experiment may be said to exist only in order to give the facts a chance of disproving the null hypothesis.” — R. A. Fisher

How do we know that something we built made a difference? In a small company the question often answers itself: the founders ship a change on Monday, watch the numbers on Tuesday, and believe what they see. At 400 engineers it grows harder in every direction. Forty teams ship changes in the same week, the season and the tour calendar move the numbers, and a team that wants its change to have worked will find a reading of the dashboard in which it did. We need a way of separating the effect of one change from everything else that is happening, and a discipline for choosing, before we look, which number will count as the effect. This chapter is about both: the metrics we choose, the instruments that record them, and the experiments that let us read them honestly.

The method we rely on was worked out in a field of wheat. In 1919 Ronald Fisher, then a schoolteacher, was hired by John Russell, the director of Rothamsted Experimental Station in Hertfordshire, to make sense of the decades of records from the station’s long-running field trials. Agricultural experimenters of the day wanted to know whether a fertilizer raised the yield, but the soil varied from plot to plot in ways no one could list: a damp corner here, a richer strip there. However carefully an experimenter arranged the plots, the arrangement might line up with some unseen variation in fertility, and the fertilizer would then be credited or blamed for what the soil had done. Fisher’s answer, set out from Rothamsted in “The Arrangement of Field Experiments” (1926), was to assign plots to treatments “deliberately at random”. He refined the idea into the randomized block: divide the field into blocks of similar ground, and within each block assign the treatments “independently, and wholly at random”. He knew his readers would resist. “Most experimenters on carrying out a random assignment of plots”, he wrote, “will be shocked to find how far from equally the plots distribute themselves.” Chance distributes the plots unevenly, but its unevenness is the kind probability can measure, whereas the experimenter’s own arrangement errs in ways nobody can measure at all. Figure 4.1 sets the two arrangements side by side.

Figure 4.1 — A field arranged by hand and the same field in randomized blocks, drawn as a schematic. The hand arrangement lines the fertilizer up with the richer ground; the blocks hold the ground alike, and chance decides the rest.

Fisher left Rothamsted in 1933 to become Galton Professor at University College London, and there he wrote The Design of Experiments (1935). Its second chapter opens with a lady who “declares that by tasting a cup of tea made with milk she can discriminate whether the milk or the tea infusion was first added to the cup”. As Fisher’s daughter and biographer Joan Fisher Box told the story, the lady was Muriel Bristol, an algal biologist at Rothamsted, and popular accounts add that she named every cup correctly. Fisher’s interest was in how to test her claim: eight cups, four of each kind, in random order, so that guessing all eight by luck has a probability of 1 in 70. The book then states the two ideas on which every online experiment still rests. The null hypothesis, the claim that the treatment does nothing, “is never proved or established, but is possibly disproved, in the course of experimentation”. And “the simple precaution of randomisation will suffice to guarantee the validity of the test of significance”. Fisher did not invent randomization; Charles Sanders Peirce and Joseph Jastrow had randomized the order of stimuli in a psychology experiment in 1884. His contribution was the argument that randomization is what makes a test valid, together with designs that made it practical. He was also a committed eugenicist all his life, which is why his Cambridge college removed a window commemorating him in 2020.

Encore needs Fisher now. Verified Resale, the bet this book follows, has a memo with a north star and guardrails in it, and the memo will be judged by them. Somebody has to decide exactly what those metrics mean, record the tracking events that compute them, and test the smaller changes along the way, such as whether showing verified listings on a sold-out event page keeps fans on it. Encore already runs experiments, and a team, Experimentation, owns the platform for them. But the bet as a whole cannot be tested the way a button color can, because fans, seats and organizers are shared between any test group and any control group: a seat resold to one fan is a seat another fan cannot buy. Section 4.1 chooses the metrics, Section 4.2 treats the events that feed them as contracts, Section 4.3 runs controlled experiments, Section 4.4 builds the platform that lets 40 teams run them, and Section 4.5 asks what to do when a clean experiment is impossible, a question Fisher’s fields never had to answer.

4.1 Choosing Metrics

What makes a metric useful

A metric earns its place by changing what someone does. Alistair Croll and Benjamin Yoskovitz, in Lean Analytics (2013), call this their most important rule: a number that will not change how we behave is a bad number, however interesting it looks on a screen. They give four properties of a good metric. It is comparative, so that we can set it against last month, another group or a target. It is understandable, so that people remember and discuss it. It is a ratio or a rate, because ratios survive changes in scale. And it changes behavior, the property the other three serve. Their sharpest distinction is between vanity metrics and actionable ones. Total registered fans is a vanity metric: it only ever rises, and no decision depends on it. The share of this month’s buyers who bought again within 90 days is actionable, because it can fall, and when it falls someone has to find out why.

Lean Analytics also separates leading metrics, which predict what will happen, from lagging ones, which confirm what already has. Churn is lagging: by the time a customer has left, the number can only explain the departure. A fall in weekly logins among one cohort of organizers is leading, because it gives the account team time to act. We need both, and we need to know which is which, since a leading metric that is never checked against the lagging one it claims to predict is only a hope. Chapter 3 met a close relative of these ideas in Eric Ries’s innovation accounting (Section 3.2); this section takes the general criteria further, because a company of 40 teams needs them to hold across every product.

Criteria tell us whether a metric is good; they do not tell us where to find one. The most tested bridge is the one Kerry Rodden, Hilary Hutchinson and Xin Fu described from Google in 2010, Goals–Signals–Metrics. We begin with a goal in plain words (“fans who cannot go can pass their ticket on safely”), ask what behavior would signal that the goal is being met (“listings are created, and they sell before the show”), and only then choose a metric that counts the signal. The same paper offered the HEART categories (happiness, engagement, adoption, retention and task success) as a checklist of goals. The order of the steps matters more than the categories, because a team that starts from the metrics its tools happen to offer ends up optimizing what is easy to count.

What a team should count also depends on its business model and its stage, and Lean Analytics devotes most of its length to that point. A two-sided marketplace watches its liquidity, the chance that a listing finds a buyer and a buyer finds a listing; a software-as-a-service business watches churn and expansion revenue. The book’s benchmark figures come from startups of 2012 and 2013 and should not be quoted as current, and its stages assume a young company. Encore is not young, and it runs several models at once: a retailer of primary tickets, a SaaS vendor to venues, and now a resale marketplace, whose liquidity belongs to Section 2.3 and to the Software book’s Sections 8.2 and 8.5. What every product shares is the weekly reading of cohorts, groups of customers who started in the same period, followed over time. A cohort table shows whether fans who first used resale in March behave differently from those who started in May, which an overall average cannot show, because it mixes old customers with new ones.

North star and its inputs

A large organization needs one number that says what value the product delivers. A north star metric is a single measure of the value customers receive from a product, chosen so that growth in it means the business is growing for the right reason. The practice has a widely used playbook, John Cutler and Jason Scherschligt’s North Star Playbook from Amplitude, but no controlled evidence behind it, so we should treat it as a communication device: a way of telling 40 teams which direction is north, so that each can work out its own heading. The playbook is candid that a north star “is not immediately actionable”, which is why it pairs the star with input metrics, the few measures that teams can move directly and that together drive the star. It expects the star to be revised every one to three years in an established company.

The Verified Resale memo names its north star as verified resales completed per eligible event, and the work of this section is to say what “completed” means. Encore’s answer is a resold ticket that scans at the door. A listing is an intention and an order is a payment; the scan is the moment the buyer actually walks into the concert. Counting at the door means that a resale reissued wrongly, revoked by mistake or blocked at a gate whose revocation list had gone stale does not count, even though every earlier step reported success. It also makes the metric lagging, since it completes only after the show. Three inputs give the teams something earlier to steer by: the listing rate (the share of eligible tickets that are listed), the resale conversion (the share of listings that sell before the resale window closes) and the reissue success (the share of sold listings whose reissued ticket is accepted at the gate). Resale owns the first two; Inventory, Venue Entry and Fan Messaging share the third.

The north star sits at the level of strategy. Objectives and key results, which turn strategy into quarterly commitments, belong to Section 2.5 and are not re-taught here. What this chapter adds is the step below: the metric that decides an individual experiment.

The overall evaluation criterion

Ron Kohavi, Diane Tang and Ya Xu, in Trustworthy Online Controlled Experiments (2020), define the overall evaluation criterion (OEC) as “a quantitative measure of the experiment’s objective”. In practice it is a metric, or a small weighted combination of metrics, agreed before an experiment runs, measurable within the experiment’s few weeks, and believed to predict the long-term value the north star describes. The north star tells the company where it is going; the OEC tells one experiment whether it helped. Its great virtue is social. When the OEC is agreed in advance, an argument about a result becomes an argument about the metric, and that argument can be held calmly before anyone’s work is on the line.

The trouble is that a metric can be a ratio, understandable and comparative, and still reward harm. Kohavi and colleagues reported a case from Bing at KDD 2012. A bug in an experiment showed users very poor search results, and two of Bing’s key metrics went up: “distinct queries per user went up over 10%, and revenue per user went up over 30%”. Users were searching more, and clicking more ads, because they could not find what they wanted. A team with queries per user as its OEC would have shipped the bug. Bing put sessions per user in its OEC instead, on the reasoning that a user who finds answers comes back more often, while one who cannot simply searches harder.

Medicine has learned the same lesson at a higher price. A trial that waits to count deaths may take years, so trials often measure a surrogate endpoint, a quicker measurement believed to predict the outcome that matters, such as blood pressure. In the late 1980s the Cardiac Arrhythmia Suppression Trial tested drugs that suppressed the irregular heartbeats known to predict sudden death after a heart attack. The drugs did suppress them, and patients on two of them also died at a higher rate than patients on placebo; that part of the trial was stopped early in 1989. An OEC is a surrogate endpoint for the north star and carries the same risk. The analogy has a limit, in our favor: a software company can run thousands of experiments and check, across them, whether movements in its OEC were later followed by movements in its long-term outcomes, a check few clinicians can make.

For experiments on the resale flow, Encore’s OEC is resale conversion within an experiment’s triggered population, read alongside the reissue success, because an experiment that sells more listings but produces more failed reissues has only moved a problem from one screen to another. Experiments elsewhere in the fan app use their own OECs, and Section 4.4 explains how the definitions are kept consistent.

Guardrails

An OEC says what we hope to improve. A guardrail metric is a metric that must not get worse, however much the OEC improves. Kohavi, Tang and Xu distinguish two kinds. Organizational guardrails protect the business and its customers: latency, errors, revenue, the experience of groups the experiment was not aimed at. Trust guardrails protect the experiment itself, by checking that it was run correctly, and Section 4.3 takes them up. Latency is the classic organizational guardrail, because almost any feature can be made more engaging by making it heavier. When Kohavi and Stefan Thomke reported a Bing experiment, discussed again in Section 4.3, that raised revenue by 12% “without hurting key user-experience metrics”, it was the guardrails that entitled them to the second half of the sentence.

The word “guardrail” means something else in the Architecture book’s Section 12.4 and the Resilience book’s Chapter 8, where it names runtime safety checks on AI models; here it means a metric. For Verified Resale the guardrails come from the bet’s own risks: resale must not eat primary sales, reopen the door to bots or fraud, drive organizers away or flood Support. One guardrail Encore did not have to invent. The Resilience book’s checkout objective (Section 1.2), 99.9% of checkout attempts ending well within 3 s, already describes what fans expect of any purchase, and reusing it keeps one meaning for “checkout works” across the company. Figure 4.2 gathers the set.

Figure 4.2 — Verified Resale’s metrics. The north star is lagging by design; the inputs give teams something earlier to steer by, and the guardrails name what the bet must not cost.

The share of resales sold at the face-value cap is not in the table, and its absence is deliberate. Encore watches it, because it says something about demand for each event, but it is neither good nor bad in itself, and a metric with no direction cannot guard anything.

Recap. A good metric is comparative, understandable, a ratio, and changes what someone does; Goals–Signals–Metrics finds such metrics by starting from what users are trying to achieve. A north star tells many teams which way is north, its inputs give them something to steer by, and the OEC decides each experiment. The Bing ranking bug shows that a sensible-looking OEC can reward harm, which is why it travels with guardrails, both for the business and for the experiment’s own trustworthiness.

Exercise 4.1. For a product you know, write a north star, three input metrics and three guardrails, using Figure 4.2 as a model. For the north star, name the moment at which value is actually delivered, and check whether your current metric counts that moment or an earlier one. Then name the case, like the Bing bug, in which your OEC would improve while users were worse off.

4.2 Instrumentation as a Contract

Tracking events are an API

A word first, because Encore’s vocabulary forces it. At a ticketing company an event is a concert (encore.event_id, in the Operations book’s registry, names one). So this book calls the records that product analytics collects tracking events: structured records, emitted when a person or the system does something of interest to the product, that downstream metrics, experiments and models consume. Their names follow one grammar, set for both books: snake_case, an object and a past-tense action, with the domain as a prefix. resale_listing_created records that a listing came into being; resale_ticket_reissued records that the barcode core issued a new ticket identity. The past tense matters because a tracking event records a fact that has happened, the same reason the Architecture book names its domain events SeatSold and OrderPlaced.

Once named, a tracking event is an interface with consumers the producing team will never meet: experiment scorecards, Organizer Insights dashboards, forecasting models, and next year’s analyst. The Architecture book has already argued that a published event is a contract, evolved by addition (Section 6.1), that a stream such as ticket-sales.v2 should be run as a product (Section 6.5), and that data products follow the same logic (Section 9.4). Tracking events add something particular to that argument, which is the way they fail. When a backend team renames a field in an API, a client breaks and somebody is paged. When a frontend team renames listing_created to listing_published in a refactor, nothing breaks. A dashboard line falls to zero, or an experiment’s treatment arm quietly loses a third of its conversions, and the number that comes out is wrong without being absent.

Surveyors have a practice that captures what is at stake. Across Britain, the Ordnance Survey cut bench marks into the walls of churches and bridges, each a known height from which other heights could be measured. If a builder chiselled one off and recut it a foot lower, every level taken from it afterward would still produce a figure, and every figure would be wrong by a foot. A silent rename is a recut bench mark. The comparison breaks in one useful place: a bench mark is cut in stone so that it cannot change, whereas tracking events must change as the product does. So they change in public, with a version, an owner and a review, and every consumer learns of the change before it lands.

The tracking plan

The tracking plan is the reviewed list of a product’s tracking events: for each one, its name, its properties and their types, the trigger that emits it, its owner, and the question it exists to answer. The last field is the one most often left out and the one I would least like to lose, because an event whose question nobody can state collects data nobody will use, which a company must still secure, retain and eventually erase. The tracking plan as a reviewed artifact is common practice, supported by vendor tools such as Segment’s Protocols and Snowplow’s schemas, though not by peer-reviewed study; Trustworthy Online Controlled Experiments states the principle behind it, that logging is designed and reviewed with the feature.

Here is one entry from the Resale team’s plan:

- name: resale_listing_created
  owner: Resale team
  trigger: Resale service accepts a listing, after organizer rules pass (server side)
  question: How many eligible tickets do fans offer, and through which channel?
  properties:
    encore.event_id:   {type: string, required: true}
    listing_id:        {type: string, required: true}
    seller_pseudonym:  {type: string, required: true}
    channel:           {type: enum, values: [encore_app, rowmate, tessera]}
    price_to_face:     {type: number, min: 0, max: 1}
    seats:             {type: integer, min: 1}
  consumers: [listing_rate, organizer_insights_resale]
  version: 1

Three choices in the entry are worth reading closely. The trigger is server-side and fires only after the organizer’s rules pass, so the event counts listings that exist, and partner listings from Rowmate and Tessera are counted by the same code as Encore’s own. The identifiers reuse the names in the Operations book’s attribute registry (Section 4.2 of that book), so encore.event_id means the same concert in a tracking event, a trace and a sales report, and the three can be joined. And the seller is a pseudonym, never a name or an email address, for reasons the next subsection gives.

Encore checks the plan in continuous integration: a change that emits an event not in the plan, or changes a property’s type, fails its build, and a change to the plan itself needs a review from the event’s listed consumers. This is Encore’s choice, and a sound one at its size, but the evidence for it is practitioners’ reports. The tracking-event schemas sit in the schema registry that the Event Backbone team runs, beside the domain events, and the Experimentation team owns the plan’s format and the check.

Instrumentation in the platform

Where an event is logged matters as much as what it records. Kohavi, Tang and Xu give separate chapters to client-side experiments and to instrumentation, and the lesson of both is that clients and servers see different things. A server knows for certain that a listing was created, because it created it. Only the client knows whether a fan actually saw the verified-listings panel, which may have loaded below the fold or after the fan left. Outcomes therefore belong on the server, and exposures, the moments at which a person actually met a variant, on the client. Client logging also loses data in ways server logging does not: browsers close, phones go offline, ad blockers drop requests, and each loss can differ between the arms of an experiment.

The best evidence that this matters comes from Microsoft. Aleksander Fabijan and colleagues reported at KDD 2019 that about 6% of Microsoft’s experiments showed a sample ratio mismatch, a split between the arms that differed from the one designed, and one cause in their taxonomy was a faster variant. Because the treatment loaded faster, its telemetry had more time to be sent before users left, so the experiment recorded more users in the faster arm. The logging had counted them more completely, and a bug in instrumentation had become a bias in the result.

That is the argument for putting logging in the platform. If every team writes its own code to batch, retry and send tracking events, every team will lose data in its own way. Encore’s web and mobile shells carry one tracking SDK, owned by the Experimentation team, which batches events, queues them while offline, stamps the registry identifiers and active experiment assignments on each one, and enforces the consent state described below. The same reasoning shaped the Operations book’s account of OpenTelemetry (its Section 4.2): a shared vocabulary stays shared only if the code that writes it is shared too. The SDK’s behavior on phones is Section 8.4’s subject.

Consent and the fans we cannot see

European law makes consent part of the instrument. The ePrivacy Directive (2002/58/EC, Article 5(3), as amended in 2009) requires consent before information is stored on or read from a user’s device unless it is strictly necessary for the service the user asked for, and the European Data Protection Board’s Guidelines 2/2023 extend this to pixels, URL tracking and similar techniques. The GDPR requires a lawful basis for processing (Article 6), limits retention (Article 5(1)(e)), and says in Recital 26 that pseudonymized data is still personal data. In Planet49 (2019) the Court of Justice of the EU held that a pre-ticked box is not valid consent. The ePrivacy Regulation meant to replace the Directive was withdrawn in February 2025, so the Directive still governs; the Commission’s November 2025 “Digital Omnibus” proposal to move cookie rules into the GDPR remains a proposal.

Within those rules there is one well-documented path for measurement without consent. France’s data protection authority, the CNIL, exempts audience measurement from consent under strict conditions: users are informed and can object; the purpose is limited to audience measurement and A/B testing; the data is not combined with other processing; the tracker is first-party; the last byte of the IP address is truncated; and trackers live no longer than 13 months. Encore built a narrow first-party channel to those conditions, which counts exposures and outcomes for experiments and nothing else. It uses the channel in France, where the CNIL’s guidance applies, and wherever else a national regulator accepts the same scope; everything beyond it waits for consent. Raw tracking events are kept for 13 months, pseudonymously, in Encore’s EU regions, and the erasure machinery of the Architecture book (Sections 6.2 and 9.5) reaches every copy when a fan asks to be forgotten.

The design question most teams skip is what consent does to the numbers. Where consent is required, between 52% and 78% of Encore’s fans grant it, depending on the country. Those who refuse are not a random sample; in Encore’s data they skew toward desktop users, so a metric computed from consented fans describes a different population from the one Encore serves, and differently in each country. Consent can also differ between the arms of an experiment. If a treatment moves or delays the consent banner, a different share of each arm consents, and the consented population splits unevenly, which the trust checks of Section 4.3 exist to catch. Encore’s rule is that no experiment may change the consent experience except one designed to study it, and that the exempt channel carries the OEC wherever the law allows.

Two kinds of telemetry

Operational telemetry, the traces, metrics and logs of the Operations book’s Chapter 4, answers the question “is the system working?”. Product telemetry answers “is the product worth it?”. The first is read by on-call engineers in minutes, kept for weeks and sampled heavily; the second is read by product trios over weeks, kept for months, and never sampled in a way that would bias an experiment. The boundary blurs, and honestly so: latency is an operational signal and also, as Kohavi’s book shows, one of the most powerful product metrics there is. Encore’s resolution is that the two kinds share identifiers, so that encore.event_id and encore.country mean the same in both and can be joined, while keeping separate owners, retention and contracts. The Reliability team owns the operational registry, the Experimentation team owns the tracking plan, and each reviews the other’s additions to the shared names.

Recap. A tracking event is an interface whose consumers its producer never meets, and its characteristic failure is silent: a renamed event yields wrong numbers instead of errors. A tracking plan names each event, its owner, its trigger and the question it answers, and a shared SDK makes 40 teams log alike, because instrumentation bugs become biased results. Consent law shapes the instrument, and refusals bias the sample in ways that differ by country and can differ between arms.

Exercise 4.2. Write the tracking plan for one feature you own: five tracking events, their properties, their triggers, their owners and the question each answers. Mark each event as server-side or client-side and say why. Then find one event in your current analytics whose question nobody can state, and decide whether to delete it.

4.3 Online Controlled Experiments

Most ideas fail

The strongest argument for experimentation is how often our ideas turn out to be wrong. In a 2009 paper for Microsoft’s ThinkWeek, of which a public version was released, Kohavi and colleagues reported that of experiments designed to improve a key metric, “only about one-third were successful at improving the key metric”. Kohavi and Thomke wrote in 2017 that “at Google and Bing, only about 10% to 20% of experiments generate positive results”, and that at Microsoft as a whole a third prove effective, a third are neutral and a third are negative. These are ideas that experienced people believed in enough to build. A company that ships without testing is therefore shipping a great deal of neutral and harmful work, with no way of knowing which part.

Kohavi gave the opposing habit a name. In a KDD paper of 2007 he and his co-authors urged readers to listen to their customers and not to the HiPPO, the HIghest-Paid Person’s Opinion. The joke has lasted because it names a real force. In the absence of evidence, decisions go to rank, and rank is no better at predicting what users will do than anyone else.

The same evidence cuts the other way, and that half is less often told. At Bing, an employee proposed lengthening ad titles by moving text from the line beneath into the title. The idea sat in the backlog “for more than six months”, judged low priority, until an engineer tried it in a cheap experiment. Revenue rose 12%, worth more than $100 million in the United States alone by the authors’ account, and it became, in their words, “the best revenue-generating idea in Bing’s history”. Intuition does not only overvalue bad ideas. It also buries good ones, and here it was a cheap test that brought one back.

Designing one experiment

An online controlled experiment, or A/B test, randomly assigns units, usually users, to two or more variants of a product, and compares a metric between them, so that any difference beyond chance can be attributed to the variant. It is Fisher’s randomized block with fans for plots. The design has six parts, and each must be written down before the experiment starts: a hypothesis, a unit of randomization, a trigger, an OEC with its guardrails, a sample size, and an analysis plan.

Encore’s first experiment on Verified Resale is planned for the weeks after the launch widens across France. Its hypothesis is that showing verified listings on a sold-out event page reduces the share of fans who leave with no next step: they neither open a listing, join the event’s alerts nor move to another event. The unit is the fan, identified by account when signed in and by device otherwise. The trigger is the condition that admits a unit into the analysis, here reaching the sold-out page of an event with resale open; it matters because most visitors never see a sold-out page, and analyzing everyone would bury a real effect among millions of fans the change could not have touched. One caution belongs in the plan. The test changes what fans see, but if treated fans buy listings that control fans would otherwise have bought, the arms are no longer independent. Section 4.5 takes that problem up; for a display test in which few fans convert, Encore judges the bias small, and says so in writing.

The sample size follows from a short calculation that every product team should be able to do. For a test at the usual 5% significance level and 80% power, the number of units needed in each arm is approximately 16σ²/δ², where σ² is the variance of the metric and δ is the smallest change worth detecting. The 16 is not magic: it is twice the square of the sum of the two normal quantiles involved, 2 × (1.96 + 0.84)², which comes to 15.7. For a rate p, the variance is p(1 − p). On the sold-out pages, about 80% of fans currently leave with no next step, so σ² is 0.8 × 0.2, or 0.16. Figure 4.3 works the arithmetic for the roughly 70,000 fans a week Encore expects to trigger in France.

Figure 4.3 — Sample size for the sold-out-page experiment, from 16σ²/δ² with σ² = 0.16. Halving the effect we want to detect quadruples the fans we need: a 2-point change (80% to 78%) needs 6,400 fans per arm, a quarter-point change 409,600 and about 12 weeks. The 1-point and 2-point tests finish in under a week but run for two, a minimum that covers two full weekly cycles of on-sales.

The curve shows the most important fact about experiments: sample size grows with the square of the precision we ask for. A team that wants to detect a quarter-point change at this traffic must wait three months, and probably should not try. One well-proven method buys precision cheaply. Alex Deng, Ya Xu, Ron Kohavi and Toby Walker described CUPED at WSDM 2013: it uses each unit’s behavior before the experiment to remove variance unrelated to the treatment, and on Bing it cut variance by about half, halving the traffic or time an experiment needs. Every major platform now offers it, and Encore’s applies it by default where a fan has history.

The analysis plan is the part most often written afterward and most in need of being written before. Fisher’s null hypothesis, that the panel makes no difference, can be disproved by the data but never proved, and the plan says in advance what disproving it will look like: which test, which metric, which threshold, which duration, and what decision follows from each outcome. Once results are visible, there are many reasonable-looking analyses to choose from, and the one we choose tends to be the one we hoped for. Here is the Resale team’s pre-registration, as it sits in the experiment’s record:

experiment: resale-soldout-listings-v1
hypothesis: Verified listings on a sold-out page reduce the no-next-step rate
unit: fan (account id; device id if signed out)
trigger: fan reaches sold-out page of an event with resale open
variants: {control: current sold-out page, treatment: verified listings panel}
oec: no_next_step_rate          # lower is better
guardrails: [checkout_slo, resale_support_contacts, bot_fraud_rate, page_latency_p75]
trust_checks: {srm: "chi-squared, halt if p < 0.001", aa_preperiod: true}
design: {alpha: 0.05, power: 0.8, mde: 1 point absolute, variance_reduction: cuped}
duration: 14 days fixed; no decision before day 14
decision: ship if OEC improves and no guardrail degrades beyond its threshold
interference_note: display only; listing sell-out by arm monitored daily

Trusting the result

A surprising result deserves suspicion before celebration. Kohavi and his co-authors call this Twyman’s law, after the British audience researcher Tony Twyman: “Any figure that looks interesting or different is usually wrong.” The law is a statement about base rates. Because most changes have small effects, a large one is more likely to come from an error in logging, assignment or analysis than from a breakthrough, and the right first response to a large effect is to look for the error.

Two checks do most of the work. The first is the A/A test, an experiment in which both arms receive the same experience. At a 5% threshold it should show a significant difference about 5% of the time, and a platform whose A/A tests do so more often has something wrong with its assignment or its statistics. Encore runs A/A tests continuously on a slice of traffic and over each new experiment’s pre-period. The second is the sample ratio mismatch (SRM) check, a test of whether the number of units in each arm matches the designed split. If an experiment designed as 50/50 records 50.8% in one arm across a million units, chance is a very unlikely explanation: something, a redirect, a crash, a logging difference, a consent banner, has removed users from one arm selectively, and no statistics afterward can repair the comparison. Fabijan’s study found SRMs in about 6% of Microsoft’s experiments, and observed that “a product running ten thousand experiments in a year can expect to see at least one SRM per day”. On Encore’s scorecards an SRM hides the results until someone has explained it.

Twyman’s law is a check, and it can be passed. Within hours of starting, the ad-title experiment at Bing was producing such abnormally high revenue that it triggered a “too good to be true” alert. Such alerts usually signal a bug, Kohavi and Thomke wrote, but not in this case: the analysis that followed showed the gain was real, and that it came without hurting key user-experience metrics. The law did its job by forcing the second look, and the result survived it.

Novelty, peeking and the long run

Three habits undermine well-designed experiments. The first is peeking: watching the p-value every day and stopping as soon as it falls below 0.05. Each look is another chance for noise to cross the line, so a team that looks daily and stops at the first crossing declares far more false winners than its 5% threshold promises. The proven remedies are of two kinds. One is a fixed horizon, as in Encore’s pre-registration. The other is a sequential method built for repeated looks, such as the group sequential tests that Spotify’s engineers compared and recommended in 2023 where the maximum sample size can be estimated, or the always-valid inference of Ramesh Johari and colleagues. Each family is sound when its conditions hold.

The second habit is mistaking novelty for improvement. Users react to change itself: a new panel attracts clicks because it is new, and the effect fades as it becomes familiar. The opposite also happens, when experienced users resist a change they will come to prefer. Plotting the treatment effect day by day shows both: an effect that shrinks steadily is novelty, and one that grows may be learning.

The third is forgetting the long run. Henning Hohnhold, Deirdre O’Brien and Diane Tang reported at KDD 2015 how Google measured the long-term effects of the ads on its search pages. Short experiments suggested that fewer ads would cost revenue, but long-running holdouts showed that users learn, growing blind to ads when shown many, so that the effect of a change in ad load keeps developing for months. Google applied the method to a 50% reduction in ad load on mobile search, a change a short test would have rejected. Encore’s lesson is to keep a small, long-lived holdout, a group deliberately kept without a change for months. For Verified Resale it is nine French organizers whose events go without resale for the first six months after launch. Because that affects their fans, the organizers are told what the holdout is for, consent to it, and may leave at any time.

What Encore will not test

Experiments are done to people, and people have interests the experimenter does not share. Trustworthy Online Controlled Experiments takes its ethics from The Belmont Report (1979), written for research on human subjects in the United States after the abuses of the Tuskegee syphilis study had come to light. The report asks for respect for persons, beneficence and justice: people should be treated as able to decide for themselves, the risks they bear should be weighed against the benefits, and the burdens and benefits of research should be fairly shared. Most product experiments raise none of these questions, because both arms are reasonable experiences and the risk to anyone is small.

The case that taught the industry otherwise was published in 2014. Adam Kramer, Jamie Guillory and Jeffrey Hancock reported in the Proceedings of the National Academy of Sciences an experiment in which, for one week, Facebook reduced the positive or the negative posts in the News Feeds of 689,003 users, to see whether their own posts shifted in the same direction. The journal later published an Editorial Expression of Concern about the study’s consent. What made it a lesson was its subject. The treatment aimed to change people’s emotional state, without their knowledge, and nothing in a terms-of-service agreement amounts to consent to that.

Encore’s rules are stricter than the law requires, because selling scarce seats makes some experiments unfair by their nature. Encore never experiments on the price a fan pays, or on which fan gets a seat. Packaging and presentation may be tested, as Section 5.5 sets out: how the all-in price is explained, which listings appear first. In every such test each fan pays exactly what any other fan would pay for the same ticket, and a fee changes only by a deliberate, announced decision that applies to everyone. The cautionary history behind the first rule, Amazon’s price test of 2000, has its home in Chapter 5, and the rules agree with Section 3.2’s rule for demand tests: a fake door must never take money or hold a seat.

Recap. Most ideas fail to move the metrics they target, and some good ones are buried by intuition, so testing pays in both directions. An experiment is designed in writing before it runs: hypothesis, unit, trigger, OEC and guardrails, sample size and analysis plan, and the arithmetic of sample size makes precision expensive. Twyman’s law, A/A tests and the sample ratio check protect against errors that look like discoveries, fixed horizons or sequential methods protect against peeking, and holdouts see what short tests cannot. Some experiments are not ours to run.

Exercise 4.3. For one change your team shipped last quarter, compute the sample needed per arm to detect a 2% relative change in its main metric at your traffic, using 16σ²/δ². Was an experiment ever possible? If it was, write the pre-registration you should have written; if it was not, say what you would have had to change, such as the trigger, the metric or the variance, to make it possible.

4.4 An Experimentation Platform for 40 Teams

Assignment as the substrate

An experiment at a large company is one of hundreds running at once, and what makes that possible is the machinery of assignment. Each fan must land in an arm deterministically, seeing the same variant on every visit and device, without the platform storing a lookup for each fan. The standard technique is to hash the unit’s identifier with an experiment-specific salt into buckets; Fabijan and colleagues describe Microsoft’s ExP platform hashing users into 1,000 buckets of 0.1% each. Hashing is the modern form of what Fisher called “the physical act of randomisation”: nothing about the fan, and nothing about the experimenter’s hopes, influences which arm the fan lands in.

Encore’s assignment runs through the same flag system the Resilience book describes for progressive delivery (its Section 4.3). A release flag and an experiment flag are evaluated by the same SDK, against the same targeting rules, and they differ in purpose and in lifetime. A release flag exists to roll a change out safely and is removed once the rollout ends; an experiment flag exists to answer a question and, in the Resilience book’s words, “should live as long as its experiment”. The Software book (Section 4.5) covers how flags are handled in development; what this section adds is the assignment discipline that turns a flag into an experiment.

That discipline has three parts. The first is the unit of randomization, the entity whose identifier is hashed: a fan, a device, an event or an organizer. It must be at least as large as what a variant can affect, or one fan may see both arms and contaminate both. The second is layering: running many experiments on the same fans at once, each with its own independent hash. Fisher argued for the same efficiency in 1926, against the advice of his day: “No aphorism is more frequently repeated in connection with field trials, than that we must ask Nature few questions, or, ideally, one question, at a time. The writer is convinced that this view is wholly mistaken.” Layering works because independent randomization spreads each experiment’s effect evenly across the arms of every other; where two experiments change the same screen in conflicting ways, the platform puts them in one exclusive layer. The third part is exposure logging, recording on the client the moment a fan actually met the variant, so that the analysis counts the fans who were exposed and not merely those assigned. Section 8.4 shows how this is done on phones.

Metrics defined once

The commonest failure of experimentation at scale has nothing to do with statistics. It is that every team computes its own version of the same metric. One team’s “conversion” divides orders by sessions, another’s by visitors, and a third’s excludes fans who arrived from a partner. Each is defensible on its own. Together they make the experiments of different teams impossible to compare, and they turn every leadership meeting into a debate about whose number is right.

The history of measurement offers a parallel. Before the metric system, neighboring French towns might use different feet, and a merchant trading between them could be cheated by either. Revolutionary France defined the metre in the 1790s so that one length would mean the same everywhere, kept by a central authority with a physical standard. A metric definition kept in one place does the same work across teams. The analogy breaks because a metre was meant never to change, whereas a metric definition must sometimes change with the product, and so it needs versions, owners and a record of why each version replaced the last.

Airbnb’s Minerva platform is the best-documented case of a company solving this. It grew out of the metric repository behind Airbnb’s experiments, on the principle of “define once, use everywhere”: a metric is defined in one place, in code, and every experiment scorecard, dashboard and report computes it from that definition. Encore’s version is a metric catalog owned by the Experimentation team, in which each metric is a reviewed definition:

metric: resale_conversion
owner: Resale team; reviewed by Experimentation
definition: listings sold before window close ÷ listings created
numerator:   count(resale_order_completed) by listing_id, dedup
denominator: count(resale_listing_created) where window_closed
unit_of_analysis: listing; aggregated to the experiment's unit
direction: higher is better
used_as: [input metric, OEC for resale-flow experiments]
version: 2   # v1 counted orders, including those later reversed

Every experiment’s scorecard then shows three things together: the OEC with its confidence interval, the guardrails with theirs, and the trust checks, the sample ratio test and the A/A pre-period. Putting them on one page is a design choice with a purpose. A team reading only the OEC will ship a win that broke a guardrail, and a team that never sees the trust checks will trust a result that should have been discarded.

Crawl, walk, run, fly

Organizations do not adopt experimentation all at once. The clearest description of how they grow is by Aleksander Fabijan, Pavel Dmitriev, Helena Holmström Olsson and Jan Bosch (ICSE 2017), who drew on case studies centered on Microsoft and named four phases: crawl, walk, run and fly. Early on, specialists set up and analyze a few experiments by hand; in the middle, a platform, standard metrics and trust checks appear and more teams test; later, experimentation becomes the default way to ship a change, an OEC is agreed, and the organization learns from its accumulated results. The model is descriptive: it shows what growth looked like in the companies studied, and promises nothing about following it elsewhere.

By its own judgment Encore is at walk. Last year it ran about 310 experiments, two-thirds of them from seven of its 40 teams; the rest of the company tests rarely or never. The platform, the metric catalog and the trust checks exist. What Encore lacks for the next phase is an agreed OEC for each main product, the habit of testing ordinary changes as well as headline features, and the institutional memory described next. Those are matters of practice and agreement more than of software, which is why moving from walk to run is usually slower than building the platform was.

Memory and review

An experiment’s result is worth more than the decision it informs, because it tells the next team something about how fans behave. Most companies throw that value away: the results live in a slide deck, and two years later another team pays for the same lesson. Raphael Lopez Kaufman, Jegar Pitchforth and Lukas Vermeer described in 2017 how Booking.com spread experimentation across its teams, naming among the ingredients a shared archive of past results, a reusable code library, monitoring of data pipelines, and safeguards that let individuals own their experiments end to end. Kohavi, Tang and Xu give a chapter to the same idea, institutional memory and meta-analysis across many experiments.

Encore’s archive is the experiment record itself: the pre-registration, the scorecard and a short written conclusion, kept together and searchable by product area, metric and screen. Before an experiment starts, a reviewer from another team reads its pre-registration and asks whether the OEC, the unit and the trigger fit the hypothesis. The archive’s most valuable entries are its failures, recorded with their reasoning, and they are also the entries most likely to be lost, since nobody presents a failure at the quarterly review.

Build, buy or rent

Experimentation platforms are now products a company can buy, and the decision is the kind the Architecture book’s Section 13.1 maps, with its four options (build, adopt, buy or rent) and its key term, dependence: how much of our ability to act we place in someone else’s hands. Encore builds assignment, the metric catalog and the integration with its tracking plan, because those are tied to its own data contracts, consent rules and flag system. It rents the statistics engine, which computes intervals, variance reduction and sequential tests, because that part is the same for every company and specialists maintain it better than a team of Encore’s size could.

The dependence is real, and recent history shows how quickly it can change. In May 2025 Datadog announced that it had acquired Eppo, one of the leading independent experimentation vendors, and in September 2025 OpenAI acquired Statsig, another; within four months, both had new owners with priorities of their own. Open-source platforms such as GrowthBook offer a different form of dependence. Encore’s split is a judgment, not a finding, and its main defense is that the rented part is replaceable: the statistics engine reads Encore’s assignments and metric definitions, and a different engine could read them too.

Recap. Deterministic hashing makes assignment reproducible and impartial, and layering lets many experiments share the same fans, as Fisher’s factorial designs once shared the same field. Metrics defined once, in code, keep 40 teams’ numbers comparable, and the scorecard shows the OEC, the guardrails and the trust checks together. Maturity grows from a few experiments to a default practice, and what holds most organizations back is agreement and memory more than software. Build what is tied to your own contracts and rent what is generic, keeping the rented part replaceable.

Exercise 4.4. Find where your organization’s experiment results live today. Ask a colleague on another team to find last year’s least successful experiment in your area, with its hypothesis and result, and time how long it takes. If they cannot find it, write down what an archive entry would have needed to contain.

4.5 When You Cannot A/B Test

When the plots touch

Fisher’s method rested on an assumption so natural in a wheat field that he hardly needed to state it: the fertilizer on one plot does not change the yield of the next. Statisticians later gave the assumption a name. The stable unit treatment value assumption (SUTVA) says that each unit’s outcome depends only on its own treatment, never on the treatments of other units. Online experiments inherit it, and for most of them it holds well enough: whether one fan sees a new search box has no effect on what another fan finds.

Marketplaces break it. Thomas Blake and Dominic Coey showed at ACM EC 2014 how an eBay email-marketing experiment went wrong for this reason. Treated and control bidders competed in the same auctions, so when the treatment made some bidders more active, they won items that control bidders would otherwise have won. The treatment group looked better partly because the control group had been made worse, and the naive comparison overstated the effect. Kohavi, Tang and Xu give the problem its own chapter, on leakage and interference between variants.

At Encore the interference is literal. A resold seat goes to exactly one fan. If an experiment made resale more attractive to fans in the treatment arm, they would buy listings that control fans would otherwise have bought; the treatment would show more completed resales, the control fewer, and the difference would be mostly a transfer between arms, not an effect of the product. The same applies to sellers, who list into one pool that both arms buy from, and to organizers, whose rules govern every fan of their events. A user-level A/B test of Verified Resale itself would measure something real, but not the thing the memo needs to know: whether resale, available to everyone, makes the market better. The question is how to choose units large enough to contain the interference.

Randomizing clusters

The standard answer is cluster randomization: randomizing groups of units that interact, so that the interference stays inside each group. The best marketplace evidence for it comes from Airbnb. David Holtz, Felipe Lobel, Ruben Lobel, Inessa Liskovich and Sinan Aral ran a meta-experiment around a fee increase, published in Management Science in 2025, in which the same change was evaluated both by randomizing individual listings and by randomizing clusters of listings that compete for the same guests. At least 20% of the effect the individual-level design estimated was interference bias, which the cluster design removed, and there was suggestive, though not significant, evidence that the bias was larger where demand was constrained. The case could hardly be closer to Verified Resale: a fee, in a marketplace, with the bias measured.

The price of clusters is statistical power. An experiment’s precision depends on the number of independent units, and an experiment with 500 clusters has the precision of something nearer 500 units than of the millions of fans inside them. Clusters must also actually contain the interference. For seats, the event is a natural cluster, because fans buying resale for one concert do not compete with fans buying for another. But fans browse several events, and a fan who finds no listing for one concert may buy for another, so interference leaks across clusters at their edges, and the design reduces the bias without removing it.

Encore’s cluster experiment uses events as units, and it asks a narrow question: how the resale fee should be shown. One version shows the all-in price as a single figure; the other shows the same all-in price with the fee and its split among partner, organizer and Encore itemized beneath it. The total is identical in both, so no fan pays a different price from any other, as Section 4.3’s rule requires. When the launch widens to all consenting French organizers, 38 of them agree to let about 520 of their events be randomized between the two presentations, so that every fan of a given event sees one presentation and fans comparing notes about one concert see the same thing. The outcomes are completed purchases and the Support contacts that ask where the money went. With 520 clusters the experiment can detect only large effects, and the pre-registration says so. Figure 4.4 shows why the event, not the fan, is the unit.

Figure 4.4 — Randomizing fans against randomizing events. When fans of one event are split, treated fans take seats from control fans and the difference is mostly a transfer; when whole events are split, each arm buys from its own listings, and only fans who browse several events leak across the edges.

Switchbacks and geography

When interference runs through a whole market, two other designs become useful. A switchback experiment applies the treatment to an entire market for a window of time, then switches it off and on again in a randomized sequence of windows, and compares the windows. DoorDash described the method in 2018 for changes to how deliveries are assigned to couriers: every order in a region during a window receives the same policy, so treated and control orders never compete for the same couriers, and the analysis is at the level of region and window. It works only when the treatment’s effect dies out well within one window, so that windows do not carry over into each other, and when there are many windows, because each window is one observation.

Those conditions decide where Encore can use switchbacks, and for Verified Resale the answer is nowhere yet. A one-off on-sale defeats them, because its whole effect falls within one window. The resale market looks more promising at first, since it runs continuously across thousands of events, and the Resale trio first proposed a switchback for the fee presentation, alternating it across the whole French market by the hour. They rejected it for two reasons. A fan who returned an hour later could find the same fee presented differently, which sits badly with Encore’s rule on price integrity and with the EU’s requirement that totals be shown clearly. And a resale listing persists from one window into the next, so what one hour’s presentation did to the pool of unsold listings would carry into the following hour. Iavor Bojinov, David Simchi-Levi and Jinglong Zhao, who set out the theory of switchback designs in Management Science in 2023, show that such carryover must die out quickly relative to the window for the design to be valid. Neither condition held, which is why the fee presentation sits in the cluster design above. A delivery order is consumed within its window and a resale listing is not, and the analogy with DoorDash breaks exactly there.

Geo experiments randomize geographic regions instead of time. Jon Vaver and Jim Koehler described the method at Google in 2011 for measuring advertising: non-overlapping regions are randomly assigned to treatment or control. It needs many regions, and that is where it fails Encore. Encore sells in too few EU countries for a randomized design among them to have any power, and they differ too much in size and tour calendars to be exchangeable. Cities offer more units, but fans travel to concerts, and a resale in Lyon may be bought by a fan from Grenoble. Encore therefore runs no geo experiment on Verified Resale, and its evaluation plan says so, so that nobody later mistakes the market-level comparison below for one.

Quasi-experiments

When the rollout itself was not randomized, as Encore’s is not, a quasi-experiment estimates the effect by constructing a comparison the data did not supply. Difference-in-differences compares the change over time in the treated group with the change in an untreated group, so that anything affecting both, such as the season, cancels out. Its best-known use is David Card and Alan Krueger’s 1994 study of a minimum-wage rise in New Jersey, compared with fast-food restaurants in neighboring Pennsylvania. The method depends on the two groups having moved in parallel before the treatment, and nothing in the data can prove they would have kept doing so.

Synthetic control builds the comparison more carefully. Alberto Abadie, Alexis Diamond and Jens Hainmueller introduced it in 2010 in a study of California’s tobacco-control program of 1988. No single state resembled California well enough, so they built a weighted combination of other states that matched California’s cigarette sales and their drivers before the program. Afterward the paths diverged, and by 2000 per-capita sales in California were about 26 packs lower than in its synthetic counterpart. Abadie’s guide of 2021 sets out when the method fails: it needs a long pre-period, a donor pool the treatment did not reach, and no anticipation, in which people change their behavior because they expect the treatment.

Encore launches Verified Resale in France first, for business reasons: its design partners and its contract season. A market chosen for reasons cannot also be chosen at random, so Encore judges the market-level effect with a synthetic control instead, built from nine EU markets where it sells but has not launched resale. The question is the one the memo’s guardrail asks, whether resale eats the primary sale, and the outcome is primary sell-through per eligible event, week by week, over a pre-period of 104 weeks. The memo’s second kill criterion, K2, is written in the same terms: at month 18 the bet stops if primary sell-through in France has fallen more than 2 points below its synthetic counterpart (Section 10.1). Each of Abadie’s conditions must be argued. Two years of weekly history give a long pre-period, and nine donors give a pool. Spillover cannot be taken for granted, because fans cross borders for tours, and resale in France could change what Belgian fans do, so Belgium is removed from the donor pool before the analysis begins. And the launch was announced in advance, so some French fans may have changed their buying in anticipation, which the plan names as a threat it cannot remove.

The fraud the memo’s diagnosis described is harder to measure, because the grey market happens on sites and in messages Encore never sees. The closest measurement Encore owns is at its own doors: the rate of duplicate-ticket rejections at the gate per 10,000 scans, which Venue Entry has recorded everywhere for years. Encore reads a fall in that rate on resale-enabled events as supporting evidence, and claims nothing it cannot measure about displacement of the grey market.

The rollout across countries brings one more caution, which the plan for Verified Resale originally lacked. When a change reaches different countries in different months, the usual regression for difference-in-differences, with fixed effects for country and period, can produce biased and even wrongly signed estimates, because it implicitly uses already-treated countries as controls for later ones. Andrew Goodman-Bacon, and Brantly Callaway and Pedro Sant’Anna, showed this in 2021 and offered estimators that avoid it. Encore will use one of those estimators as the launch widens beyond France, and it will record the choice in the evaluation plan before the second country goes live.

Deciding without certainty

None of these designs gives the certainty of a clean randomized test, and the temptation is to pretend that one of them does. The honest alternative is to combine several imperfect methods, each strong where another is weak, and to decide against criteria written before any of them reports; Trustworthy Online Controlled Experiments devotes chapters to the complementary and observational methods that support this. Fisher’s standard from 1926 still applies: “A scientific fact should be regarded as experimentally established only if a properly designed experiment rarely fails to give this level of significance.” His emphasis was on replication, and a result that keeps appearing across different designs deserves more trust than any single one. Figure 4.5 shows how Encore’s evaluation plan for Verified Resale assigns each question to a design.

Figure 4.5 — Encore’s evaluation plan for Verified Resale. Each question gets the strongest design its interference allows, and the plan names each design’s main threat before any result arrives.

Two rules make the plan honest. The first is that its decision criteria, the thresholds for each metric and what follows from each outcome, are written before launch, in the pivot-or-persevere discipline of Section 3.2 and beside the memo’s kill criteria (Section 10.1). The second is that the decision record, in the form of the Architecture book’s decision records (Section 1.4), states the residual uncertainty plainly: which conclusions rest on randomized evidence, which on quasi-experiments, and which on judgment. A combined judgment is weaker evidence than a randomized test, and the people who revisit the decision a year later need to know how much weight it can bear. I have come to think that the most dangerous moment in measuring a product is the one in which a quasi-experiment is reported as if it were an A/B test, because from then on everyone who reads it believes something more certain than anyone ever knew. Chapter 10 shows the plan in use when Verified Resale launched (Section 10.4).

Recap. When units compete for the same seats, the stable-unit assumption fails, and a user-level test measures transfers between arms more than effects. Clusters contain interference at a cost in power, as Airbnb’s fee experiment measured; switchbacks need effects that die out within a window, which persistent resale listings do not, and geo experiments need more regions than Encore has countries. Quasi-experiments estimate effects from rollouts that were not randomized, under conditions that must be argued, and a decision built from several designs must record how uncertain it remains.

Exercise 4.5. Name one decision in your product where interference would bias an A/B test: who competes with whom, and for what? Choose the unit or design that would contain it, estimate how many independent units that leaves you, and write one sentence saying what the design cannot show.

Kata: Two Baskets, One Courier

Your brief: “Pantry.” Pantry is a grocery-delivery marketplace with 150 engineers in 22 cities across four countries. Customers order from about 1,800 partner stores, and about 9,000 independent couriers deliver within an hour; at six in the evening in any one city, every courier on shift serves the same pool of orders. Operations wants to launch batching, in which one courier collects two orders from nearby stores and delivers both on one trip. A two-week pilot in one city reported 9% more orders delivered per courier-hour, and the product director wants it everywhere next month. But complaints about late deliveries rose 14% in the pilot city that fortnight, couriers’ forums say batched trips pay less per hour, and nobody can say whether batching or the weather caused either change. An analyst has proposed an A/B test randomized by customer. The head of data has asked you to design the evaluation before anything ships, in the spirit of DoorDash’s switchbacks, without copying them.

Deliverables:

1. The metrics — Write an OEC for the batching decision, and guardrails for all three sides: customers (delivery time against promise, late-order rate), couriers (earnings per hour on shift, including waiting time) and stores (order hand-off wait). Say which guardrail you would defend first if the OEC improved while it fell, and who should agree to that ordering.
2. The interference map — Explain in a paragraph why randomizing by customer would bias the result: who competes with whom, and for what. Then name the smallest unit that contains the interference.
3. The design — Specify a switchback: the market unit, the window length and why it exceeds the carryover (a courier mid-batch at a window’s end), the number of windows, and how windows are randomized. Say what the design cannot show, and how a long-term holdout or a later city-level comparison would cover it.
4. The sample — Estimate how many windows are needed to detect a 3% change in the OEC, stating the variance you assumed and where it came from. Say whether variance reduction could shorten the experiment.
5. The pre-registration — Write the experiment’s pre-registration in the form of Section 4.3: hypothesis, unit, trigger, OEC, guardrails, trust checks, duration and decision rule.
6. The decision record — Write the half-page record you would want the product director to sign before rollout: the criteria, the designs combined, and the uncertainty that remains.

Rubric:

Where you now stand

You can now choose a metric that changes behavior, name the moment at which value is delivered, and set guardrails beside it. You can treat tracking events as contracts, design an experiment in writing before it runs, compute what it needs, and recognize the errors that look like discoveries. And you can tell when a clean experiment is impossible, choose a design whose unit contains the interference, and record how certain the result really is. Every metric in this chapter took one thing for granted: the price was settled, and Encore measured what happened at it. For Verified Resale the price is part of the product. The fee, its split among partner, organizer and Encore, and the cap that protects fans decide whether three sides will join the market at all. Chapter 5 takes up pricing, and with it the kind of experiment this chapter has ruled out.

References

• Ron Kohavi, Diane Tang, Ya Xu — Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020.
• Ronald A. Fisher — The Design of Experiments. Oliver and Boyd, 1935 — free online.
• Ronald A. Fisher — The Arrangement of Field Experiments. Journal of the Ministry of Agriculture 33, 1926 — free online.
• Alistair Croll, Benjamin Yoskovitz — Lean Analytics. O’Reilly, 2013.
• Ron Kohavi, Stefan Thomke — The Surprising Power of Online Experiments. Harvard Business Review, September–October 2017.
• David Holtz, Felipe Lobel, Ruben Lobel, Inessa Liskovich, Sinan Aral — Reducing Interference Bias in Online Marketplace Experiments Using Cluster Randomization: Evidence from a Pricing Meta-Experiment on Airbnb. Management Science 71(1), 2025 — free online.
• Aleksander Fabijan, Jayant Gupchup, Somit Gupta, Jeff Omhover, Wen Qin, Lukas Vermeer, Pavel Dmitriev — Diagnosing Sample Ratio Mismatch in Online Controlled Experiments. KDD, 2019.
• Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, Ya Xu — Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained. KDD, 2012.
• Aleksander Fabijan, Pavel Dmitriev, Helena Holmström Olsson, Jan Bosch — The Evolution of Continuous Experimentation in Software Product Development. ICSE, 2017.
• Alberto Abadie, Alexis Diamond, Jens Hainmueller — Synthetic Control Methods for Comparative Case Studies. Journal of the American Statistical Association 105(490), 2010.
• Alberto Abadie — Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects. Journal of Economic Literature 59(2), 2021.
• Henning Hohnhold, Deirdre O’Brien, Diane Tang — Focusing on the Long-term: It’s Good for Users and Business. KDD, 2015 — free online.
• Ramesh Johari, Pete Koomen, Leonid Pekelis, David Walsh — Always Valid Inference: Continuous Monitoring of A/B Tests. Operations Research, 2022 — free online.
• Jon Vaver, Jim Koehler — Measuring Ad Effectiveness Using Geo Experiments. Google, 2011 — free online.
• David Kastelman, Raghav Ramesh — Switchback Tests and Randomized Experimentation Under Network Effects at DoorDash. DoorDash Engineering, 2018 — free online.
• Kerry Rodden, Hilary Hutchinson, Xin Fu — Measuring the User Experience on a Large Scale: User-Centered Metrics for Web Applications. CHI, 2010 — free online.