Building Products in the AI & Cloud Era

← the whole book on one page

Chapter 3: Product Discovery — Learning Before Building

Every product begins as a guess about other people’s lives.

Chapter 2 chose where Encore would play. Its strategy for Verified Resale has a diagnosis, a guiding policy and a set of coherent actions, and it reads well. Yet every sentence in it rests on beliefs about how other people will behave, and nobody has yet checked those beliefs. Will an organizer who has spent years fighting touts sit down and set resale rules for a show? Will a fan believe that a ticket reissued an hour ago, under a new barcode, will open the gate? A strategy cannot answer such questions, because they are about behavior, and behavior is found out by watching. This chapter is about how a team does that watching, and how it turns what it sees into a decision before 12 teams have spent a year building. Product discovery is the work of finding out, quickly and cheaply, whether a proposed product is worth building and what form it should take, before the cost of building it is paid. The Architecture book separated the essential complexity of a problem from the accidental complexity we add to it, and Chapter 1 argued that most of a product’s essential complexity lives in the decision about what to build. Discovery is where we meet that complexity directly.

Our touchstone is one of the longest-running habits in the history of software products. From Quicken’s early days, employees of Intuit waited in computer shops for someone to buy their product, then asked to follow the buyer home and watch. The program has outlived every method that was fashionable around it, and it teaches what this chapter develops: the people who build a product should see its customers regularly, in the customers’ own setting, because what people do is better evidence than what they say they will do.

Encore stands at month zero of the bet. The bet memo that Chapter 10 examines has funded a first tranche of work, and that tranche is for discovery. The resale trio introduced in Chapter 1, Amira as product manager, Paolo as designer and Katrin as tech lead, has four months to learn whether the bet deserves the next tranche. It has a history to learn from: Partner Listings, which since Encore’s sixth year has linked sold-out event pages to Rowmate and checked nothing beyond the partner’s word; the Rowmate key leak, which showed what that trust was worth; and the grey market around the Vesper Hale tour. Over the four months the trio interviews fans, organizers and Rowmate every week, recruits six promoters as design partners, runs a hand-operated resale for four real shows, and shapes the first release as a written pitch. Along the way it finds that organizers care far more about controlling resale than about the size of their share, and that finding changes what Encore builds first.

3.1 Continuous Discovery

Following customers home

Intuit began in 1983, when Scott Cook, a former Procter & Gamble marketing manager, met Tom Proulx, a Stanford student, after posting flyers near the university library in search of a programmer. The idea came from Cook’s kitchen table, where he had watched his wife pay the household bills and dislike every minute of it. Quicken shipped in 1984. The software market of that year had no appetite for an unknown company’s first product, so Intuit sold Quicken through banks at first, starting with Bank of Hawaii. By September 1988 it was the best-selling consumer software product. Cook later described his method plainly: he had taken the Procter & Gamble playbook, with its long tradition of studying how consumers really use soap and detergent, and applied it to a new category.

The practice that came to be called “Follow Me Home” was that playbook made concrete. An Intuit employee would wait beside the Quicken shelf until someone bought a copy, then ask to come home with the buyer and watch the first installation. The rule of the visit, as Fortune reported it in 2017, was “You don’t interview the customer; you just observe.” The struggle was the data, and observers debriefed at once. Intuit’s own history says that Cook started the program in Quicken’s early days and that the company reinstated it in 1989. By 2017 Intuit reported some 10,000 hours of these visits a year. The figure is hours of observation, and secondary sources that turn it into 10,000 visits misquote it. Figure 3.1 sets these dates in order and sketches a single visit.

The best-known product to come out of this attention needs telling carefully. Intuit’s regular customer surveys asked whether people used Quicken at home or in the office, and about half said in the office. Cook first assumed they were doing personal finances at work. “It just bugged me,” he told Fortune. “Why were people answering that question wrong?” When Intuit went and looked, it found small-business owners running their companies on Quicken, because accounting software built for accountants was too hard for them. Intuit built QuickBooks, released in 1992, as, in Cook’s words, “an accounting product that didn’t seem to have any accounting.” It passed the market leader, DacEasy, within two months, at twice the price and with fewer features. A number raised the question, and watching people answered it. Neither would have been enough alone, and the rest of this chapter, together with Chapter 4, is built on that partnership.

Figure 3.1 — Follow Me Home, from Intuit’s founding to Fortune’s count of 10,000 hours a year. The program began in Quicken’s early days and was reinstated in 1989; each visit turned on one rule, to watch and not to interview.

The definition and the habit

The modern name for Intuit’s habit comes from Teresa Torres, whose Continuous Discovery Habits (2021) defines continuous discovery as “at a minimum, weekly touchpoints with customers, by the team building the product, where they conduct small research activities in pursuit of a desired outcome.” “Weekly” sets a rhythm short enough that what the team hears can still change what it does next. “By the team building the product” excludes the arrangement, common in large companies, where a research department studies customers and hands the builders a report. And “in pursuit of a desired outcome” keeps discovery from becoming a pleasant habit of chatting to customers. Torres herself calls the weekly cadence a benchmark to aspire to. What she insists on is continuity: contact with customers that never stops for long, so that no decision waits months for its evidence.

The habit is older than the name. Intuit’s visits are one ancestor. Another is Desirée Sy’s 2007 account of usability work at Alias, later part of Autodesk, which is the earliest peer-reviewed description of designers working a cycle ahead of developers, investigating the next iteration’s questions while the current one was being built. Chapter 1 introduced that arrangement as discovery and delivery running as two tracks. What continuous discovery adds is a cadence for the first track. Occasional large studies do not replace the cadence, and the cadence does not replace them either. A survey of 5,000 fans or an experiment on a checkout page can measure what weekly conversations can only suggest, and Chapter 4 does the measuring. The two kinds of evidence answer different questions, much as Intuit’s survey and Intuit’s visits did.

Civil engineering learned the same lesson about structures. A bridge is not inspected once, when it opens, and then trusted for half a century; since the collapse of the Silver Bridge over the Ohio River in 1967, American highway bridges have been inspected on a fixed cycle, because traffic, weather and corrosion change a structure in ways no design calculation foresees. Customers change in the same way, as competitors, prices and habits move around them, and a team that interviewed its customers once, at the start of a project, is trusting an inspection that has expired. The analogy breaks at an instructive point. An inspector’s gauges report what the structure is doing now. A customer asked about their own future behavior answers sincerely, and often wrongly, and so interviewing customers needs a technique of its own.

Interviewing well

The central rule of discovery interviewing is to ask about the past. “Would you use a feature that let you resell your ticket safely?” invites a fan to imagine a better version of themselves, one who plans ahead, reads terms and acts on good intentions, and the fan will answer for that imagined person. “Tell me about the last time you couldn’t go to a show you had a ticket for” invites a story, and a story is full of facts the teller did not choose to emphasize: when they realized, who they asked, what they tried first, what they gave up on, how the money moved. Torres builds her interviews around that prompt, and the method has deep roots. In 1954 the psychologist John Flanagan published “The Critical Incident Technique”, which collected specific, recent episodes of behavior instead of general opinions, a method that grew out of aviation psychology and has been used widely in research ever since. Hugh Beyer and Karen Holtzblatt’s Contextual Design (1998) took the same principle into the customer’s workplace, where the researcher watches real work as an apprentice watches a master and asks questions only about what is happening in front of them. Follow Me Home is contextual inquiry with the questions removed.

Chapter 2 taught Clayton Christensen’s switch interviews, which reconstruct the timeline of a real purchase and the forces of push, pull, anxiety and habit acting on it. Those forces need no reteaching here. What this section adds is the discipline that turns such interviews into a weekly practice. Interviews are short, about 30 minutes, and each is captured the same day on a single page. Torres calls the page an interview snapshot: a one-page summary of one interview, holding a memorable quote, the story as told, the opportunities it revealed and any follow-up. The snapshot matters more than it looks, because it fixes what the interviewer heard before the next interview overwrites it, and it gives the rest of the team something to read in two minutes. Here is one of the resale trio’s, from the sixth week:

Interview snapshot — fan who resold, F-17, week 6 (anonymized)
Quote          "I sent the screenshot and then just prayed."
Story          Couldn't attend an arena show bought from another ticketer last year.
               Sold to a stranger from a fan group at face value; paid by bank
               transfer. The buyer got in, but she checked her phone all evening.
Opportunities  Fear of being blamed if the ticket fails at the door.
               No safe way to be paid by a stranger.
Insight        Trust in the buyer mattered more to her than the price.
Follow-up      Did the event's own resale rules ever come up? (No; she didn't know any.)

Two cautions come with interviewing fans about resale. The first is that some stories describe resale that was unlawful where it happened, since several EU states restrict resale above face value. The snapshot therefore anonymizes the person and leaves out the place, because the team needs the behavior and has no use for evidence of an offence. The second is that a team that slips back into hypothetical questions will collect a great deal of enthusiasm and very little information; Rob Fitzpatrick’s short book The Mom Test (2013) is a practical corrective.

Recruiting on autopilot

Weekly interviews stay weekly only if finding people to talk to costs the team nothing each week. Torres’s advice is to automate recruitment: ask customers while they use the product, ask through colleagues who already talk to customers, and keep a standing pool of volunteers. Encore uses two channels. A short invitation on the post-purchase screen asks fans whether they would talk to the team for half an hour, and for organizers, the organizer success team, the account managers who speak to promoters every week, asks at the end of routine calls. Without such channels, interviews happen only when the team has spare time, which is rarely.

Automated recruitment has blind spots. A prompt after purchase reaches only buyers, yet the fans Chapter 2 called non-consumers, those who could not go and simply absorbed the loss, are exactly the people a resale product most needs to understand. The trio finds them among fans who asked for refunds the policy could not grant and fans who contacted Support about a ticket they could not use. Panels also tire, so the trio rotates participants and limits how often anyone is asked, and consent comes before any recording, as Section 3.5 explains. The shared panel and findings repository that serve all 40 teams belong to Chapter 6; this section concerns only the habit of one team.

Before the first interview, the trio also looks at what Encore already knows, since Torres recommends sizing opportunities with existing data. The support assistant’s logs show how many conversations concern tickets a fan cannot use, and the fan app’s search logs record how often people type “resell” into a box that has nothing to offer them. None of this explains anything, but it says where to look, and it costs an afternoon.

The opportunity solution tree

Interviews produce stories, and stories pile up. Torres’s answer is the opportunity solution tree: a diagram that starts from one desired outcome, branches into the opportunities that could move it, branches again into candidate solutions for each opportunity, and ends in the assumption tests that would show whether each solution works. An opportunity, in Torres’s sense, is a customer need, pain point or desire, stated from the customer’s side, such as “I’m afraid I’ll be blamed if the ticket doesn’t work.” A solution is something the team might build. Keeping the two levels apart is most of the discipline, since teams naturally jump from a story to a feature and lose sight of the need in between.

Torres first described the tree in 2016, and it deserves a precise claim. No controlled study shows that teams using it build better products. Its value is that it makes a team’s reasoning visible: a leader can see which outcome the team is pursuing, which needs it has heard, which it chose and what it is testing this week. In an organization of 40 teams, most of what leaders otherwise learn about a team’s thinking arrives as a roadmap of features with the reasons removed. The root of the resale trio’s tree is the product outcome Chapter 2 set: verified resales completed per eligible event. Figure 3.2 shows the upper branches as they stood at the end of the second month.

Figure 3.2 — The upper branches of the resale trio’s opportunity solution tree at the end of month two. The root outcome is verified resales completed per eligible event; each opportunity carries whose need it is and how many of the 22 organizer or 31 fan interviews raised it, and the organizers’ wish to control resale, raised unprompted, stands out.

Comparing opportunities

The tree’s second discipline is to compare. Torres asks teams to set opportunities side by side and choose among them, judging each by how many customers it touches, how much it matters to them, and how well it fits the company’s strategy. Then it generates several solutions for the chosen opportunity and compares those too. The argument comes from decision research. Chip and Dan Heath’s Decisive (2013) reports Paul Nutt’s studies of organizational decisions, which found that decisions framed as “whether or not” to do a single thing failed considerably more often than decisions that weighed two or more alternatives. Most roadmaps are lists of “whether or not” decisions, and the tree is a device for turning them into choices.

When the trio compared its organizer opportunities, the evidence column did the arguing. Control over resale, meaning when it opens, how many tickets one fan may list or buy, and whether it happens at all, came up unprompted in almost every organizer interview, while the organizer’s share came up in about a third, usually late and as a matter of principle more than money. One promoter put the hierarchy in a sentence the trio wrote on its wall: the fee would be pleasant, but the rules were the reason to say yes. The tree recorded control as a single opportunity, without separating its kinds: control of price, of timing and of quantity were all written down as “control”, a simplification whose cost Chapter 10 counts at month 18. Even so, the finding changed the plan. The rules console moved into the first release, ahead of any work on fee reporting, and the organizer’s share became something Encore would fix and explain, which Chapter 5 does, instead of something each organizer would negotiate.

Recap. Continuous discovery is a weekly habit of contact with customers, kept by the team that builds the product and aimed at an outcome. Interviews ask about specific past episodes and are captured the same day as one-page snapshots, and automated recruitment lets the habit survive a busy week. The opportunity solution tree keeps needs and solutions apart and makes the team’s reasoning visible, and comparing opportunities, then solutions, produces better choices than deciding on one idea at a time.

Exercise 3.1. Hold one story-based interview this week with a customer of your product, beginning “Tell me about the last time…”, and write its snapshot within the hour. Then add one opportunity from it to a tree whose root is your team’s current outcome, stated in the customer’s words.

3.2 Assumptions and Cheap Tests

Assumptions inside an idea

An idea that has survived the tree is still a bundle of beliefs, and some of those beliefs are more dangerous than others. Torres asks teams to unpack each idea into its assumptions and sort them into five types: desirability (do people want it?), viability (should the business do it?), feasibility (can we build it?), usability (can people use it?) and ethics (should we do it at all?). The five types line up with the four product risks and the ethical question of Chapter 1, Section 1.3: the risks are the dangers, and the assumptions are the particular beliefs that, if false, would make each danger real. The practical move is to rank the assumptions by how much the idea depends on each and how little evidence the team has for it, and to test the riskiest first.

That ranking is the core of cheap testing. A team that tests the whole idea by building it learns only whether the bundle worked, and if it fails, cannot tell which belief was false. Whole ideas do get tested eventually, through live-data prototypes and the controlled experiments of Chapter 4, but last, when the remaining doubts are cheap ones. Figure 3.3 shows the resale trio’s map for the idea at the heart of Verified Resale.

Figure 3.3 — The assumption map for Verified Resale’s central idea. Two assumptions ranked riskiest, that buyers will trust a reissued ticket and that organizers will set rules and keep resale switched on, because the idea depends on them entirely and the only evidence for either was interviews; the pilot and the rules-screen prototype tested them first.

The MVP as an experiment

Eric Ries gave the most widely used name to the cheapest kind of test. In The Lean Startup (2011) the minimum viable product (MVP) is the version of a new product that lets a team collect the most validated learning about customers with the least effort: an experiment that happens to look like a product. Many teams now use the phrase for a small first release, which loses the point, since the cheapest way to learn whether an assumption is true is often something that could never be shipped at all.

Ries describes two such variants. In a concierge MVP, the team delivers the service by hand to a few customers who know they are being served by people, as Food on the Table, an Austin meal-planning company founded in 2009, did when it began by serving its first customers in person, with the founders preparing their meal plans and shopping lists by hand. In a Wizard-of-Oz MVP, customers use what looks like a working product while people behind the scenes do the work the software will later do. Ries’s example is Aardvark, a question-answering service bought by Google in 2010, which for months had people performing parts of its back end that software was later built to do. The technique is much older than the book. In 1983 John Gould, John Conti and Todd Hovanyecz tested a “listening typewriter”, a speech-recognition system that did not yet exist, by having a hidden typist transcribe what users dictated, and in 1984 J. F. Kelley named the approach the “OZ paradigm” in a paper on natural-language interfaces. The name honors the man behind the curtain, and only the startup vocabulary is recent.

Theater makes the analogy obvious. The audience sees a storm on the heath, and the crew in the wings produces it with ropes, lights and timing; the illusion must hold from the audience’s seat, so everything the audience can see must be real and everything behind it may be improvised. The analogy breaks at consent. A theater audience has bought tickets to an illusion and knows it, while a customer in a Wizard-of-Oz test usually does not know that a person is doing the software’s job. That is acceptable only when the customer receives exactly what was promised, so that the hidden labor changes how the service is produced and nothing about what the customer gets. Paul Graham’s essay “Do Things That Don’t Scale” (2013) describes Stripe doing this in its early days, setting up merchant accounts by hand behind an interface that looked automatic. Ries’s measure of a team is how quickly it turns his build-measure-learn loop, and a Wizard of Oz turns it fast because it skips most of the building.

Prototypes of every fidelity

Not every assumption needs a working service, even a hand-operated one. Marty Cagan’s INSPIRED (2017) distinguishes four kinds of prototype, each suited to a different risk. A feasibility prototype is throwaway code that answers an engineering question. A user prototype is a simulation, from paper sketches to a clickable mock-up, that shows people what the product would be like without doing anything real. A live-data prototype is a limited implementation that handles real data and real traffic, enough to see how people actually behave. A hybrid prototype combines elements, and a Wizard-of-Oz test is a hybrid: a real front end with simulated machinery behind it.

Chapter 1 told how Katrin’s feasibility prototype answered whether the seller’s old ticket could be made to stop working. No phone could be trusted to remove it, but a gate could refuse it, if every venue’s scanners held a list of revoked barcodes even when offline. Because that answer pointed to a known mechanism, owned by Venue Entry, feasibility does not top the risk list in Figure 3.3. The more instructive prototype in this period was the designer’s. The trio built a clickable rules screen for organizers, with a resale window, per-fan limits and an off switch, and took it to all six design-partner promoters within two weeks. The screen did nothing, which was the point: it cost three days, and it answered whether organizers could express their rules without help. Mostly they could. Two of the six read “resale window” as the dates during which a buyer may attend, and every promoter wanted a plain-language sentence summarizing what they had set, which foreshadows the rule assistant of Chapter 6. Usability testing as a monthly practice is also Chapter 6’s subject.

Encore’s Wizard of Oz

The riskiest assumption on the fans’ side of Figure 3.3, that buyers would trust a reissued ticket, needed real fans, real money and real gates, and so did its companion, that fans would list at all. They did not need a Resale service, which would have taken most of a year to build. So the trio ran a Wizard of Oz.

Two design-partner promoters offered four shows: two dates of a mid-size tour at a 3,000-capacity hall in Lyon, and two at a hall of similar size in Antwerp, about 11,200 tickets in all. Everything a fan touched was real. Ticket holders for those shows saw “Resell this ticket” in My Tickets, and buyers saw verified resale listings on the sold-out event pages, inside Encore’s own app and website. No partner was involved, since a partner integration would have added months of work to a six-week test. Buyers passed Bot Screening and the per-buyer limits as they would in a primary sale, and paid the face value plus a 10% fee, shown in the all-in price from the first screen. Their payment was authorized at purchase and captured only after the reissue had been completed, so if Support could not finish a resale within 12 hours, the authorization lapsed and the listing returned. Figure 3.4 shows both sides of the curtain.

Figure 3.4 — The Wizard of Oz pilot. Every screen, payment and gate in front of the curtain was real; behind it, Support agents did the checks and the reissue by hand, and no payment was captured until the reissue was confirmed.

Behind the screens, a rota of four Support agents was the machinery. For each sale, an agent checked that the seller had bought the ticket through the Sale Gate and within the per-fan limit, checked the promoter’s rules (resale closing 24 hours before doors, two tickets per fan), used Support’s long-standing lost-ticket replacement screen to cancel the seller’s ticket and issue a new one to the buyer, and confirmed that the old ticket appeared on the show’s revocation list. Sellers were paid by hand after the show. The agents did by hand what the barcode core could not yet do by machine: written in Encore’s first year, for a world in which a ticket had one holder for life, it had no notion of transfer, and Lena, its last remaining author, is the only person who fully understands it. The Software book tells how that code was made safe to change (Sections 2.4 and 11.2); for a discovery test, a careful person with an old admin screen was enough. The trio wrote its decision rule in the first week and had the bet’s sponsor sign it:

Pilot pre-registration — Verified Resale, Wizard of Oz (signed before day one)
Shows        Four: two dates in Lyon, two in Antwerp; about 11,200 tickets.
Measures     Resales completed; median Support minutes per completed resale;
             completed resales that broke an organizer rule; entry refusals.
Stop at once Any fan holding a reissued ticket is refused entry.
             Any payment is captured for a resale that was not completed.
Stop after a show   Median Support time above 30 min at any show after the first.
Redesign     More than 5% of completed resales broke an organizer rule.
Demand       Read qualitatively: four shows cannot measure it.
Decision     The trio and the sponsor, within one week of the last show.

The demand line deserves a word. Four shows produce a few hundred resale attempts, enough to measure Support time, rule breaches and gate results, which are properties of individual resales. They cannot estimate how many fans across Encore’s catalog would list, because demand varies with artist, venue, price and season, and four shows sample almost none of that variation. Saying so in advance protected everyone from reading more into a good result than it could bear. Figure 3.5 shows what the four shows produced.

Figure 3.5 — Results of the hand-run pilot against the rule written before it began. These numbers feed the business case of Chapter 5.

Two results mattered more than the rest. Agents caught 37 attempts to list outside the promoter’s rules and missed six, all cases of one fan listing tickets bought across two orders, which their checklist did not cover. Six in 191 is under the threshold, but the finding was plain: people checking rules by hand miss things, and a product that promises organizers their rules will hold must enforce them in software, at the point of listing, every time. That became one of the five fairness invariants that the Software book puts under test. The second was the three old passes refused at the gate: Apple Wallet passes, which do not rotate, forwarded by sellers to friends before the sale. The revocation lists held, and the fans involved were confused, a design problem that Chapter 6 takes up.

Demand and value tests

Because the pilot could not measure demand, the trio added a test that could suggest it across many events. Cagan describes a family of demand tests, of which the best known is the fake door: a button for a feature that does not yet exist, which explains that the feature is coming and counts the visit. Its partner is the qualitative value test, in which a person is shown a prototype and the team learns, by watching and asking, whether they find it valuable. The first asks how many people reach for something, the second why.

Fake doors raise an ethical question, which Cagan treats as one, and on a ticketing site it is sharper still, because a fan who reaches for a ticket is reaching for money and a seat. EU consumer law has a view. Annex I of the Unfair Commercial Practices Directive (2005/29/EC) lists practices that are unfair in all circumstances, and point 5 is “bait advertising”: inviting people to buy a product at a stated price without disclosing that the trader has reason to think it cannot supply it at that price. A fake door that showed a resale ticket for a particular seat at a particular price would sit uncomfortably close to that line. Encore’s rule is therefore strict. A fake door never shows a specific seat or price, never takes a payment and never holds a seat, and the screen behind it says plainly that the feature does not exist yet. The trio’s fake door was a “Resell this ticket” entry in My Tickets for events outside the pilot, leading to a page that said resale was being tested and offered to notify the fan when it arrived. Over three weeks, about 3,300 fans across 280 events tapped it, which shows that sellers look for such a feature where they expect it, and says nothing about whether they would complete a resale. Chapter 4 sets out the ethics of experimenting on customers, including Encore’s absolute rule against individual price experiments on fans, which is why no fake door ever mentions a price.

Pivot or persevere

Ries closes his loop with a decision. After each round of testing, a team should choose deliberately whether to persevere with its strategy or pivot, changing one fundamental part of it while keeping what it has learned. He wraps the decision in what he calls innovation accounting: establish a baseline with the MVP, try to move it, and judge progress against that baseline instead of against vanity metrics such as sign-ups or page views, which rise whether or not the product is working. The evidence Ries offers is a set of case histories, IMVU’s among them, which is good practitioner evidence and no more.

The strongest part of the idea has much older support. Writing down the decision criteria before running a test is the logic of pre-registration in experimental science, where researchers state their hypothesis and analysis in advance so that the results cannot quietly reshape the question. The reason it matters is human. A team that has spent six weeks on a pilot wants it to have worked, and after the fact almost any result can be read as encouraging. A rule signed beforehand does not care what the team wants, and it also prevents the opposite failure, a promising idea dropped because one number disappointed someone who had never agreed what disappointing meant.

Encore’s decision, taken within the week as the pre-registration required, was to persevere, with one change of course. Fans listed, reissued tickets opened gates and the Support cost fell quickly, while enforcing organizers’ rules by hand proved inadequate, so rule enforcement moved onto the list of things the first release must do in software. The form of the pilot’s stopping rule, a state and a date written in advance, carries forward into the kill criteria of the bet memo, which Chapter 10 examines.

Recap. An idea is a bundle of assumptions of five types, and testing the riskiest first lets a team learn in days what building would teach in months. An MVP is an experiment, and its concierge and Wizard-of-Oz forms, the second of which is almost 30 years older than Ries’s book, let real customers meet a service whose machinery is still people. Each kind of prototype answers a different risk, fake doors must never show a seat or a price, or take money, and the decision to persevere or pivot is only as honest as the criteria written down before the test.

Exercise 3.2. Take one item from your roadmap and list every assumption inside it, labelled with Torres’s five types. Mark the one whose failure would hurt most and for which you have the least evidence, and design a test for it that takes less than a week. Write the decision rule before you run it.

3.3 Discovery in B2B and with Partners

Customers who sign contracts

A fan who buys is the fan who uses. Organizers are businesses, so discovery with them is business-to-business (B2B) discovery, and a business buys differently. A promoter has a commercial director who signs contracts and a box-office manager who spends every day in Encore’s organizer console, and a product that pleases one can fail with the other. Marketing scholars mapped this half a century ago. In 1972 Frederick Webster and Yoram Wind described the buying center, the set of people involved in an organization’s purchase: users, influencers, buyers, deciders and gatekeepers. Product teams keep rediscovering the idea, usually after a feature that users loved failed to renew a contract.

Medicine offers a close analogy. A prescription drug has a patient who takes it, a physician who chooses it and a payer that decides whether it will be paid for, and a drug can work beautifully for patients and still fail because payers will not fund it. The analogy breaks in a way that matters for Encore: in medicine the three parties rarely sit in the same building, whereas in a promoter’s office the box-office manager and the commercial director may share a coffee machine and still not share a view of what resale is for. One thinks about revenue and the artist relationship, the other about the angry emails the morning after a sale.

So the trio’s interviews with promoters were arranged in pairs, one with the person who would use the rules console and one with the person who would sign the contract amendment that enables resale. The two conversations often disagreed, and the disagreements were among the most useful things the trio learned. The Architecture book’s Section 14.5 showed the same pattern with an enterprise tenant’s checklist of single sign-on, customer-managed keys and contractual service levels: none of it came from the staff who use the product daily, and all of it arrived through the people who sign.

Design partners

Cagan’s answer to discovery in a business market is the customer discovery program: a team builds a product together with a small set of hand-picked customers, until those customers are running it in production and willing to say so publicly. The target in INSPIRED is six such customers, all from the same target market, and Cagan is specific about why. Two or three customers from each of two or three markets, he warns, will not give the team focus, because each market’s needs pull the product a different way and no market ends up properly served. A reference customer, in his sense, is a real, paying customer, using the product in production, who is willing to tell others that it works for them.

The trio’s first plan had three organizers and two resale partners, precisely the mixture Cagan warns against. The plan Encore adopted has six design partners, all mid-size festival and tour promoters: three in France, two in Belgium and one in the Netherlands, each running between 40 and 200 shows a year on Encore. One of the French three, a promoter of arena tours, is openly hostile to resale and said so in the first meeting: in his view it rewards people who never meant to come. The trio kept him deliberately, because a design partner who wants the product to fail will find the places where it can. Rowmate takes part in a separate, smaller program, for reasons the last part of this section gives. The same six promoters reappear in Chapter 5’s conversations about fees, which keeps the evidence connected across the book.

Design partnership asks something of both sides. The promoters give a monthly working session, access to their box-office staff and a willingness to try unfinished things on real shows; Encore gives influence over the product, early access and hands-on help. That help is itself a discovery method. Graham’s essay describes Stripe’s founders setting new users up on the spot, a habit Y Combinator came to call the “Collison installation,” and Airbnb’s founders going door to door in New York to improve their hosts’ listings. Encore’s account managers sat with each pilot promoter’s box-office manager while the rules were set, and the trio’s designer sat with them, watching where the conversation stalled.

Sales-led requests

Chapter 1 stated the principle that commitments made by sales become inputs to discovery. The mechanism is best seen in a single case. Early in the period, Encore’s organizer sales team relayed a request from several promoters: “Can resale be turned off per show?” Instead of adding an off switch to the backlog, the trio asked the promoters why. Some artists’ contracts forbid resale for intimate shows; one promoter had promised a fan club that its presale tickets would not reappear at any price; another wanted resale available only in the final week, when it would fill seats that would otherwise be empty. The underlying opportunity was “I must be able to honor what I have promised the artist and the fans,” and an off switch was only one of its solutions. Resale windows and per-fan limits served it too, often better.

The distinction has a practical edge in Encore’s markets. In Ireland, section 19 of the Sale of Tickets Act 2021 voids contract terms that prevent a ticket from being resold at or below face value, so an organizer there may not switch resale off entirely. A team that understood the opportunity could offer Irish organizers what the law allows, a lower cap, a window and per-fan limits, and still serve the promise behind the request. I spent a decade maintaining a widely used open-source library, and the most durable lesson of that work was that an issue titled “please add option X” almost always describes the reporter’s workaround for some problem they have not mentioned. Cagan’s Transformed (2024) describes how product and sales teams can work this way together, on the evidence of practitioners’ experience across many companies.

The loudest customer

Business customers differ greatly in size, and size brings volume. Encore’s largest tenant, the venue chain that runs in its own silo as venue-chain-a, has an account team, quarterly business reviews and a long list of requests, and early in the period it asked that resale for its venues run only through its own box office, under its own brand. It is a reasonable request, and it describes a different product from the one the strategy chose.

Christensen’s The Innovator’s Dilemma (1997) is the best-documented account of where that road leads. Studying the disk-drive industry, he found that established manufacturers listened carefully to their largest existing customers, who wanted more capacity in the existing form, and so missed each new, smaller drive architecture until entrants using it had taken the market. The lesson for discovery is modest and practical. The opportunity solution tree should record, for each opportunity, how many interviews and which segments it came from, so that a need heard loudly from one large customer can be seen beside a need heard quietly from many. On the trio’s tree, venue-chain-a’s request sits as an opportunity with one source. Verified Resale is offered to organizers on every tier, and the venue chain gets the same rules console as everyone else.

Partners as a side

Rowmate is easy to think of as a channel, a pipe through which fans reach resale listings. Chapter 2’s treatment of multi-sided markets suggests a better view. In Verified Resale, partners are one of the sides: marketplaces where fans list tickets they bought through the Gate, with their own economics, their own customers and their own risks. They are customers of Encore’s platform as well as integrators of its interface, and discovery with them has to cover the business as well as the technology.

The trio’s six conversations with Rowmate in this period concerned three things. The first was money: four points of the buyer’s 10% fee on every sale made through it, against the margins it had earned on uncapped listings. The second was fraud. Under Partner Listings, Rowmate had absorbed chargebacks whenever a ticket sold through it turned out to have been sold twice, so reissue, a ticket that exists exactly once, was a benefit it could price. The third was support: who answers the fan when a reissued ticket has not arrived. These questions had to be settled before anyone discussed endpoints, and the key leak, caused by Rowmate’s own contractor, hung over all of them. The design of the partner interface belongs to Chapter 9, and the partner program, with its agreements, certification and support, to the Software book’s Section 8.3.

Recap. Business customers buy through a buying center, so discovery must reach the people who sign as well as the people who use the product. Six design partners from one segment, including one who opposes the idea, give a product focus and its first references, while partners on another side of the market need a program of their own. Sales requests are stories not yet heard, and the opportunity beneath them is what should be built for. Recording the source and segment of each opportunity keeps the loudest customer from setting the strategy.

Exercise 3.3. Take the last three requests from your largest customer. For each, write the problem you believe lies beneath it, then check your interview notes, support data or account records for any other customer with the same problem. Mark which requests turn out to be single-source opportunities.

3.4 Shaping the Work

Appetite

Discovery tells a team which problem is worth solving, but not how much of the company’s time the solution deserves, and asking engineers for an estimate gets that question backwards: an estimate starts from a solution, and the solution grows to fill whatever answer comes back. Ryan Singer’s Shape Up (2019), which documents how Basecamp plans its work, inverts the question. Appetite is “the amount of time we want to spend on a project, as opposed to an estimate,” and it is set first, by people with the authority to cut scope. The team then shapes a solution that fits the time. Singer’s phrase for the principle is “fixed time, variable scope,” and Basecamp works in two sizes: a small batch, one designer and one or two programmers for a week or two, and a big batch, the same team for six weeks.

Newspaper editors have always worked this way. An editor does not ask a reporter how many words the story needs; she assigns 800 words and a deadline, and the reporter decides which 800 words carry the story best. The analogy breaks at the point of cutting. A story cut from 1,200 words to 800 is still a story, since prose degrades gracefully. A product cut carelessly can lose a piece the others depend on, a ticket listed with no way to be reissued, and so cutting product scope needs the shaping work described next.

Encore’s appetite needs a precise statement, because it is easy to misread. The trio set an appetite of six weeks for two small teams, lent from Orders & Payments and Partner APIs, and that appetite covered the discovery and prototype stage only: the hand-run pilot, the rules-screen prototype and the fake door. It did not cover building Verified Resale. The full product touches 12 teams, and a six-week appetite for that would be a fiction, since Shape Up’s unit is a single small team and its six weeks are sized for one. The bet memo that Chapter 10 examines states its own appetite at a different scale, 12 months from the memo to the first ticket scanned at a gate, and the build is funded in tranches released against learning, as the Software book’s Section 10.4 describes: discovery, then the first partner live, then all partners. At that scale appetite survives as an idea, a fixed budget against which a forecast is compared and scope is cut (the Software book, Section 10.3).

Shaping and the pitch

Between a problem and a team’s work, Singer places shaping: a short period in which a few senior people work out the solution at the right level of abstraction, concrete enough that the team knows what to build and rough enough that it can still make the detailed decisions. Two techniques keep the level right. A breadboard, borrowed from electronics, shows places, the things a user can act on in each, and the connections between them, with no visual design at all. A fat-marker sketch is a drawing made with a pen so thick that no fine detail is possible. Both are ways of deciding the elements of a solution without deciding its appearance.

Shaping ends in a pitch, a short document with five parts: the problem, the appetite, the solution, the rabbit holes (the details that could swallow the time and must be settled in advance), and the no-gos (what the work deliberately excludes). The evidence for the format is Basecamp’s own published practice and its wide adoption by other teams, which is practitioner evidence and should be weighed as such. The format’s strength is that it makes the boundaries of the work explicit before the work starts, when they are cheapest to argue about. Here is the trio’s pitch for the discovery and prototype stage, as presented to the bet’s sponsor:

Pitch — Verified Resale, discovery and prototype stage
Problem      Resale of Encore tickets happens off Encore, unchecked. We don't know
             whether organizers will set rules or fans will trust a reissued ticket.
Appetite     Six weeks, two small teams (lent from Orders & Payments, Partner APIs).
Solution     Real listing and purchase screens for four pilot shows; reissue,
             revocation and rule checks done by Support by hand. A clickable
             rules screen tested with the six design-partner promoters.
Rabbit holes Automating reissue in the barcode core. Updating wallet passes.
             Seller payouts: pay by hand after each show.
No-gos       Dynamic or organizer-set resale prices. Any partner integration.
             Any change to primary checkout.

The no-gos carry as much weight as the solution. Dynamic resale prices were excluded because the strategy caps resale at face value, and every hour spent on pricing would have been an hour taken from the question of trust. Written down once, before the work began, each no-go spared the team from arguing the same point again in every planning meeting.

Betting at 40 teams

Shape Up also describes how Basecamp decides which pitches to fund. A small betting table, at Basecamp the chief executive, the chief technology officer, a senior programmer and a product strategist, meets between six-week cycles and chooses which pitches to bet on. Between cycles there is a two-week cool-down for fixing and exploring. And a circuit breaker ends any project that does not ship within its cycle by default, instead of extending it by default, so that no bet outgrows its appetite without a fresh decision.

It is worth being exact about where this evidence comes from. The book’s appendix, “Adjust to Your Size,” describes Basecamp as about 50 people in the whole company, with roughly a dozen in the product team. It does not describe a company of 40 product teams with dependencies running between them, and no well-documented public account of Shape Up’s full calendar at Encore’s size was found in preparing this book. Encore therefore borrows the mechanisms that travel, appetite, the pitch and the circuit breaker, and keeps its own quarterly planning cycle, which the Software book’s Section 10.2 compares with the alternatives.

Large companies have their own long-running way of shaping a bet. Amazon’s practice of “working backwards,” described by Colin Bryar and Bill Carr in Working Backwards (2021), begins a proposal with a press release and a list of frequently asked questions (the PR/FAQ), written as though the product had already launched, so that the customer benefit is stated before anything is built. It is a closer precedent for a company of Encore’s size, and Encore’s bet memo, examined in Chapter 10, borrows from it.

The story map

A pitch bounds a piece of work, while a release needs a picture of the whole journey. Jeff Patton’s user story map, first described in 2005 and set out fully in User Story Mapping (2014), lays out the activities of a user’s journey from left to right, in the order the user does them, as a backbone, with the details of each activity listed beneath it, most essential first. It reads as a narrative across and a priority list down, and it keeps the whole journey in view.

Verified Resale’s map is unusual in having three sides on one backbone, since the journey passes from organizer to fan to fan to partner and back to the organizer’s books. Figure 3.6 shows it, with the release slices drawn as rows.

Figure 3.6 — The story map for Verified Resale, with its backbone of six activities and three release slices. Each slice is a complete journey, thinner above and fuller below; the first, above the colored line, is what the walking skeleton must carry.

Slicing releases

Patton’s map is drawn to be cut horizontally. A line across it selects a minimum viable release: a release in which every step of the journey exists, each in its simplest useful form. Building each activity completely before starting the next produces months of work in which nothing can be used, because the journey does not yet connect.

The engineering counterpart is the walking skeleton, a term from Alistair Cockburn’s Crystal Clear (2004): the thinnest slice of real function that can be built, deployed and tested from end to end. Patton adopts it, and the credit belongs to Cockburn. Verified Resale’s first slice in Figure 3.6 is what its walking skeleton must carry: one organizer, with a resale window and per-fan limits and the face-value cap; fans listing only through Rowmate; buyers screened as in a primary sale; reissue and revocation in software; entry at one venue in Lyon; and the fee split recorded in the ledger. Everything in it is thin, and nothing is missing. The launch slice that Chapter 10 follows widens it to three organizers across France. How the skeleton is built belongs to the Software book (Sections 3.2 and 11.3), and how it is released safely, behind flags and in rings, to the Resilience book (Sections 4.3 and 4.4).

Stories are for telling

Patton’s book carries a warning that is easy to agree with and hard to practice: shared documents aren’t shared understanding. A story on a card or in a tracking tool is a reminder of a conversation, and cannot carry on its own the understanding the conversation produced. Ron Jeffries said the same in 2001 with three words from Extreme Programming: card, conversation, confirmation. The card is a token, the conversation is where the detail lives, and the confirmation, the acceptance tests, records what the two sides agreed.

The warning matters most where context must cross a team boundary. Within the trio, understanding passes easily; between teams it does not. When the first slice reached planning, its central story read, in effect, “when a resale completes, the seller’s ticket is revoked and a new one issued to the buyer.” To the Partner APIs team that was a single call. To the Inventory team, which owns the eight-year-old barcode core, it was a change to the oldest and least understood code Encore runs, with a question of ownership attached: should Resale reissue tickets itself, or ask Inventory to do it? It took two working sessions, with the trio’s tech lead present to say what the product needed, before the story meant the same thing to both teams. The ownership question was settled by a written request for comments and a decision record, which the Software book’s Section 9.5 describes.

Recap. Appetite fixes the time a problem deserves and lets scope vary, and Encore’s six weeks for two small teams applied to discovery and prototyping only, with the build funded in tranches. A pitch states the problem, appetite, solution, rabbit holes and no-gos before work begins, and Basecamp’s betting mechanisms travel to a large company better than its calendar does. A story map keeps the whole journey in view, a minimum viable release cuts across it, and its stories are prompts for conversations that must still take place, especially between teams.

Exercise 3.4. Write a one-page pitch for something on your roadmap with all five parts. Give it an appetite that someone with authority would sign, and at least two no-gos that a reasonable colleague might want included. Then show it to that colleague and record which no-go they argued with.

3.5 Discovery with AI

Prototypes in hours

Of everything in this chapter, prototyping has been changed most by generative AI. A designer or engineer can describe a screen to a coding assistant and have a working, clickable version within hours, where a user prototype once took days and a live-data prototype weeks. Addy Osmani’s Beyond Vibe Coding (2025) gives a practitioner’s account of AI-driven prototyping, and in April 2025 Shopify’s chief executive, Tobi Lütke, published a memo to staff which said that “the prototype phase of any GSD project should be dominated by AI exploration,” GSD being Shopify’s internal process for planning and running projects. Neither measures whether teams that prototype this way learn more or decide better, so this is emerging practice with practitioner evidence behind it. The broader evidence on AI and software work is in the Software book’s Section 5.1.

The proven part is older than the tools. A prototype exists to answer a question, and once the question is answered it should be thrown away; Cagan makes the point for every kind of prototype, and Cockburn draws the same line between a throwaway spike and a skeleton meant to grow. Cheap generation makes the line more important, because a prototype built in an afternoon looks finished and invites someone to suggest shipping it, and code written to learn something, with no tests and no owner, is exactly the code that should never reach production. The Software book’s Section 5.2 sets out where throwaway code belongs, and the Architecture book’s Section 15.4 explains what cheap code does to systems it drifts into. Encore’s rule is that prototypes live in a separate repository, are deleted when their question is answered, and are never promoted.

Synthesis with a trail

After a few months of weekly interviews, a trio has dozens of snapshots and hours of transcripts. Large language models (LLMs) can help find the patterns, and here the evidence is young but real. Ziang Xiao and colleagues reported in 2023 that GPT-3, given an expert-written codebook, coded qualitative data with fair to substantial agreement with expert human coders. The limit matters: that is deductive coding, applying categories the researchers already defined, and asking a model to discover the themes on its own is a harder task with weaker evidence.

The safeguard comes from qualitative research’s own standards. Yvonna Lincoln and Egon Guba’s Naturalistic Inquiry (1985) made the audit trail, a traceable path from each conclusion back to the raw data that supports it, one of the tests of whether qualitative findings can be trusted. A model’s summary has no such trail unless the team builds one. Encore’s rule is that every opportunity on the tree links to the snapshots it came from, every quotation that reaches the tree or a slide is checked against the transcript before it is used, and a model’s clustering is treated as a suggestion for the trio to examine. The checking matters, because a summary can merge two people’s stories into one fluent paragraph that neither of them told.

Synthetic users

A more ambitious idea is to skip the interviews and ask a model to answer as a customer would. The research here is clearer, and mostly cautionary. Lisa Argyle and colleagues showed in 2023 that GPT-3, conditioned on the backstories of real survey respondents, reproduced some aggregate attitudes of American subgroups with what they called “algorithmic fidelity.” A year later, James Bisbee and colleagues tested the idea against the American National Election Studies and found that while the synthetic averages tracked the real ones, the responses were far less varied than real people’s, relationships between variables often differed significantly, results shifted with small changes in prompt wording, and the same prompt gave significantly different answers three months apart. The Nielsen Norman Group reran three of its own earlier studies with synthetic users in 2024 and found answers that were shallow and over-agreeable; synthetic learners claimed to have finished online courses that real learners had abandoned.

Aviation offers a fair comparison. A flight simulator is an extraordinary tool for rehearsal, but it is built from data recorded in real aircraft and can say nothing about an aircraft that has never flown. A synthetic user is a simulator built from the average of what people have written, and it cannot tell a team how its own customers will respond to something new, which is the question discovery exists to answer. The comparison even flatters synthetic users, since a flight simulator is validated against the aircraft it models and a synthetic customer has no equivalent check. Encore accepts synthetic users for one purpose, rehearsing an interview guide to find confusing questions before a real person hears them, and never as evidence of what customers need or will do.

Consent and data

Recording and transcribing interviews, and passing the transcripts through a model, are processing of personal data, and the General Data Protection Regulation (GDPR) applies to all of it. The team needs a lawful basis under Article 6, which for research interviews is usually consent meeting the conditions of Article 7. It must tell participants at the outset what will be recorded, why, by whom and for how long, as Article 13 requires. A transcription or AI service that processes the recordings on Encore’s behalf is a processor, which needs a data processing agreement under Article 28. And if that service runs its models outside the EU, the transfer rules of Chapter V apply. A recording of a voice is personal data. It becomes biometric data, with the stricter rules of Article 9, only if it is processed to identify people, which ordinary transcription does not do. Teams should be precise here, since both ignoring the rule and overstating it lead to bad decisions. Organizer staff interviewed as employees of a business customer are data subjects in exactly the same way as fans.

Encore keeps its research repository in EU-1, under the same residency rules as its other personal data, and runs model-assisted synthesis through its AI gateway, so that transcripts reach only models approved for that data. The consent workflow across all 40 teams belongs to Chapter 6, and data residency in general to the Architecture book’s Chapters 9 and 10.

What the trio still does in person

Everything in this section makes some part of discovery cheaper. None of it changes what discovery is for, which is to put the people who build a product in contact with the people who use it, and the scarce resource in that contact has always been attention. Cheaper prototypes make that attention more valuable, since a team that can test more ideas needs more real people to test them with.

Intuit’s chief executive Brad Smith, interviewed by Fortune in 2017, put the conviction in one sentence: “What you get from a follow-me-home you can’t get from a data stream.” In the four months of discovery, the resale trio’s most consequential observations did not come from interviews at all. They came from evenings spent with the box-office staff of two design-partner promoters on show nights, watching how resale questions actually reached them: by phone, by fans at the door holding forwarded screenshots, and by an artist’s tour manager asking why a fan-club ticket had appeared on a marketplace. Those evenings are why the first message the trio designed for sellers was about the old ticket rather than the money. I have come to believe that the measure of a discovery practice is how often the people who build the product are in the room when their customers struggle, and that no tool yet built changes the answer.

Recap. AI makes prototypes far cheaper, which is emerging practice with practitioner evidence, and the proven rule that prototypes are thrown away matters more as they get easier to make. Models can help sort research notes, provided every finding keeps an audit trail to its source, while synthetic users can rehearse an interview guide and are never evidence of demand. Consent and data protection apply to every recording, and the human work of discovery, watching real customers in their own setting, becomes more valuable as everything around it gets cheaper.

Exercise 3.5. Take one AI-generated summary of customer research from your organization. Pick three of its claims and trace each one back to the raw notes or transcripts. Count how many survive unchanged, how many survive in weaker form, and how many have no source at all.

Kata: The Empty Chair

Your brief: “Clinicly.” Clinicly sells appointment-scheduling software to private clinics in Germany, Austria and the Netherlands: about 1,400 clinics, from single physiotherapy practices to dental groups of 200 sites. It has 90 engineers in 11 teams. Receptionists use the product all day; patients book through a web page and an app; clinic groups sign the contracts, through an operations director and, in the larger groups, a procurement team. No-shows are Clinicly’s customers’ biggest complaint: on average, 14% of booked appointments go unattended, and an empty chair costs a clinic the full fee. The largest dental group, a fifth of Clinicly’s revenue, has asked for automatic double-booking of risky slots, and sales has promised a waitlist that fills cancelled slots automatically in order to close a deal with a physiotherapy chain. The chief executive has offered three clinic groups places as design partners: the dental group, the physiotherapy chain and a dermatology group. Nobody on the product teams has spoken to a patient this year. Patients’ booking histories reveal health information, which is special-category data under the GDPR.

Deliverables:

1. A six-week discovery plan — the weekly cadence for one trio, automatic recruitment of receptionists, patients and clinic managers, how to reach patients who never booked, and consent and data handling for each, given that appointment histories are health data.
2. An opportunity solution tree — rooted in one product outcome about attended appointments, with at least five opportunities from at least three kinds of participant, each with its source and segment. Show where the double-booking request and the promised waitlist sit once their underlying problems are written down.
3. An assumption map and one cheap test — for your top-ranked solution, list its assumptions by Torres’s five types and design a test for the riskiest that takes under a week. If it is a Wizard of Oz or a fake door, say what the patient sees, what happens behind it, and why no patient can be left with a booking that does not exist.
4. A design-partner program — Clinicly’s three groups span three segments. Argue for one segment and a set of six clinics within it, saying what to do about the three groups already promised places, and name the clinic you would recruit because it doubts the idea.
5. A pitch — for the first piece of work to come out of discovery, with all five parts, an appetite sized for one small team, and at least two no-gos, one of which the sales team will dislike.
6. A pre-registered decision rule — the measures, thresholds and decision-makers for your test, written as it would be signed before the test begins.

Rubric:

Where you now stand

You can now keep contact with customers as a weekly habit, interview for stories about the past instead of predictions, and organize what you hear into a tree that shows leaders your reasoning. You can unpack an idea into its assumptions, test the riskiest one cheaply, with a Wizard of Oz if need be, and decide what to do next against criteria written before the test. You can run discovery with business customers who buy through many hands, keep the loudest of them in proportion, and shape the result into a pitch and a release slice that cut across a whole journey. Discovery, though, mostly tells us what people do and why. It is weaker at telling us how much, how often and whether a change caused the difference we see, and the pilot’s four shows could not say how many fans across Encore’s catalog would resell. Those are questions of measurement, and the most rigorous answers to them were first worked out in agricultural field trials, where Ronald Fisher learned to tell a real effect from the noise of weather and soil. Chapter 4 takes up metrics, instrumentation and controlled experiments.

References

• Teresa Torres — Continuous Discovery Habits: Discover Products that Create Customer Value and Business Value. Product Talk LLC, 2021.
• Marty Cagan — INSPIRED: How to Create Tech Products Customers Love, 2nd ed. Wiley, 2017.
• Ryan Singer — Shape Up: Stop Running in Circles and Ship Work that Matters. Basecamp, 2019 — free online.
• Eric Ries — The Lean Startup. Crown Business, 2011.
• Jeff Patton, Peter Economy — User Story Mapping: Discover the Whole Story, Build the Right Product. O’Reilly Media, 2014.
• Geoff Colvin — How Intuit Reinvents Itself. Fortune, 2017 — free online.
• Intuit Inc. — From Kitchen Table to Boardroom Table: History of Quicken. Press kit, 2006 — free online.
• J. F. Kelley — An Iterative Design Methodology for User-Friendly Natural Language Office Information Applications. ACM Transactions on Office Information Systems 2(1), 1984.
• Paul Graham — Do Things That Don’t Scale. 2013 — free online.
• Marty Cagan, Lea Hickman, Christian Idiodi, Chris Jones, Jon Moore — Transformed: Moving to the Product Operating Model. Wiley, 2024.
• James Bisbee, Joshua D. Clinton, Cassy Dorff, Brenton Kenkel, Jennifer M. Larson — Synthetic Replacements for Human Survey Data? The Perils of Large Language Models. Political Analysis 32, 2024 — free online.
• Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, David Wingate — Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis 31(3), 2023 — free online.
• Maria Rosala, Kate Moran — Synthetic Users: If, When, and How to Use AI-Generated “Research”. Nielsen Norman Group, 2024 — free online.
• Ziang Xiao, Xingdi Yuan, Q. Vera Liao, Rania Abdelghani, Pierre-Yves Oudeyer — Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding. IUI ‘23 Companion, ACM, 2023 — free online.
• Clayton M. Christensen — The Innovator’s Dilemma. Harvard Business School Press, 1997.
• Desirée Sy — Adapting Usability Investigations for Agile User-Centered Design. Journal of Usability Studies 2(3), 2007.