Building Products in the AI & Cloud Era

← the whole book on one page

Chapter 8: Mobile and Many Channels — Software You Cannot Recall

A release we cannot recall has to be designed as though it will never be corrected.

Every chapter so far has assumed, without quite saying so, that a mistake can be taken back. A confusing page on the fan storefront is changed on Tuesday and gone by Tuesday evening, and the next fan who loads it never sees the old version. That assumption fails the moment our software lives in a fan’s pocket. A native app is a binary that we hand over and cannot take back. It runs on a device we do not own, updates when its owner decides, and keeps calling our servers for years, and between the build and the phone stands a store run by another company under rules that company writes. This chapter asks what deciding and designing a product looks like when part of it cannot be recalled, and when the same fan reaches us through an app, a mobile browser, a wallet pass, an email and a partner’s app, each with its own reach and its own failures.

The moment that made this the ordinary condition of consumer software can be dated. On 10 July 2008 Apple opened the App Store, as a section of iTunes 7.7, the day before iPhone OS 2.0 and the iPhone 3G went on sale. It opened with more than 500 native applications, and four days later Apple announced 10 million downloads in the first weekend; Steve Jobs called the store “a grand slam, with a staggering 10 million applications downloaded in just three days.” The path to that weekend is worth remembering. When the first iPhone shipped in June 2007 it had no third-party native apps at all, and Apple’s answer to developers that summer was that they could write web applications running in Safari. Developers were not persuaded, and some jailbroke their phones to run native code. In March 2008 Apple announced a software development kit and a store in which developers would set their own prices and keep 70% of the revenue, while Apple hosted, signed and reviewed every app before it reached anyone. The question of web or native, which Section 8.1 takes up, is therefore as old as the platform itself.

We should be careful about what the opening changed. The App Store was not the first gate between a company’s software and its customers’ devices. Carrier storefronts, Qualcomm’s BREW among them, had certified and sold phone software for years, and console makers had licensed and approved games since the 1980s. What 2008 changed was scale and reach. One platform owner’s review now stood in front of all third-party software on a mass-market, general-purpose device, Google’s Android Market opened that October, and within a few years store review and binary distribution had become the normal path from a company’s build to its customers. Three consequences followed, and each is the root of a section of this chapter. A reviewer now stood between the build and the user, so every fix now had to pass through a submission. A version, once installed, stayed until its owner replaced it, which gave rise to the long tail of old versions, to forced upgrades and to remote switches that can change a binary’s behavior after it has shipped. And the rules governing payments, notifications, wallet passes and data collection became a dependency written and revised by someone else. The staged rollouts we now take for granted came years later; in 2008 an approved version went to everyone at once. Figure 8.1 sets the dates of that year side by side, with the path a build took once the store had opened.

Figure 8.1 — The road to the App Store, and the path from a build to a phone after it. The platform’s first answer was the web; once the store opened, every fix passed a reviewer and every installed version stayed until its owner replaced it.

Encore feels each of these consequences sharply, because its most important artifact, the ticket, lives on the phone. Most fans hold their tickets in the fan app, many of the rest in Apple Wallet or Google Wallet, and they present them at stadium gates where thousands of phones compete for one overloaded cell. Verified Resale adds a demand the web never had to meet. When a ticket is resold through Rowmate or Tessera, Encore reissues it under a new ticket identity, and the seller’s ticket must stop working while the buyer’s starts, on app versions up to two years old, on phones that may be offline, without the gate ever accepting both. Every section that follows is, in the end, about whether that one guarantee holds.

8.1 Choosing How to Build Mobile

The channel map

Before we choose how to build an app, we should ask which of the customer’s jobs it is for. A fan’s relationship with Encore is carried by many channels at once. A link in a message opens the mobile web; the fan buys in the app or on the web; the ticket arrives by email and is added to the app, a wallet or both; a push notification says when doors open; and on the night the fan holds up the phone at the gate. If the fan cannot go, a partner’s app may be where the ticket is listed. Each channel has its own reliability, speed of change and owner, and a product that does not decide which channel does which job will find every channel attempting every job.

A channel map is a table that assigns each customer job to the channel that owns it, and names a fallback channel for when the owner fails. There is no famous method behind the name. It is ordinary omnichannel practice, written down so that a product trio and forty teams can argue about the same thing. Its value lies in two columns that are easy to omit. The first is ownership: when the ticket view in the app and the ticket in the wallet disagree, one of them is the truth and the other is a convenience, and the team needs to know which before the disagreement happens. The second is the fallback. Every job has to survive the failure of its owning channel, because every channel fails: push notifications are dropped, email lands in spam, wallets update late, and apps are deleted to free space the night before a festival.

A touring production thinks about its show in the same way. The stage manager’s book assigns each cue to one person and one desk, and for the cues that would stop the show if missed it names a backup: the second follow-spot, the manual fly line. Nobody on a touring crew believes the lighting desk will never crash. The analogy breaks over control. A stage manager owns every desk in the building, while Encore owns only some of its channels: Apple and Google own the wallets, Rowmate and Tessera their apps, and the fan’s email provider decides what reaches the inbox. The fallbacks matter most exactly where Encore’s control is weakest.

Figure 8.2 — Encore’s channel map for the jobs Verified Resale touches. The third row is the one the whole chapter defends.

Figure 8.2 records a decision Encore took in its eighth year, when it built rotating barcodes for every mobile ticket. Today about 70% of tickets are presented from the fan app, about 15% from Google Wallet, about 12% from Apple Wallet and about 3% on paper from the box office, for fans without a usable smartphone. The printable PDF ticket was retired for mobile orders that year, because a static image that can be forwarded to five people is precisely the credential Verified Resale exists to replace. The consequence for resale is plain: reissue must work on every channel that can hold a ticket, and a resale that updates the app but not the wallet, or tells the buyer by push but not by email, has broken a row of the table. Even the convenience channel exists on terms Encore did not write: Apple’s guidelines limit wallet passes to uses such as “movie tickets, coupons, and VIP credentials,” and warn that “other uses may result in the rejection of the app and the revocation of Wallet credentials.”

Native, cross-platform, or web

With the jobs assigned, the next decision is how to build the app that owns the most important of them. Native apps are written separately for each platform, in Swift and Kotlin; cross-platform frameworks such as React Native and Flutter drive both platforms from one codebase; and the web runs in a browser or a thin native shell. Native gives the best fidelity to each platform at the cost of building most features twice. Cross-platform shares most of the work at the cost of a layer that must be debugged, upgraded and occasionally fought. The web reaches every device with one deploy and no store, at the cost of weaker access to the device. Gergely Orosz’s Building Mobile Apps at Scale (2021), drawn from his years at Uber and from interviews across the industry, spends five chapters on these trade-offs and declines, rightly, to declare a winner.

The published record is more instructive than any comparison table. In June 2018 Gabriel Peal published “Sunsetting React Native” on Airbnb’s engineering blog, one of a series in which Airbnb explained why, after two years, it was moving its React Native screens back to native code. The reasons the series gave were technical and organizational together. Sharing code did not hide the platforms as much as hoped, since text rendered differently on each and keyboards behaved differently; starting the React Native runtime before the first screen took several seconds in an app of Airbnb’s size; and shipping consistently good work required engineers with expertise in three environments, which was hard to staff. Airbnb was candid that the verdict was not unanimous: by its own survey, 63% of its engineers would have chosen React Native again. In January 2020 Shopify went the other way. “After years of native mobile development,” Farhan Thawar wrote, “we’ve decided to go full steam ahead building all of our new mobile apps using React Native.” Shopify had tried React Native in 2015 and rejected it, so the 2020 decision was a deliberate reversal made on new evidence. Five years later its engineers reported, in a January 2025 retrospective, that the bet had largely paid off, with screens loading in under 500 ms at the 75th percentile, while naming debugging and the effort of upgrading the framework as the persistent costs.

The story has a third act, and it is the one place in this chapter where the AI era changes a mobile decision. On 10 September 2026 Shopify Engineering published “Native is now the future of mobile at Shopify,” by Mustafa Ali, announcing a move back to Swift and Kotlin. The post does not say React Native is slow. Its argument is economic: “LLMs changed one of the core assumptions behind our 2020 decision, so we reevaluated our mobile stack from first principles,” and “agents have reduced the advantages of sharing implementation, while the advantages of building for each platform remain.” Shopify reports rebuilding its Shop app natively in 12 weeks, with the main Shopify app, more than 300 screens, still in progress. We should read this as what it is: one large company’s published reasoning, weeks old, by the same company that made the opposite case in 2020. It is not yet a trend, and the general evidence about coding assistants, which Section 1.5 summarizes and the Software book’s Section 5.1 examines in full, is more mixed than any single account. What the Shopify story does show, unmistakably, is that the native-versus-shared decision rests on an assumption about the cost of writing code twice, and that a decision resting on a cost assumption should be revisited when the cost moves.

A middle path has also matured. Kotlin Multiplatform, which shares business logic between Android and iOS while leaving each interface native, was declared stable in November 2023, and Compose Multiplatform extended sharing to the interface on iOS in May 2025. Shared logic suits code that must behave identically on both platforms, and Encore has exactly such code.

Backend-driven UI

A further option cuts across the first three. In backend-driven UI, the server describes a screen as a structure of known components, and the app renders whatever structure it receives. Airbnb’s Ghost Platform, described by Ryan Brooks in a 2021 post on Airbnb’s engineering blog, is the best documented example: screens are composed of sections, served through a shared data layer, and rendered by the same component vocabulary on web, iOS and Android. The appeal is obvious. A screen defined on the server can change on the server’s schedule, which for a native app means changing without a release, without review and without waiting for fans to update.

The price is paid in native feel, in testing, since the space of screens the app may be asked to render is no longer fixed at build time, and in old apps, which can break when a server sends a component they do not know. Backend-driven UI therefore works only under three conditions together: a component library already exists on every client; old clients can skip unknown components without failing; and the screens genuinely change faster than the release train can carry them.

Encore uses it for one screen, the event page, which differs by organizer: a festival wants a lineup grid and a camping map, a stadium tour a date picker across 12 cities. The variety changes weekly. The Design System team’s native components already cover the building blocks, and the fan BFF, which the Architecture book’s Section 8.4 gave to each experience, already versions its responses by app version, so old clients simply omit what they cannot draw. Everything else stays native and fixed in the binary, the ticket view above all, because the moment it most needs to render is the moment the network is least likely to be there.

The decision record

The Architecture book’s Section 1.4 taught the architecture decision record (ADR), after Michael Nygard’s 2011 essay, and the format need not be taught again. Mobile decisions add one element that deserves more emphasis than it gets: reversal conditions, the observable facts that would make us reopen the decision. Shopify’s 2026 post is a reversal condition being triggered, “a core assumption changed,” and the decision was reopened on that ground. A mobile platform decision commits forty teams for years, and the cheapest moment to say what would change our minds is the day we decide.

Encore’s record for the fan app is short, and its last section is the part most teams leave out.

ADR-031  How Encore builds the fan app            Status: accepted, year 7
Decision: Native apps (Swift, Kotlin) for fans; web for organizers.
  Shared Kotlin Multiplatform module only for ticket rendering and
  barcode rotation (identical behavior is a correctness requirement).
  Backend-driven UI only for the organizer-specific event page.
Because: the ticket must open offline and fast on every supported phone;
  rotation must match the gate's validator exactly on both platforms;
  the organizer console is desk work and needs no store.
Reversal conditions (reviewed each January):
  - a shared-logic defect ships to fans twice in one quarter;
  - building a typical feature twice costs more than 1.5× building it once,
    measured over two quarters;
  - a platform drops a capability the ticket view depends on.
Owner: Mobile App team, with the Design System and Venue Entry teams.

The second condition is the one the Shopify story bears on. Encore measures it, imperfectly, by comparing the recorded effort for the two halves of features built twice; it has not been triggered.

Recap. A channel map assigns each customer job to one owning channel and names a fallback, because every channel fails and several belong to other companies. Native, cross-platform and web each buy something real, and the published record, from Airbnb in 2018 to Shopify in 2020 and again in 2026, shows that the right choice depends on costs that move. Backend-driven UI earns its keep only where components already exist, old clients degrade safely and screens change faster than releases. A decision record for mobile should say in advance what would reopen it.

Exercise 8.1. Write your own channel map. List every customer job your product supports, the channel that owns each, and the fallback when that channel fails. Mark every channel you do not control. For any job without a fallback, write one sentence on what a customer experiences on the day its owning channel fails.

8.2 Releasing Through a Store

Mistakes are not reversible

On the web, a defect has a lifetime measured from the deploy that introduced it to the deploy that removed it, and with a good pipeline that lifetime is minutes. A defect in a native app has a lifetime with three parts, and only the first is under our control. We must build and test a fix. We must then pass review: Apple states that on average 90% of submissions are reviewed in less than 24 hours, though developers report longer waits at busy times; an expedited review exists for critical fixes. And then, longest of all, we must wait for fans to update. Some update the same day, many within a week, and a stubborn minority never do. Orosz gives an early chapter to exactly this point, under the title “Mistakes Are Hard to Revert,” and it is the root from which most of mobile release engineering grows. The honest figure for a mobile defect is a day or more to ship the fix, then days to weeks before most users have it, and months before nearly all of them do. Figure 8.3 sets the two lifetimes side by side.

Figure 8.3 — The life of a defect on the web and in a native app, drawn as a schematic. Only the first of the app’s three parts is ours, and the longest belongs to the fans.

Publishing has lived with this for centuries. Once a print run leaves the bindery, an error is permanent in every copy sold, so printers invest heavily in proofs: a mistake caught at proof costs a pencil mark, and one caught in the bookshop costs a reprint. The analogy breaks in an instructive direction. A printed book is inert, while a shipped app keeps executing and keeps reacting to whatever data we send it, so an old version can begin to fail long after it shipped, without any change to its own code.

The clearest public demonstration came on 6 May 2020. A change on Facebook’s servers caused many apps that embedded Facebook’s iOS software development kit (SDK), Spotify, TikTok, Pinterest and Tinder among them, to crash at launch, including for users who had never logged in with Facebook. The apps’ own code had not changed. Code already on millions of phones had read a remote response it could not handle, and none of the affected companies could ship a fix faster than Facebook could revert its server change. It happened again on 10 July 2020. The lesson for release engineering is broader than third-party libraries, though Section 8.4 returns to those. A binary we cannot recall is exposed to every remote input it parses, so what a mobile app reads at launch is part of its release surface, and it must fail safe.

That is why testing and flagging are worth more on a phone than on the web. The Architecture book’s Section 8.4 called the seam between the fan app and its backend a contract measured in years; the release engineer’s corollary is that the levers which shorten a defect’s life after shipping are only the ones designed in beforehand.

The release train

A release train is a fixed release cadence on which a new version leaves at a scheduled time with whatever is ready, and features that are not ready wait for the next one. Like a railway timetable, it does not wait for a late passenger. The alternative, a release that waits for a set of features, turns every late feature into a late release and every late release into pressure to cut testing. A feature that misses a weekly train loses a week, which is cheap, but only if it was built to be switched on separately from its arrival, as Section 8.4 explains.

The best evidence for trains comes from Facebook. In a peer-reviewed paper at the 2016 Foundations of Software Engineering conference, Chuck Rossi, Kent Beck and colleagues reported that Facebook’s Android app had moved over four years from a release every eight weeks to one every week, and that across seven years of data, more frequent releases had not reduced either productivity or quality. They singled out dogfooding, the practice of employees using pre-release builds every day, and alpha and beta feedback, as critical to release quality. Spotify described its own weekly train in 2025 in unusual detail. The release branch is cut on Friday, after which “only critical bug fixes are permitted”; a public beta is built from the branch; the build goes to the stores the following Monday; roughly 1% of users receive it first and the rest the next day; and Spotify reports that it is “able to roll out more than 95% of releases to all our users.” Each release, by Spotify’s account, carries perhaps three to five blocking bugs that must be fixed on the branch before it ships. The two accounts, nearly a decade apart, describe the same arrangement: a fixed cadence, a short stabilization period on a branch, and a gradual exposure.

The beta tracks are where the train earns its safety. TestFlight on iOS and the testing tracks on Google Play put a build into real hands before review. A staff beta catches what automated tests cannot imagine, and a public beta of volunteer fans catches what staff cannot, because staff phones are newer, staff networks better and staff know where the buttons are.

Encore runs a weekly train, and its week is arranged around the one day the company cannot afford surprises. Figure 8.4 shows it.

Figure 8.4 — Encore’s weekly app train. A missed train costs a feature one week; it never delays the train.

The Friday rule is the mobile form of the Resilience book’s policy (Section 4.3) that server canaries take no steps during on-sales: a rollout step on an on-sale morning would mix two sources of change in the hour that can tolerate the fewest surprises.

Staged rollout and halting

A train that delivers every release to every fan at once would turn each defect into a defect for everyone. Both stores therefore let a new version reach users gradually, and the mechanics differ in ways that matter. The general theory of progressive delivery, canaries and their analysis belongs to the Resilience book’s Section 4.3; this section adds only what the stores allow and what they do not.

Apple’s phased release spreads an update over seven days, at 1%, 2%, 5%, 10%, 20%, 50% and 100%, but only among users with automatic updates on; Apple’s documentation is explicit that the update “can be manually downloaded from the App Store by anyone at any time.” The phasing can be paused for up to 30 days in total, or ended early, but never reversed: a paused version stays on the store and on every phone that has it, and restarting a controlled rollout means withdrawing the version and submitting another. Google Play gives more control, since the developer sets the percentage and halting stops new users from receiving the version, but users who already received it “will remain on that version.”

Neither store, then, offers a rollback. A mobile “rollback” is always a roll forward, either a new build that undoes the defect or a remote switch that disables it, and it is the Resilience book’s roll-forward principle in its strictest form. A staged rollout limits how many fans a defect reaches before we notice it. It does not undo the defect for the fans it has reached.

Halt criteria must be written before the rollout begins, for the reason the Resilience book gave for error budget policies: decisions made in calm weather are better than decisions made in an incident channel. Google publishes two thresholds with consequences. A user-perceived crash rate above 1.09% overall, or 8% on any single phone model, or a user-perceived rate of “application not responding” (ANR) events above 0.47% overall, or 8% on any model, measured over 28 days, may lead Google Play to reduce the app’s visibility and warn on its store listing. Those lines separate unacceptable apps from merely poor ones, so Encore treats them as distant ceilings. Its own criteria, in Figure 8.5, sit well inside them and add the one measure no store can compute.

Figure 8.5 — Encore’s halt criteria for a staged app rollout. Any single line halts; widening needs all of them clear.

The fourth row matters most. A version in which the ticket fails to render offline would show up nowhere in the store’s numbers, because the app would not crash; it would show a spinner to a fan in a queue. So the Mobile App team records every attempt to open a ticket and whether a barcode appeared within a second, queuing the measure on the device until the network returns.

The long tail and forced upgrades

A web product has one version in production, or two during a deploy. A mobile product has dozens, and the shape of their distribution is a long tail: most sessions come from the last few versions, and a thin but stubborn fraction comes from versions many months old. At Encore, the oldest version still above 1% of sessions is about 14 months old, and versions up to two years old still open the app on match days, usually on phones that were used for one festival and then forgotten in a drawer. Every one of those versions still calls the fan BFF, which is why the Architecture book’s Section 8.4 had that BFF version its responses, track live app versions and keep a forced-upgrade floor. The architecture is settled there. What remains is a set of product decisions, which architecture cannot make.

The car industry knows this tail. A manufacturer recalling a car can write to every registered owner but cannot drive the cars to the workshop, and some owners never bring them in. Aviation shows the other end. An airworthiness directive legally obliges every operator of the affected aircraft to make a correction by a deadline, and an aircraft that has not complied may not fly. A version floor is the app’s directive: the minimum app version the backend will serve, below which the app must update before it can continue. Like the directive, it should be used rarely, and the analogy breaks over who bears the cost. An airline pays for its own grounded aircraft; a fan blocked by a hard floor pays for ours.

So Encore distinguishes two floors. Below the soft floor, the app shows a prompt explaining why an update is worth having and lets the fan continue. Below the hard floor, the app refuses everything except one thing, and the exception is the heart of the policy: the ticket must still open. A fan whose app is too old to buy, list or browse can still show a valid ticket at the gate, because the ticket view and its rotation logic are deliberately the most stable code in the app. Android offers a system mechanism for prompting updates, the Play in-app updates API with its “immediate” and “flexible” flows, which correspond roughly to the hard and soft floors; iOS has no equivalent, so the app checks its version against the backend and deep-links to the store.

The authority to raise the floor sits with the Mobile App team’s product manager, who must consult the product trio of any module whose users the change strands, and it is exercised for three reasons only: security, a store requirement, or one of the fairness invariants. Verified Resale supplied the third reason. A fan running a version that predates reissue would, as a seller, go on seeing a ticket the gate will refuse, and, as a buyer, could not display the new one at all. The gate’s revocation list keeps that from becoming a fairness failure, as Section 8.3 explains, but it would still be a fan turned away with a ticket that looked valid on the phone. So Encore did two things. It raised the soft floor to the first version carrying the reissue-aware ticket view three months before launch, and watched the share of active ticket holders below it fall week by week. And it made that version a requirement for taking part in resale at all: an app below it cannot list a ticket or buy one, and offers the update instead. The general hard floor did not move, because no fan who merely holds a ticket needed to be stranded for resale to be safe.

A quieter piece of the same work concerns the cached ticket on the phone, with its rotation secret, seat and status. It has a schema of its own, and Orosz’s chapter on client-side data migrations draws the tedious rule: every new version must migrate every older cache it may find, because there is no knowing which version a fan is updating from.

Store policy as a dependency

The last consequence of 2008 is the one most often discovered too late. A store’s rules are an API owned by someone else, with no versioning promise, and like any such API it deserves an owner who reads its changelog. Several rules bear on Encore directly, and the useful habit is to check them before design, while they can still shape it.

The first concerns payment. Apple’s guideline 3.1.3(e) states: “If your app enables people to purchase physical goods or services that will be consumed outside of the app, you must use purchase methods other than in-app purchase to collect those payments, such as Apple Pay or traditional credit card entry.” A concert is consumed outside the app, so in-app purchase does not apply to Encore’s primary or resale sales. The check was made before the resale checkout was designed, and recorded so that it can be revisited if the rule changes. No guideline mentions ticket resale at all, which is itself worth knowing.

Other rules impose a calendar. Google Play requires apps to target a recent Android API level by 31 August each year, which forces a yearly Android update whatever the roadmap says. Apple’s privacy manifests, enforced from 1 May 2024, require apps and many SDKs to declare the data they collect. And in the European Union the Digital Markets Act has, since iOS 17.4 in 2024, allowed alternative app marketplaces and payment providers on iOS; the fee terms have changed more than once, and Encore has no need of them.

At Encore, the Mobile App team keeps a policy watch: a short register of the store rules the app depends on, each with the guideline number, the date last checked and the decision it supports, reviewed monthly and whenever Apple or Google publishes changes. It is the same discipline a team applies to a vendor’s API, applied to a vendor that also reviews every release.

Recap. Store review and binary distribution give a mobile defect a long life that only the binary’s owners can end, which is why testing, trains and switches are worth more on phones than on the web. A weekly train with beta tracks, as Facebook and Spotify have documented, keeps the cadence regular and the cost of a missed feature small. Staged rollouts limit exposure but never undo it, so halt criteria must be written in advance and must include the product’s most important job. The version floor is a rarely used instrument whose policy matters more than its mechanism, and store rules are a dependency to be read before design.

Exercise 8.2. Plot the share of your app’s sessions by version for the last 30 days. How old is the oldest version above 1%? Then write down what a user on that version can still do if you raise your hard floor tomorrow, and who in your organization has the authority to decide it.

8.3 What Only Phones Do

Tickets offline

A stadium gate is one of the worst network environments a consumer product will ever meet: tens of thousands of phones on the same cell in the same half hour, and a fan at the front of the queue who needs the ticket now. The first rule of the ticket view follows: it must open with no network at all. Everything the phone needs to display a valid ticket must already be on the phone, and the gate must be able to check it without asking anyone.

That rule collides with the second requirement, which is that a ticket should be hard to copy. A static barcode, the kind printed on a PDF, is just an image, and an image can be screenshotted and sent to five friends, of whom the first to reach the gate gets in. (The Architecture book’s Section 3.1 called Encore’s entry credential a QR code; it is the same thing that this book, following the trade, calls a barcode.) The industry’s answer is the rotating barcode: a code generated on the device from a per-ticket secret and the current time, so that it changes every few seconds and a screenshot expires almost as soon as it is taken. Ticketmaster introduced the best-known version, SafeTix, in 2019, with a barcode that refreshes every 15 seconds. The mechanism underneath is the time-based one-time password of RFC 6238, published in 2011 for two-factor login codes: a shared secret and a clock produce the same short code on two machines that never talk to each other.

What rotation buys is narrower than the marketing suggests. In 2024 an independent researcher writing as conduition published a reverse-engineering of SafeTix which found, as the design implies, that the codes are computed from secrets the app downloads in advance, so anyone who extracts the secrets can generate them too. Rotation stops the screenshot and the forwarded image, which is most of the problem in practice, but not a determined holder, and so it is only half of Encore’s design.

The other half is the revocation list: a list, held by every gate scanner, of ticket identities that are no longer valid, so that the scanner can refuse them without a network. Each scanner, run by the Venue Entry team’s software, holds the per-ticket secrets it needs to validate codes and the list of revoked identities for its event, syncs both every 30 seconds while it has a connection, and may run offline for up to four hours. When Encore reissues a resold ticket, the old identity joins the revocation list, the buyer receives a new identity with a new secret, and from that moment the seller’s phone can generate perfectly well-formed codes that every gate will refuse. A scanner’s clock is part of the system too. RFC 6238 advises validators to accept a small window of adjacent time steps, and Encore’s scanners accept one step either side; the app corrects for a fan’s wrong phone clock by recording the server’s time at each sync and computing codes from the corrected time, so that a fan who set the clock by hand still gets in.

One canonical rule makes the arrangement tight. Encore’s default resale window closes two hours before doors open, and organizers may only shorten it. Every revocation for an event therefore exists before the first fan reaches a gate, and a scanner that has synced once after the window closed holds a complete list. A scanner that has not, perhaps because it was switched on in a basement, shows an amber status to the gate supervisor, who knows to move it. The guarantee that the gate never accepts both rests on the list and the window together; nothing the fan’s phone does is part of it. Figure 8.6 follows a reissue from Encore to the gate.

Figure 8.6 — A resold ticket is reissued. The two vermilion paths carry the guarantee, and both end at the gate scanner; the phones and the wallets only display what the gate will judge.

Wallet passes

Wallet passes are the convenience channel of Figure 8.2. Apple announced Passbook on 11 June 2012 and shipped it with iOS 6 that September; it became Wallet in 2015. From the beginning a pass could be updated after issue: the device registers the pass with the issuer’s web service, the issuer’s server sends an empty push when the pass changes, and the device asks what changed and downloads it. Google Wallet offers the same through its own API.

The mechanism works only when the device is online, and its notifications are delivered on a best-effort basis; a phone in airplane mode or in a stadium cell that has given up receives the update later. That draws a hard line around what a wallet update may be trusted with. When a ticket is resold, Fan Messaging, the team that owns the Notifications service, pushes an update that marks the seller’s pass “Transferred” and removes its barcode, and delivers a new pass to the buyer. Both are courtesies. The seller’s pass might go on displaying its old barcode for hours, and the gate will refuse it, because the revocation list is the guarantee.

The two wallets also differ in what they can show. Google Wallet supports rotating barcodes for event tickets through a rotatingBarcode field built on RFC 6238, and Google’s guidance keeps the secret out of the token used to save the pass and lists fallbacks in order: its Smart Tap NFC protocol, then the rotating barcode, then a static barcode, noting that the static fallback weakens security. Apple Wallet does not display rotating barcodes; Ticketmaster’s rotating tickets in Apple Wallet use near-field communication (NFC), which requires Apple-approved NFC passes. Encore has no such approval. Its Apple Wallet passes therefore carry a static barcode, which is the weakest credential Encore issues, and the company accepts it knowingly: the pass is a convenience, the app is the credential of record, and a forwarded copy of a static pass is defeated at the gate by the same revocation list that defeats a reissued ticket. The cost is that two people holding copies of one unresold static pass race to the gate, which is the old problem in a smaller form. The Resilience book’s Section 7.2 holds the signing keys for passes and their rotation.

The buyer’s side has a promise attached. The Resilience book’s Section 1.2 set Encore’s ticket-delivery objective: 99.5% of orders with the ticket in the wallet within five minutes, over a rolling 28 days, measured by Notifications and confirmed by a wallet receipt. A reissued ticket is an order like any other for this purpose, and Verified Resale was designed to meet the existing objective, with no weaker one of its own.

Push notifications

Push notifications are cheap to send, and their costs are borne by someone else, which is why they are so often misused. Both platforms now require the user’s permission; Android made it a runtime permission in Android 13, in 2022. Apple’s Human Interface Guidelines advise that an app “request permission only when your app clearly needs access to the data or resource,” and warn that asking too early “can make it hard for people to trust your app.” Encore’s app asks the first time a fan buys a ticket, explaining that it will say when doors open and when anything changes about the ticket, and never on first launch, when the fan has no reason to agree.

Apple’s guideline 4.5.4 adds two constraints that are product decisions in disguise. Push “must not be required for the app to function,” and must not be “used for promotions or direct marketing purposes unless customers have explicitly opted in.” Encore therefore keeps two kinds of notification apart. Transactional messages, about a ticket the fan holds, rest on the permission alone. Marketing messages, from the Fan Growth team, need a separate opt-in, recorded with its consent language and revocable in the app.

Underneath both lies a technical fact that settles a design question. When a device is offline, Apple’s push service stores only the most recent notification for each app and discards it after a limited time, and Google’s equivalent service makes no promise of delivery either. Push delivery is best effort by design. So nothing critical depends on push alone. When a fan’s ticket sells, the news travels three ways at once: by push, by email, and as an entry in the app’s inbox, which the app fetches whenever it opens. A fan who missed the push, never granted permission or deleted the email will still find the message the next time they look.

Deep links

A deep link is a URL that opens a specific screen inside an app when the app is installed and falls back to the same content on the web when it is not. Apple’s universal links, from iOS 9, and Android’s app links, from Android 6.0, both introduced in 2015, made this reliable by having the app and the website vouch for each other through files the website serves, so that no other app can claim the links. For Encore the deep link is how other channels hand a fan to the app. A link in an email opens the ticket; a link from Rowmate’s listing opens the resale checkout with Rowmate’s attribution carried in the URL, so that the partner earns its four points of the fee when its link produced the sale.

The history of deep links offers a lesson in the dependency risk of Section 8.2. Many companies built their links on Firebase Dynamic Links, a Google service that handled the install-then-open dance for them. Google deprecated the service and shut it down on 25 August 2025, after which every such link, including those on custom domains, stopped working. Every email ever sent and every partner page ever published with one of those links broke on the same day. Encore owns its link domain and uses the platforms’ own verified links, so that the links it prints in emails and hands to partners will work for as long as Encore exists. Partner attribution and the partner API itself belong to Chapter 9.

Biometrics and device security

A completed resale cannot be undone, since the ticket has been reissued to someone else, so Encore asks the fan to confirm a sale with Face ID, a fingerprint or the device passcode. That is sound interface design, and there are two common mistakes in its engineering.

The first is to treat the biometric check as proof. An app that checks the fingerprint locally and sends the server a flag saying “authenticated” has told the server nothing a modified app could not forge. The check proves something only when it unlocks a key held in the phone’s secure hardware, the Secure Enclave or the Android Keystore, and that key signs the request. Encore’s app registers such a key’s public half when the fan signs in and signs each sale confirmation with it. The surrounding rules, no long-lived secrets in the binary and authorization-code sign-in with proof keys, are the Architecture book’s Section 7.4.

The second mistake is to confuse this step with the European rules on payment. Strong customer authentication under the revised Payment Services Directive (PSD2), in force since September 2019 and enforced for online card payments in the European Economic Area from the end of 2020, requires two of three factors, knowledge, possession and inherence, and it is carried out by the card issuer or the payment provider, through 3-D Secure or through wallets such as Apple Pay and Google Pay that bundle it. Encore’s Face ID on “sell” is Encore’s own step-up for an irreversible transfer of a ticket, and not a regulatory requirement; the buyer’s payment carries its own strong authentication through Orders & Payments, as every Encore payment does.

Recap. A ticket must open with no network, and rotation computed from a pre-delivered secret makes copying harder without making it impossible, so the guarantee against double entry belongs to revocation lists held by the gates and to a resale window that closes before doors. Wallet updates and push notifications are best effort, so they are courtesies that never carry a guarantee, and anything a fan must know travels by several channels. Deep links should rest on domains we own, and a biometric check means something to a server only when it unlocks a key that signs the request.

Exercise 8.3. Put your phone in airplane mode and try to use your product’s most important artifact: the ticket, the boarding pass, the receipt, the document. Write down what fails, and what the user would have to do at that moment. Then name the component that would have to hold the guarantee if the phone could not.

8.4 Flags, Experiments, and Telemetry on Devices

Remote configuration

If a binary cannot be recalled, the next best thing is a binary whose behavior can be changed from outside. Remote configuration is a set of flags and parameters the app fetches from a server and applies to code already in the binary, so that a feature can be switched on, tuned or switched off without a release. The kinds of flag and their lifetimes, release, experiment, operational and permission, are taught in the Resilience book’s Section 4.3 and the Software book’s Section 4.5, and this section adds only what changes when the code reading the flag sits on a phone.

Two things change. The first is that the app cannot count on reaching the server. Firebase’s guidance for its remote configuration service is blunt: “Don’t rely on network connectivity to obtain Remote Config values,” and “Do set in-app default parameter values.” Every flag therefore has a default compiled into the binary, and for a risky new feature the safe default is off. The second is timing. Of the loading strategies Firebase describes, the gentlest and most common applies at the next launch the values fetched during this one, and Firebase advises against changing the interface mid-session. The consequence, often discovered in an incident, is that a switch flipped now reaches most users only when they next open the app.

A kill switch is a remote flag whose only purpose is to disable a feature quickly when it misbehaves, and a kill switch is only as fast as its slowest path to the phone. Encore’s app applies values at the next launch for ordinary flags, and listens for real-time updates to kill switches while it is in the foreground, so that about 80% of active sessions see a flipped switch within 15 minutes. Every risky feature ships behind one, and resale had several: one for listing, one for buying, and one for the reissue-aware ticket view’s new status display, which could fall back to the old display without hiding the ticket.

The Facebook SDK crashes of 2020 add a requirement. What the app fetches at launch can itself be the hazard, and third-party SDKs often fetch remote data at launch on their own. Encore’s Mobile App team keeps an approved list of SDKs, each with an owner, a privacy manifest and a budget for app size and launch time, and initializes each one behind a kill switch of its own, so that a vendor’s bad day can be switched off without a release. Orosz gives a chapter to third-party libraries, and Encore’s list rests on the same view: a library inside the binary ships in Encore’s release, whoever wrote it.

Experiments on clients

The statistics of online experiments belong to Chapter 4, and nothing about them changes on a phone. What changes is the plumbing, and Kohavi, Tang and Xu give a chapter of Trustworthy Online Controlled Experiments (2020) to the difference between thin clients such as browsers and thick clients such as native apps. Four practical points follow for a team running experiments in an app.

First, every variant must already be in the binary. The server can choose which variant a fan sees, but it cannot send code, so an experiment that needs a new interface waits for a train and then for adoption. Second, assignment must be stable across sessions and devices, which Encore achieves by hashing the fan’s account through the same flag service that Chapter 4 described for the web, and caching the assignment on the device so that an offline session shows the same variant as the last online one. Third, exposure should be logged when the fan actually sees the variant, because many fans whose app fetched the configuration never reach the screen, and counting them dilutes the effect; this is the triggering that Kohavi and his co-authors treat at length. Fourth, the population on a new version is not a random sample. Fans who update early differ from those who update late, so an experiment that starts with a new version runs at first on a skewed population, and the app version belongs in the analysis as a covariate. Logs, too, arrive late from offline phones, and their timestamps are only as good as the phone’s clock, so the analysis should wait for the stragglers.

Chapter 4’s rule for Encore holds on every channel: no experiment ever varies the price an individual fan pays.

Crash and performance reporting

The Operations book treats crash triage on call, but it has no mobile material, so the client measures that Encore’s halt criteria depend on are defined here. A crash-free session rate is the share of app sessions that end without a crash; a crash-free user rate is the share of users who experienced no crash over a period. The two answer different questions. Sessions weight heavy users heavily and are the better guide to how the app behaves; users weight everyone equally and are the better guide to how many people were hurt. Google’s user-perceived crash rate and the related rate of “application not responding” events, the measures behind the thresholds in Figure 8.5, are counted per daily user and only for crashes the user could notice. Alongside them sit the measures of speed: the time from tapping the icon to a usable first screen, the share of frames drawn late enough for the eye to catch a stutter, and network failures broken down by carrier and country, which at Encore is how a fault in one mobile network shows up before anyone files a complaint.

None of this works without symbolication, the translation of a crash’s raw addresses back into function names and lines. The files that make it possible, dSYM files on iOS and mapping files on Android, come from the build and cannot be recreated later, so Encore’s pipeline uploads them with every release build and fails the build if the upload fails. Spotify gates store submission on “crash rates and key metrics” below defined thresholds, and Encore’s Wednesday go or no-go does the same.

Product analytics on devices

Chapter 4 set out Encore’s tracking plan and the consent rules under the General Data Protection Regulation (GDPR) and the ePrivacy Directive, and they apply to the app unchanged. It is worth correcting a common confusion about the platform rules, because it leads teams to the wrong checklist. Apple’s App Tracking Transparency (ATT), required since iOS 14.5 in April 2021, governs tracking a user across other companies’ apps and websites, and access to the advertising identifier. Encore does neither, and if it ever did, ATT would apply. Its own first-party analytics fall under the European rules, under Article 5(3) of the ePrivacy Directive for anything stored on or read from the device, and under Apple’s guideline 5.1.1(ii), which says that apps “must secure user consent for the collection, even if such data is considered to be anonymous,” and that “paid functionality must not be dependent on or require a user to grant access to this data.”

The device adds mechanics of its own. Events are batched to save battery and data, and queued when the phone is offline, with a maximum age, after which they are dropped, since an event that arrives days late would distort analyses already run. Consent is stored on the device and honored before the first event is recorded, which means the consent screen must come before any analytics initialization, a sequencing bug that is easy to ship and hard to notice. Event times are recorded on the phone’s clock and corrected with the same server offset the ticket view uses. And each third-party SDK that collects anything appears in the app’s privacy manifest.

One feedback channel is particular to apps: the store reviews, prompted sparingly through the platforms’ rate-limited in-app rating requests. Each module’s trio reads its reviews weekly as one input to the continuous discovery of Chapter 3.

Flag debt on clients

Flags accumulate, and Orosz’s chapter title for the result, “Feature Flag Hell,” describes it well enough. The cleanup disciplines, an owner for every flag, an expiry date and removal as part of done, are the Software book’s (Section 4.5) and the Resilience book’s (Section 4.3). A phone adds one constraint that changes the arithmetic. A flag in a shipped binary is read by that binary for as long as the binary is in use, so the server cannot stop serving the flag, or change its meaning, while any version that reads it remains above the version floor. Removing the code from the next release is easy; retiring the flag takes as long as the long tail takes to drain.

Encore therefore ties every client flag’s expiry to a version, and records it in a manifest that the app’s build reads and a linter checks:

flag: resale.listing.enabled
owner: resale-team
kind: release                  # release | experiment | operational | permission
default_in_binary: false       # safe value when config is unreachable
kill_switch: true              # real-time while app is in foreground
first_version: 9.14.0
code_removed_in: 9.40.0        # planned; linter warns 4 trains before
retire_server_value_when: hard_floor >= 9.40.0

The last line is the mobile-specific idea in compact form: the version floor is the flag’s expiry date. A flag cannot be retired from the server until no supported version reads it, and so every client flag’s true lifetime is set by the floor policy of Section 8.2. Encore counts the flags its app reads at launch each quarter. When resale launched there were 212, of which 31 had no named owner; the count of ownerless flags is the number the Mobile App team now reports, because retiring a flag needs someone whose job it is.

Recap. Remote configuration lets a fixed binary change its behavior, provided every flag has a safe compiled default and every kill switch has a known reach time. Experiments on devices keep the statistics of Chapter 4 and add their own plumbing: variants in the binary, stable cached assignment, exposure logged on sight and version as a covariate. Client crash and performance measures, defined here because no other book in the series defines them, set the halts of the release train. And the version floor decides when a client flag can finally be retired.

Exercise 8.4. Count the remote flags your app reads at launch. For each, record its owner, its default in the binary and the oldest supported version that reads it. How many have an owner and an expiry date? How many could you retire from the server today without breaking a version you still support?

8.5 Mobile at Many Teams

Modularization

A web storefront can be split among forty teams in several ways, as Chapter 7 showed. A native app offers fewer. In the end it is one binary, built by one pipeline and shipped on one train, and every team’s code shares its memory, its launch time and its crash rate. The tool that lets many teams work in one binary is the same one the Architecture book’s Section 1.3 recommended for any large system, modules whose boundaries match team boundaries, applied with rules the compiler can enforce.

The published accounts agree on the shape. Uber described its move to a single iOS repository in 2017, and by 2022 its Android app was built from more than 10,000 modules. Slack spent a year from February 2021 breaking its mobile codebases into modules of three kinds, features, services and libraries. Airbnb’s iOS engineers described in 2021 a scheme of module types with visibility rules, under which a feature module may not depend on another feature and the two talk only through small interface modules. The common rule is that features never import each other. They depend downward, on shared services and libraries, and sideways only through interfaces, so that a team can change its feature without rebuilding or breaking anyone else’s.

Encore’s fan app has a thin shell and 14 feature modules, each owned by one team: Discover, Search, Event, Waiting Room, Seat Map, Checkout, Payment Completion, My Tickets, Account, Inbox & Preferences, Help & Refunds, Presales, Festivals and Following. The shell, owned by the Mobile App team, holds navigation, the version check and the wallet ticket view, and very little else. Ownership is enforced where review happens, in a CODEOWNERS file at the root of the repository:

# Fan app: one owner per module; shell and build belong to Mobile App
/app-shell/                     @encore/mobile-app
/build-logic/  /ci/             @encore/mobile-app
/features/my-tickets/           @encore/fan-accounts
/features/checkout/             @encore/checkout
/features/payment-completion/   @encore/checkout
/features/inbox/                @encore/fan-messaging
/features/resale/               @encore/resale
/shared/ticket-core/            @encore/mobile-app @encore/venue-entry
/shared/design-system/          @encore/design-system

The ticket-core line deserves a word. The shared Kotlin Multiplatform module that renders tickets and computes rotating codes has two owners, because its behavior must match the validator in Venue Entry’s scanners exactly, and a change that one side reviews alone is a change that can turn fans away at the gate.

The mobile platform team

Every published account of mobile at scale ends up with a team whose users are the other mobile engineers; Orosz devotes a chapter to such teams. At Encore that role belongs to the Mobile App team, a stream-aligned team of nine in the fan group which, for the other module teams, behaves as a platform. The Software book’s Chapter 6 treats the internal platform as a product in full, and everything it says applies here. The charter is specific to mobile. The team owns the build and its speed; the continuous integration pipeline; the train and the release captain rotation; release tooling and the halt criteria; the version floor; the policy watch; and review of every new third-party SDK. It does not own features, because a platform team that also builds features is tempted, under deadline, to put its own first.

Build and test pipeline

Build time is the quiet tax on a large app. Encore’s median clean build of the fan app takes about 24 minutes on a developer laptop and its median incremental build about 90 seconds; both are reported monthly as developer-experience measures (the Software book, Section 6.5), and caching, sharding and continuous integration as a product are its Sections 4.2 and 4.4. Two points are particular to mobile. Real-device tests are slow and costly, so Encore’s device farm runs only the critical journeys, buying a ticket, opening it offline, listing it and receiving a reissued one, on the oldest and newest supported operating systems. And snapshot tests, which compare a rendered screen with a stored image, break whenever the operating system changes its fonts, so their simulator image is pinned and upgraded deliberately. Spotify lets teams with enough automated coverage opt out of manual regression, and Encore applies the same rule module by module.

Shared native design system

Chapter 7 described Encore’s design system, its tokens and its governance, and Chapter 6 the accessibility requirements, including the European Accessibility Act, which since 28 June 2025 covers consumer e-commerce apps as well as websites. On native platforms the Design System team shares the tokens with the web, colors, spacing and type scales generated from one source, and builds the components natively for each platform. It never wraps web components inside the app, because a wrapped web control behaves like neither platform. Where a platform’s conventions conflict with a token, the convention wins: navigation follows iOS and Android patterns, and text follows Dynamic Type on iOS and font scaling on Android, so that a fan who has set large text gets large text. Every component is tested with VoiceOver and TalkBack before release, and the governance is Chapter 7’s, unchanged.

Who owns a cross-cutting change

Verified Resale is the kind of change modules are designed to resist. It added a fifteenth module, Resale, and it reached into three that belong to other teams: the shell’s wallet ticket view, which the Mobile App team owns; the inbox and pass updates of Fan Messaging; and the Design System’s components for the Verified badge, the capped price and the reissue status. My Tickets and Checkout were touched only through interfaces they already exposed, so their teams were consulted and asked for nothing. Encore’s answer, which no textbook names but which fits the collaboration mode Skelton and Pais describe in Team Topologies, was to give the change one owner, the Resale team, and to ask each module owner for a time-boxed contribution: a reviewed interface for the Resale module to call, or a small change made by the owning team on an agreed train. Team design in general is the Software book’s Section 9.2, and cross-team planning its Section 10.2. The capstones of both books trace the same change from the product side and the engineering side.

What made the plan workable was that it was train-aware, which is to say laid out backward from the launch on the weekly calendar of Figure 8.4. The resale code shipped dark, behind the kill switches of Section 8.4, eight trains before launch, so that it could ride to fans’ phones well before anyone used it. The reissue-aware ticket view shipped first of all, because it had to precede the soft floor by enough weeks for adoption to begin. The soft floor rose three months before launch. Staff and the public beta exercised listing and buying on real events for four trains. Only then, with the code already on the great majority of phones and the floor already in place, was the flag turned on for three organizers in France. No train was held for resale, and no feature had to arrive on the day it was switched on.

Recap. Many teams can share one binary when modules match teams, features never import one another and CODEOWNERS enforces the boundaries. A mobile platform team owns the build, the train, the floor and the rules of the store, and never features. Build time, device tests and snapshot tests need deliberate care, and a native design system shares tokens with the web while building every component natively and accessibly. A change that crosses modules needs one owner, time-boxed help from the others and a plan laid out on the train’s calendar, with code shipped dark long before it is switched on.

Exercise 8.5. Measure your app’s median clean and incremental build times over the last month. Estimate the engineer-hours a year they cost, and what halving each would require. Then take your largest recent cross-module change and draw it on your release calendar: when did its code reach phones, and when was it switched on?

Kata: Unlocking Underground

Your brief: “Pedal.” Pedal runs bike sharing in 12 European cities, with about 40,000 bikes and 1.6 million monthly riders, and its app is the only way to unlock a bike. Eight mobile teams of about six engineers each build one native app on a fortnightly release that slips about one time in three, and a ninth team builds the lock firmware, which updates over the air and is just as hard to recall as the app. About 9% of unlocks happen in underground garages and stations with no signal, where the app currently waits for the server and times out; riders walk away, and support tickets peak every rainy Monday. Last spring a parsing bug in one version made the app crash whenever a city added a new bike type, and it took five weeks for crashes to fall below 1% of sessions, because the version floor had never been used and nobody knew who could raise it. The fleet team also wants stolen or reported bikes refused at the lock even when lock and phone are both offline. Pedal’s leadership is debating a cross-platform rewrite.

Deliverables:

1. Channel map and decision record — Map every rider job (find, unlock, pay, end the ride, report a fault) to its owning channel and fallback, as in Figure 8.2. Write the ADR for how Pedal builds its app, including what would be shared between platforms and the reversal conditions that would reopen the decision.
2. Offline unlock design — Design an unlock that works with no network on the phone or the lock: what secret each holds, how time is handled, and how long a pre-delivered credential lives. Say how a stolen bike is revoked when nothing is online, and how the window between revocation and the lock’s next sync is kept short.
3. Release train and halt criteria — Lay out a weekly train with a branch cut, beta tracks and staged rollout on both stores. Write halt criteria that include the unlock success rate, offline and online, alongside the store’s crash and responsiveness lines.
4. Version floor and flag policy — Write who may raise the soft and hard floors and for what reasons, and what a rider below the hard floor can still do. Write a flag manifest entry, like the one in Section 8.4, for the new bike-type parser, with its expiry tied to the floor.
5. Firmware and app together — Explain how an app release and a lock firmware release that depend on each other are sequenced so that neither ever has to arrive before the other.

Rubric:

Where you now stand

You can now map a product’s jobs to channels with a fallback for each, choose how to build an app and say in advance what would change your mind, run a release train whose halts include the job that matters most, and set a version floor that strands as few people as possible. You can design an artifact that works offline, say honestly what its security buys, keep guarantees out of best-effort channels, and organize many teams around one binary. One question has hovered at the edge of the chapter. Rowmate hands fans to Encore through deep links, partners list tickets through Encore’s interfaces, and the fan app itself is, as the Architecture book put it, a public API consumer that happens to be Encore’s own code. Interfaces that others build on are products in their own right, with users, versioning promises and support, and Chapter 9 takes them up.

References

• Gergely Orosz — Building Mobile Apps at Scale: 39 Engineering Challenges. The Pragmatic Engineer, 2021.
• Chuck Rossi, Elisa Shibley, Shi Su, Kent Beck, Tony Savor, Michael Stumm — Continuous Deployment of Mobile Software at Facebook (Showcase). Proceedings of FSE 2016, ACM, 2016 — free online.
• Jacob Vesterlund — A Behind-the-Scenes Look at How We Release the Spotify App (Part 1). Spotify Engineering, 2025 — free online.
• Ron Kohavi, Diane Tang, Ya Xu — Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press, 2020.
• Mustafa Ali — Native is now the future of mobile at Shopify. Shopify Engineering, 2026 — free online.
• Farhan Thawar — React Native is the Future of Mobile at Shopify. Shopify Engineering, 2020 — free online.
• Gabriel Peal — Sunsetting React Native. Airbnb Engineering, 2018 — free online.
• Apple — Release a version update in phases and App Store Review Guidelines. Apple Developer documentation — free online.
• Google — Release app updates with staged rollouts and Android vitals. Google Play and Android documentation — free online.
• Apple — Adding a Web Service to Update Passes. Apple Developer documentation — free online.
• Google — Rotating Barcodes. Google Wallet documentation — free online.
• conduition — Reverse Engineering TicketMaster’s Rotating Barcodes (SafeTix). conduition.io, 2024 — free online.
• D. M’Raihi, S. Machani, M. Pei, J. Rydell — RFC 6238: TOTP: Time-Based One-Time Password Algorithm. IETF, 2011 — free online.
• Ryan Brooks — A Deep Dive into Airbnb’s Server-Driven UI System. Airbnb Engineering, 2021 — free online.
• Firebase — Remote Config loading strategies. Firebase documentation — free online.
• Apple — iPhone App Store Downloads Top 10 Million in First Weekend. Apple Newsroom, 14 July 2008 — free online.