Building Software at Scale in the AI & Cloud Era

← the whole book on one page

Chapter 4: The Development Loop — From Laptop to Main

A branch is a promise to merge, and the promise grows heavier every day it is left unkept.

Every change to Encore begins in one engineer’s head and ends in the main line of a repository, where it becomes everyone’s problem. Between those two points lies a short journey repeated many thousands of times a week: the engineer edits code, runs it, tests it, asks a colleague to look at it, waits for a machine to build and check it, and merges it. Each step is small and familiar. Repeated by 400 people, they make up the development loop, the cycle by which a change travels from an engineer’s laptop to the main line, and the speed and safety of that loop set a ceiling on everything else the engineering system can do. A company whose loop takes an hour can try an idea before lunch and abandon it after. A company whose loop takes a week has, without anyone deciding so, committed to fewer and larger changes, which are slower to review, harder to integrate and riskier to ship. The question of this chapter is how a large organization keeps that loop short as it grows, and why tools alone have never been enough to do it.

The modern history of the loop turns on a few days in April 2005, when Linus Torvalds, having lost the free license for the version-control system the Linux kernel had used for three years, began writing his own. He called it Git. Within days it was managing its own source, and within a few years it had made branching so cheap that the industry came to think of a branch as costing nothing. What Git did not make cheap, and could not, was the integration of one person’s work with everyone else’s. The industry spent the better part of a decade learning the difference, and the practices this chapter teaches are largely what it learned.

Encore is in the middle of that lesson. Its new Resale team, formed in month four of Verified Resale from engineers drawn from Orders & Payments and Partner APIs, starts its service from the golden-path template and within weeks meets every slow step in Encore’s loop at once. Its engineers need Seat Inventory and Notifications to test anything, and nobody can run Encore on a laptop. Their changes to the reissue path need reviewers from Inventory and Orders & Payments, and those reviews stall for two days. Every change to barcode issuance waits 31 minutes for continuous integration on encore-core, Encore’s oldest repository. And no fan may see the product until month 12, so months of unfinished work must live somewhere without rotting. The five sections that follow take these problems in turn.

4.1 The Inner Loop

Ten days in April 2005

For its first decade the Linux kernel had no version-control system at all. From 1991 to 2002 changes travelled as patches and tarballs, mailed to Torvalds and applied by hand: one person acting as the integration point for thousands of contributors, with his own attention as the only merge tool. In 2002 Torvalds moved the kernel to BitKeeper, a proprietary distributed system made by Larry McVoy’s company BitMover, which offered it free of charge to open-source developers. The choice was controversial; some kernel developers refused to use a proprietary tool, and Richard Stallman objected publicly. But BitKeeper changed how the kernel was built. Every developer’s tree was a full repository, branching was cheap, and changes moved between trees by pulling. The maintainers learned habits from it that they would never give up.

The arrangement ended in the spring of 2005. Andrew Tridgell, who worked on unrelated projects at the Open Source Development Labs (OSDL), Torvalds’s employer at the time, wrote a client that could read BitKeeper’s repository metadata, and BitMover, treating this as reverse engineering, withdrew the free license. On 6 April 2005 Torvalds told the linux-kernel mailing list, in a message titled “Kernel SCM saga..”, that he would stop using BitKeeper. He was generous to its owners: “don’t blame BitMover,” he wrote, and of the tool: “It didn’t work out, but it sure as hell made a big difference to kernel development.” The message did not announce Git; its postscript pointed readers, if they must, at Monotone.

He had already started. By the commonly cited chronology, work began on 3 April, and on 7 April Git committed its own source for the first time, with the message “Initial revision of “git”, the information manager from hell.” The first merge of several branches followed on 18 April, Git managed the 2.6.12 kernel release on 16 June, Torvalds handed maintenance to Junio Hamano on 26 July, and version 1.0 appeared on 21 December 2005. Figure 4.1 sets the dates out. The sequence is often compressed into the legend that Git was “written in two weeks.” The defensible version is more interesting: it hosted itself within days and ran a kernel release within about 10 weeks, because its author designed it around a few demanding goals and nothing else. His rule was to take CVS, the older system much of the industry still used, “as an example of what not to do; if in doubt, make the exact opposite decision.” The goals Pro Git records are speed, a simple design, strong support for non-linear development with thousands of parallel branches, full distribution, and efficiency on very large projects.

Figure 4.1 — Git’s first year, with April drawn enlarged. Notice how little separates the break with BitKeeper from Git hosting itself, and how much longer the road ran from there to a kernel release and a 1.0.

Git did not invent cheap branching. BitKeeper had it, and so did Monotone and darcs, and Olivia Mackall (then publishing as Matt Mackall) announced Mercurial on 19 April 2005, for the same reason. What Git did was make cheap branching fast at the scale of the kernel. In Git, as Pro Git puts it, “Creating a new branch is as quick and simple as writing 41 bytes to a file (40 characters and a newline).” Commits, diffs and switches between branches happen on the engineer’s own machine in a fraction of a second. When GitHub launched in 2008 and made the pull request the unit in which most of the industry proposes a change, the loop Git had made fast became the loop nearly everyone worked in. (Version control itself is much older; Marc Rochkind’s SCCS came out of Bell Labs in the early 1970s, as Chapter 6 tells.)

Git belongs at the head of this section because it is, above all, a tool for the inner loop. Its speed was a design goal, chosen by a man who integrated other people’s work all day and could not afford to wait on his tools, and it teaches the section’s first lesson: the time a developer waits between making a change and learning whether it works is a quantity worth engineering.

Why the inner loop decides productivity

The inner loop is the part of the development loop that one engineer runs alone, many times an hour: edit the code, build it, run it or its tests, look at the result, and edit again. The outer loop is everything that involves other people or shared machinery: review, continuous integration, merging, deployment. The distinction matters because the two loops have different economics. The outer loop runs a few times a day for each change, and its waits can be scheduled around. The inner loop runs dozens of times an hour, and every second it costs is paid again on each turn.

Chapter 1 introduced the three dimensions of developer experience that Abi Noda, Margaret-Anne Storey, Nicole Forsgren and Michaela Greiler proposed in 2023: feedback loops, cognitive load and flow state. Their framework synthesizes earlier qualitative research and makes no causal claim, and Chapter 6 uses it to measure Encore’s platform. Here we need only the first dimension. A feedback loop is the time it takes a developer to learn whether an action had the effect intended, and the inner loop is the feedback loop that runs most often. Practitioners aim for an edit-build-test cycle measured in seconds. Nobody has measured the effect of that target directly, but the reasoning is sound and old. A loop of a few seconds keeps the problem in the engineer’s head. A loop of a few minutes invites a glance at messages, and the glance becomes a context switch that costs more than the wait. A loop of half an hour changes how people work: they batch several changes into one build, so that when it fails they no longer know which change broke it.

A musician practicing a difficult passage shows the same economics. She plays four bars, hears the wrong note and plays them again, and the speed of that cycle is most of what practice is. If she had to wait 10 minutes after each attempt to hear a recording, she would practice far less, and she would play longer passages between recordings to make each wait worthwhile, which is exactly the habit that makes mistakes hard to find. The analogy breaks at one useful point. The musician’s instrument is complete in her hands. A developer’s program is not: at Encore it depends on services owned by other teams, running elsewhere, and the inner loop is only as fast as the slowest of those it must touch. Most of the work of a fast inner loop at scale is deciding what the developer does not need to run.

Local environments

The first condition of a fast inner loop is a working checkout, and it fails in familiar ways: a toolchain version that differs from CI’s, a database installed by hand, environment variables known only to the team’s longest-serving member, a setup document last accurate two years ago. Each failure is paid again by every new joiner and every engineer who moves between repositories. The remedy is long practice, though its published evidence is mostly company case reports: a reproducible development environment, defined in the repository itself, that brings a fresh machine to a working, tested checkout with one command. Dev containers, an open specification many editors support, are one common form; pinned toolchain files and a bootstrap script are another.

GitHub’s 2021 account of moving its own engineers to Codespaces is the most detailed public case, and since GitHub sells Codespaces it should be read with that in mind. Its lesson is independent of the product. GitHub’s main repository was almost 13 GB on disk, and bootstrapping an environment for it took “upwards of 45 minutes.” A shallow clone cut cloning from about 20 minutes to 90 seconds, caching brought creation to about 5 minutes, and prebuilding environments ahead of demand brought it to about 10 seconds. The step that mattered was doing the slow work before anyone asked for it, and that needs an owner, who keeps the prebuilt environment current and measures the time from a fresh machine to a first successful build, a number that decays quietly as a repository grows.

Encore’s golden-path template gives every new service such an environment; keeping templates current is a platform problem for Chapter 6, and the onboarding test of a first commit in the first week belongs to Chapter 9. The Resale team found the difference at once. Its own service, born from the template, was running in minutes. encore-core needed a page of instructions and an afternoon, for reasons that run through the rest of this chapter.

Running against a fleet

The harder problem is the one the musician never faces. To list a ticket, the Resale service must ask Seat Inventory whether the seller bought it through the Gate; to sell one, it must screen the buyer with Bot Screening, run a purchase saga in Orders & Payments and ask Inventory to reissue; afterwards it must tell Notifications to replace two wallet passes. An engineer working on any of this has three broad ways to give the code its neighbours, and each trades cost against fidelity.

The first is to run local stubs or fakes, small stand-ins that answer like the real service for the cases the developer needs. Chapter 3 set Encore’s rule that the team owning a service ships its fake with the service and keeps it honest with contract tests (Sections 3.1 and 3.3), so the Resale team runs the Seat Inventory fake that Inventory ships from encore-core instead of writing its own guess. The second is to call shared remote dependencies, real instances running in a shared environment such as staging, which is cheap and faithful as long as that environment is stable. The third is a per-change environment in the cloud, the next subsection’s subject.

The most instructive public account of choosing among these comes from Lyft, which published a four-part series in 2021. Lyft grew from about 100 engineers to more than 1,000, and for years its developers used Onebox, a Docker-based environment that ran a full copy of the stack for each developer, later on large cloud instances. By Lyft’s own account, Onebox “could not scale anymore.” Lyft moved to local-first testing of a single service with its dependencies stubbed, and to isolating requests inside a shared staging environment where real neighbours were needed. It removed or converted thousands of integration tests (Section 3.4 tells that half of the story), and pull requests became, in Lyft’s words, “ready to merge in minutes.”

The Resale team settled on a mixture, and Figure 4.2 sets out the options as Encore weighs them.

Figure 4.2 — Ways to give a developer’s code its neighbours, ordered from cheapest to most expensive. Fidelity rises down the table, and so does cost; the last row is where the cost outruns what fidelity can repay.

The file that defines the Resale team’s inner loop is short, and it is worth reading because every line records a decision about what to run where.

# resale/dev.yaml — what `encore dev up` starts for the Resale service
service: resale
laptop_budget: 10              # Encore's ten-container rule
local:
  - resale                     # the service itself, rebuilt on save
  - postgres                   # its own database, seeded from fixtures
  - event-broker               # single node, carries ticket-sales.v2
  - seat-inventory-fake        # shipped by Inventory, pinned to contract v14
  - bot-screening-stub         # "allow" unless a test scripts a verdict
remote:
  notifications: staging       # Fan Messaging's shared instance
  orders-payments: staging     # real saga, test PSP behind it

Five containers on the laptop, two services reached remotely, and none of the others. Notifications is remote because Fan Messaging keeps a stable shared instance and the Resale engineers mostly need to see that a message would be sent; a remote call costs less than a fake someone would have to keep faithful.

Ephemeral environments

Some questions cannot be answered from a laptop. Does a resale purchase, run through the real saga, make Inventory reissue the ticket, put the old identity on Venue Entry’s revocation feed and replace the seller’s pass? That journey crosses four teams’ services, any of which may be changing at the same moment. For such questions teams use ephemeral environments: short-lived environments, created for a change and destroyed when it merges, in which the change runs against real services.

The obvious design gives each pull request a full slice of the system. It works for a handful of services and fails for hundreds, for the reason Onebox did: the slice grows with the fleet, and so do its cost and the work of keeping its data and versions consistent. The large published accounts converge on another design. Lyft’s staging overrides (2021) and Uber’s SLATE (2022) deploy only the services that changed into a shared, continuously updated baseline, and route requests carrying a marker to the changed versions while everything else goes to the baseline. Lyft carried the marker as baggage in the request and routed on it in its Envoy mesh, keeping the offloaded deployments out of ordinary service discovery so that no untagged request could reach them. The conditions are specific: every service must propagate the request’s context, the mesh must be able to route on it, and the deployed versions must be torn down on merge.

Encore already pays for preview environments; the Resilience book’s Section 2.5 counts them among the build-pipeline costs in the cost per ticket. What changed in year nine was their shape. Encore’s services already propagate the W3C traceparent header (the Architecture book, Section 5.3) and its mesh can route on headers, so the Runtime team moved previews to the request-routed form: a preview deploys only the changed services into the staging cell and receives the requests that carry its key. One key can now cover changes in several repositories, which is what the resale journey needs, since its changes land in the Resale service, in encore-core and in Venue Entry’s revocation feed. Encore keeps a full slice only for the rare journey whose test data cannot be isolated inside the shared baseline. Figure 4.3 follows one resale request through such a preview.

Figure 4.3 — A request-routed preview for the resale journey. Only the three changed services are deployed; a request carrying the preview key reaches them, and falls through to the shared baseline wherever, as with Notifications, nothing changed.

The laptop budget

One line in the Resale team’s file records a choice that is Encore’s own. laptop_budget: 10 refers to the ten-container rule: no service’s inner loop may require more than ten containers on a developer’s machine, and when a service would need more, the platform owes its developers a fake or a remote option for the excess. I know of no published precedent for a written limit of this kind, and I offer it as Encore’s heuristic. Lyft’s Onebox shows what happens without one. The containers a laptop must run grow by one whenever a service gains a dependency, nobody notices any single addition, and by the time the loop is unbearable, reversing it is a project. A written budget turns a drift nobody sees into a line somebody crosses. The number ten is a judgment about Encore’s laptops and services; what matters is that the limit is written down, checked when encore dev up runs, and treated by the platform group as its own problem when breached.

Recap. Git made the inner loop fast because its author treated speed as a design goal, and the same goal applies to every step a developer repeats many times an hour. A reproducible environment, prebuilt and owned, gets a developer to a working checkout; a deliberate mixture of stubs, fakes, shared remotes and request-routed previews gives the code its neighbours without running the fleet. Encore’s ten-container rule is its own way of keeping that mixture from drifting back toward the full stack that failed at Lyft.

Exercise 4.1. Time your own edit-to-feedback loop for a one-line change in a service you work on, from saving the file to seeing a test result you trust. Write down each step and its duration, name the slowest, and say which option in Figure 4.2 would remove it.

4.2 Repositories and Builds

Trunk-based development

Git made a branch cost 41 bytes. It did not change what a branch is: a promise to integrate later. The longer a branch lives, the more the main line moves beneath it, and the final merge must reconcile two sets of assumptions as well as two texts. Torvalds made the point himself in a talk at Google in 2007: branches are worthless unless they are merged, and merging was the problem Git had to solve well. Martin Fowler’s 2020 essay on branching patterns puts it in one line: “branching is easy, merging is harder.” Git merges text very well. It cannot tell whether two changes, each correct alone, still work together; Fowler calls that gap a semantic conflict, a pair of changes that merge cleanly and “may build but fail at run-time.”

The response is trunk-based development: every engineer integrates into a single shared main line, the trunk, at least once a day, and branches, where they exist at all, live for hours or a day or two. DORA’s research, reported in Accelerate (2018) from survey data gathered in 2016 and 2017, found higher delivery performance in teams that, in the words of DORA’s capability page, have “three or fewer active branches” that merge to trunk “at least once a day” and have no code freezes or integration phases. Software Engineering at Google is blunt: “Long-lived dev branches are not a good default plan.” Google’s own monorepo was shared in 2020 by roughly 50,000 engineers making 60,000 to 70,000 commits a working day. Fowler’s rule of thumb is the most practical form: “you should never have more than a day’s work sitting unintegrated in your local repository.”

Much of the industry spent the decade after Git going the other way, and the most honest account of that detour comes from one of the people who led it. In January 2010 Vincent Driessen published “A successful Git branching model,” describing git-flow: long-lived develop and main branches, feature branches that could live for weeks, and release and hotfix branches between them. It was clear and handsomely drawn, and it became, in Driessen’s later words, “a dogma or panacea” for teams it did not suit. In March 2020 he added a note at the top of the post: “If your team is doing continuous delivery of software, I would suggest to adopt a much simpler workflow (like GitHub flow) instead of trying to shoehorn git-flow into your team.” Few authors of a popular practice retract it so plainly, and the retraction carries weight because its author watched the practice spread for ten years. Git-flow still suits software released in versions, such as a library or a desktop application. For services deployed many times a day, its long-lived branches only postpone the integration that trunk-based development performs daily.

A newsroom shows why one trunk works better than many. Reporters do not produce private newspapers and merge them at the end of the week. They file to one copy desk, all day, for the one edition printed tonight, and stories not yet ready are kept off the page without being kept out of the system, so two reporters covering the same event are found out the same day. The analogy breaks because a newspaper starts fresh each morning and a codebase never does. Yesterday’s code constrains every change made today, which is why software must also keep trunk releasable at all times, and why Section 4.5’s flags exist.

Encore has worked this way since the golden path arrived. The practice has one pitfall that DORA names on its capability page, and it leads to Section 4.3: trunk-based development suffers when review is slow, because a change waiting two days for a reviewer is a two-day branch, whatever the team intends.

Monorepo or polyrepo

The second choice is how many repositories to have. A monorepo keeps the code of many projects, often of a whole company, in one repository; a polyrepo arrangement gives each project, usually each service, its own. Both work at large scale, and both have documented costs.

Google is the famous monorepo. Rachel Potvin and Josh Levenberg described it in 2016: as of January 2015, about two billion lines of code in about nine million source files, 86 TB of data and about 35 million commits, developed on trunk. What makes it work is investment: a custom version-control system, a build system that understands the whole dependency graph, code search across everything and ownership files on every directory. A 2018 survey at Google by Ciera Jaspan and colleagues, one of the few studies of engineers who had worked both ways, found that the benefit they valued most was visibility: searching all the code, finding examples, and updating every caller of an API in the same change. What they valued in multi-repository setups was freedom of toolchain, finer access control and stability. Google’s authors are candid: “no, we don’t think the monorepo approach as we’ve described it is the perfect answer for everyone.” Microsoft’s move of Windows to Git in 2017, about 3.5 million files and 300 GB for about 4,000 engineers, needed a virtual file system built for the purpose, which shows the condition plainly: a monorepo at scale needs infrastructure that ordinary tools do not provide.

Encore is polyrepo for its services. About one repository per service, each generated from the golden-path template, holds most of the backend, and a few repositories hold shared libraries. The fan app has its own repository, with its shell and 14 modules; the web frontend is a monorepo, for reasons the Architecture book’s Section 8.2 gave and the Products book’s Chapter 7 develops; and the desired state of every environment lives in encore-deploy (the Resilience book, Section 3.4). Then there is encore-core, the year-one monolith’s repository and the shared repository of Chapter 1. It still holds Seat Inventory & Ticketing, with barcode issuance inside it, the Orders & Payments saga and some shared libraries, and its CI takes 31 minutes, against about 6 for a golden-path service.

Encore’s reasons for staying polyrepo are the ones Jaspan’s survey records: teams own their pipelines, one service’s broken build cannot block another, and Encore has never had the platform staff to build Google’s tooling. The cost is the other side of the survey. A change that spans services spans repositories, needs several merges in the right order and cannot be atomic. Nor does one commit give Encore a single version of every dependency. It approaches the One-Version Rule by the route Section 2.5 described: a shared version catalog in every service’s build, upgrade changes opened by a bot, and the team that owns a catalog entry responsible for moving every caller. That works, but through many merges over days where a monorepo would need one, and Encore accepts the slower convergence in exchange for independent pipelines.

Build systems

Beneath the repository lies the build, where the inner and outer loops slow down together. Software Engineering at Google draws the distinction that matters in a chapter by Erik Kuefler. In a task-based build, the engineer writes tasks (compile this, copy that, run these tests) and the tool runs them in order. Such builds give “too much power to engineers and not enough power to the system”: a task can do anything, so the tool cannot know what it depends on or whether its output can be reused. In an artifact-based build, the engineer declares the targets and their inputs in “a declarative manifest,” and the system decides how to build them. Because every input is declared, the system knows what a change affects, and because each action runs isolated in a sandbox, the same inputs produce the same output on any machine. That property, hermeticity, is what makes a build cacheable and trustworthy. Google’s internal system was released as open source in 2015 as Bazel.

The costs deserve plain statement. Moving a large existing codebase to an artifact-based build means declaring every target, finding every hidden dependency and fixing every non-hermetic test. Below some size, a well-kept task-based build with good caching is enough. Encore’s golden-path services build in about 6 minutes with the template’s task-based tool, and migrating them would buy little. The problem was encore-core, an eight-year-old Kotlin codebase whose Gradle build had grown by accretion: its tests reached the network, a code-generation step wrote into the source tree, and nobody could say with confidence what a given change might affect.

Remote caching and execution

A remote cache stores the outputs of build actions, keyed by a hash of their inputs, so that any machine, laptop or CI runner, can download an output already built elsewhere instead of rebuilding it. Remote execution goes further and runs the actions on a shared farm. Both depend on hermeticity: a cache over a build with undeclared inputs will one day hand someone a wrong result, which is worse than a slow one because nobody suspects it.

The best first-party account of what this buys comes from Uber’s Go monorepo, described in 2022: about 500,000 files, more than 10 million Bazel targets, and about 50 changes an hour in working hours. Moving the remote cache to Buildfarm in 2021 “cut down the build time by about half,” and the 95th-percentile time, which had fluctuated between 40 and more than 100 minutes, fell consistently below 40. Uber also computes, for each change, which targets it alters, in about 5 minutes; this is the dependency-graph test selection Section 4.4 returns to.

Encore’s version is smaller. It did not move encore-core to Bazel. Instead, the Developer Portal team and two engineers from Inventory spent most of months five to eight making the existing build hermetic enough to cache: pinning the toolchain, declaring the code generator’s inputs, cutting the tests off from the network and splitting the modules so that the build graph reflected which code depended on which. Then they added a remote cache shared by CI and laptops, and dependency-graph test selection. Figure 4.4 shows where the 31 minutes went.

Figure 4.4 — Continuous integration on encore-core, step by step, before and after months five to eight, from 31 minutes to 9. Test selection and the remote cache account for 17 of the 22 minutes saved; neither would have been safe before the build was made hermetic.

The largest saving came from running fewer tests, which is safe only because the graph that decides which tests to skip can now be trusted. The 9 minutes remain above the golden path’s 6 because encore-core is still large, and Encore expects to close the gap by moving code out of it over time, having judged that a different build system would cost more than it returned.

Ownership in the repository

A repository also records who owns what. Chapter 1 showed encore-core’s CODEOWNERS file, in which barcode issuance belonged to Inventory and the shared library libs/http-client had had no owner since year five. GitHub introduced CODEOWNERS files in July 2017, inspired by Chromium’s OWNERS files, and they can require an owner’s approval before a change merges; Google’s OWNERS files, older and stricter, guard every directory.

Ownership files help only if they match the organization, which is Conway’s law seen from the repository (the Architecture book’s Section 5.5 is its home). A file that names a dissolved team, a departed person or nobody sends review requests nowhere. Encore now runs a CI check that fails when a path lacks an owner or names one that does not exist in the service registry the Operations book’s Section 7.2 describes. Its first run on encore-core found 41 such paths. The HTTP client library went to the Web & API Paths team as an interim owner, and Chapter 8 asks whether a library every service uses should be run as InnerSource instead. The check takes seconds and removes the stalled review in which nobody knew whom to ask.

Recap. Git made branches cheap and left integration as hard as ever, which is why trunk-based development, with branches measured in hours, remains the proven way to integrate, and why git-flow’s own author now recommends something simpler for continuously delivered software. Monorepo and polyrepo both work at scale, each with costs that only tooling pays; Encore stays polyrepo and approaches a single version through a shared catalog. Hermetic builds make caching and test selection safe, and they took encore-core from 31 minutes to 9, while ownership files that match the team map keep reviews from going nowhere.

Exercise 4.2. Take one feature your organization shipped recently. Count the repositories it touched, the merges it took to land, and the days between the first commit and the last merge. Then find the longest-lived branch involved and ask what it was waiting for.

4.3 Code Review at Scale

What review is for

Code review is older than the tools that host it. In 1976 Michael Fagan of IBM described formal design and code inspections, meetings with defined roles in which a team read code line by line to find defects before testing. The form most teams use today, modern code review, is lighter: an author proposes a change in a tool, colleagues read the diff and comment asynchronously, and the change merges when they approve. It is far more frequent than an inspection, and that changes what it is for.

The best evidence on what it does comes from a 2013 study at Microsoft by Alberto Bacchelli and Christian Bird, who surveyed 165 managers and 873 programmers, interviewed 17 developers and classified 570 review comments. Finding defects was the motivation most often named first, by 44% of the programmers. Yet only 14% of the comments concerned defects, the fourth most common of nine categories, while the most common, at 29%, were improvements to the code. The authors concluded that reviews are “less about defects than expected” and also bring knowledge transfer, team awareness and alternative solutions. Engineers say they review to find defects, but most of what reviews actually produce is better code and shared understanding. The hardest part of reviewing, the study found, was understanding the change and its context.

Google’s account, published in 2018 by Caitlin Sadowski and colleagues from interviews, a survey and the logs of some nine million changes, names four expectations of review: education, maintaining norms, gatekeeping and accident prevention. Review began at Google as an educational tool. The four belong together, because a team that treats review only as gatekeeping designs it for suspicion, and one that treats it only as education lets dangerous changes through politely.

Publishing divides a related job between two people. The editor reads for sense: does the argument hold, does the author mean what the sentence says? The copy editor reads for consistency: spelling, house style, the numbering of figures. No good publisher spends the editor’s scarcer attention on the copy editor’s work, and review at scale divides the same way, with tools as the copy editor and people as the editor. The analogy breaks where it matters. A book’s editor serves the book, and once it is published the editor’s knowledge of it has done its job. A reviewer’s knowledge of a change is itself one of review’s products, because the reviewer will be among those who must understand that code next year.

Small changes, fast reviews

The strongest single lever on review is the size of the change. Sadowski’s study gives Google’s numbers. The median change is 24 lines; more than 35% of changes touch a single file and about 90% fewer than 10. The median wait for a first response is under an hour for small changes and about 5 hours for very large ones, and the median whole review takes under 4 hours, against 14.7 to 19.8 hours in earlier studies at AMD, Chrome OS and Microsoft. The median change has one reviewer, and more than 80% need at most one round of revision. Google’s guidance for engineers states the target: “100 lines is usually a reasonable size for a CL, and 1000 lines is usually too large” (a CL, or changelist, is Google’s word for a change).

Size works so strongly because a reviewer opening a 600-line change must first build a model of what it is trying to do, which takes a block of uninterrupted time the reviewer may not have until tomorrow, while a 60-line change can be understood in the gap between two tasks. Small changes are reviewed sooner because they fit into the day.

Real work does not come in 60-line pieces, and the practitioners’ answer is stacked changes: a sequence of small dependent changes, each reviewed separately and each building on the one before, so the author keeps working while the first is in review. Stacking is the norm at Meta and common at Google. It needs tool support, since stacking by hand in Git is error-prone, and it pays most where review takes longer than writing.

Review latency as a team metric

The Resale team’s first changes to the reissue path bore out Chapter 1’s survey. Verified Resale’s changes touch seats and money, so under the Resilience book’s review tiers (Section 4.5) they are Tier 1: two reviewers, one of them the domain owner. For barcode issuance the domain owner was Inventory, and in practice it was Lena, the last of the module’s original authors, who had approved every change to it for three years. For the purchase saga it was Orders & Payments. In its first weeks, months four and five, the team’s typical change was about 600 lines across the Resale service and encore-core, and the median time from opening a change to merging it was a little over two working days. Nobody was slow on purpose. The changes were large, they arrived unannounced in other teams’ queues, and the one person who could approve the most important of them was also doing her own team’s work.

DORA’s 2023 report gives the general finding: “Teams with shorter code review times have 50% better software delivery performance.” It is a correlation from survey data and does not say which way causation runs. The practical question is how to shorten review without making it worse, and Meta’s 2022 account is the most useful answer. Meta measures Time In Review, the time a change spends waiting on reviewers, at the 75th percentile, because the slow tail is where frustration lives; it found “a correlation between slow diff review times (P75) and engineer dissatisfaction.” A bot that nudges reviewers of overdue changes cut average Time In Review by 7% and the share of changes waiting more than three days by 12%. Meta also set a guardrail, “Eyeball Time,” the time reviewers actually spent looking at a change, so that speed could not be bought by rubber-stamping.

Speed must not destroy the reviewer’s flow either. Google’s guidance holds both sides: “One business day is the maximum time it should take to respond to a code review request,” and “If you are in the middle of a focused task, such as writing code, don’t interrupt yourself to do a code review.” The reconciling rule is to review at the next natural break and to make sure breaks come often enough that nobody waits more than a day.

Encore’s fix had four parts, and none was a new tool. The Resale team split its work into stacked changes of under about 200 lines, each description naming its step in the reissue path. The teams involved agreed review latency targets: a first response within four working hours, and Time In Review at the 75th percentile under one working day, which each team sees on its own scorecard and nobody uses to rank teams. They adopted Meta’s guardrail in a simple form, flagging Tier 1 changes over 50 lines approved within two minutes of being opened. And, most important, Inventory widened the circle of domain owners for barcode issuance. From month five two Inventory engineers, one of them Jonas, paired with Lena on every barcode change; DORA notes that pair-programmed code can count as reviewed, and the teams agreed to record those sessions as the review itself. Chapter 2 and Chapter 11 describe the larger work of raising the module’s truck factor; review was one of its instruments. By month eight the median time from opening a change to merging it was about seven working hours, and the P75 Time In Review was under a day.

Reviewers and owners

Google asks three things of every change: “A correctness and comprehension check from another engineer,” “Approval from one of the code owners,” and “Approval from someone with language ‘readability’.” One person can provide all three. The structure separates three questions that blur easily: is the change correct and understandable, does the person accountable for this code accept it, and can the organization maintain it? Google’s readability certification is costly and peculiar to Google, and I would not import it whole, but its idea, that readable code is a skill taught through review, travels well. A human reviewer should look for what tools cannot see: whether the change does what its description says, whether that is the right thing to do, whether its design fits, and whether its tests would catch the failure that matters.

Not every change needs to wait for review. Rouan Wilsenach’s “Ship / Show / Ask” (2021) names three options: ship merges directly under CI, show merges and invites review afterwards, and ask waits for review. It needs strong CI and is excluded for anything in Tier 1. At Encore a documentation fix or a dashboard change may be shipped or shown; anything that touches seats or money is always asked.

Automation in review

The copy editor of code review is static analysis, and Google’s Tricorder, described by Sadowski and colleagues in 2015, is the most instructive account. It runs more than 100 analyzers in more than 30 languages on more than 50,000 review changes a day, and its findings appear as comments inside the review tool while the reviewer is reading. Two rules explain why engineers trust it. New checks must “Produce less than 10% effective false positives,” and the overall rate stays “just below 5%.” And every comment carries a “Not useful” button that files a bug against the analyzer, so the authors of a check hear when it wastes someone’s time.

The same checks belong earlier still, in the editor or a pre-commit hook, configured from the same file as CI, so that most of what CI would catch is caught before a push. When local and CI configuration differ, developers learn that a clean local run means nothing and stop running it. As AI assistants write more code, this division of labor matters more, since human attention is what does not scale; Chapter 5 takes that up.

Recap. Modern code review produces better code and shared understanding more often than it finds defects, and Google names four purposes that a review practice should serve together. Size is the strongest lever on review latency, and stacked changes keep work moving without large changes. Encore cut its cross-team review wait from two days to under one by making changes smaller, setting latency targets each team sees for itself, guarding against rubber-stamping, and widening the circle of owners, while tools took over everything a tool can check.

Exercise 4.3. Export your team’s last 100 merged changes with their size in lines and their time from opening to first response and to approval. Plot latency against size. Then find the five slowest changes and, for each, say whether size, a missing owner, a cross-team reviewer or something else explains the wait.

4.4 Continuous Integration as a Product

A practice before a server

Continuous integration is often confused with the server that runs it. The Resilience book’s Section 4.2 traces its history; we need only the definition. In the 2024 revision of his essay, first published in 2000, Martin Fowler defines it as each member of a team merging “their changes into a codebase together with their colleagues changes at least daily,” each integration verified by an automated build. A CI server running on long-lived feature branches is the server without the practice. The practice’s second half is easy to forget: when main breaks, fixing it comes first. Fowler quotes Kent Beck, “nobody has a higher priority task than fixing the build,” and gives the quickest remedy: “The best way to fix the build is to revert the faulty commit from the mainline.”

The commit stage

The first stage of the pipeline, the one a developer waits on after every push, is the commit stage: compile, run the fast tests and static analysis, and build the artifact every later stage will use. Jez Humble and David Farley give its budget in Continuous Delivery (2010): “Ideally, a commit stage should take less than five minutes to run, and certainly no more than ten.” The Resilience book’s Section 4.2 takes the pipeline onward; this chapter owns the commit stage from the developer’s side, what keeps it fast and green.

The golden path’s commit stage takes about 6 minutes. encore-core’s took 31, which is why 54% of Encore’s engineers named it among their three worst delays. A 31-minute commit stage costs more than the wait. Engineers batch changes to avoid paying it twice, so changes grow and reviews slow. They push and start something else, so a failure is noticed an hour later, with the context gone. And they merge before CI finishes, on the reasonable belief that it will probably pass, which makes main the place where failures are found.

Test selection

There are two ways to choose which tests to run. Dependency-graph test selection runs only the tests whose code depends, directly or transitively, on what a change modified. It is proven at scale in Google’s continuous testing system, TAP, which aims to “run the minimal set to be sure the change is safe,” and in Uber’s per-change calculation of affected targets. It depends on an accurate build graph: a graph that misses a dependency skips a test that should have run, and the failure surfaces later, where it costs more. That is why Encore made encore-core hermetic before trusting the graph to skip anything.

Learned test selection goes further. Meta’s predictive test selection, described by Mateusz Machalica and colleagues in 2019, trains a model on the history of changes and test results to predict which tests a change is likely to break. It runs about a third of the tests that transitively depend on the modified code, still catches more than 95% of individual test failures and more than 99.9% of faulty changes, and halves the cost of testing infrastructure. It is proven at Meta’s scale, which supplies an enormous history to learn from. An organization of Encore’s size should expect dependency-graph selection to deliver most of the benefit, and reach for a learned model only when it has the data and the graph has stopped being enough.

Encore still runs encore-core’s full suite every hour on the head of main, so that anything selection missed is caught within the hour and traced to a small range of commits.

Keeping main green

Two changes can each pass CI and still break main together. Change A renames a function; change B, written the same morning against the old main, adds a caller of the old name. Each is green against the main it was tested on, and together they do not compile. With a few committers this is rare and easily reverted; in encore-core, where a dozen teams commit, it happens weekly. The remedy is a merge queue: each change ready to merge is tested against main plus the changes ahead of it, so main only ever advances to a state that has passed. Uber described its version in “Keeping Master Green at Scale” (2019); GitHub made a merge queue generally available in July 2023, and GitLab offers merge trains. Encore turned one on for encore-core once CI fell to 9 minutes, since a queue of 31-minute runs would have been unbearable. Service repositories, each with one team committing, do not need one.

The other enemy of a green main is the flaky test, one that passes and fails on the same code. Chapter 3 explained why tests flake and how to design against it; living with the flakes that remain belongs here. Encore borrowed its mechanics from Google, whose tooling, as John Micco described in 2016, “automatically quarantines the test,” taking it off the critical path and filing a bug. Encore’s version is policy in the CI configuration the Developer Portal team maintains:

# ci/policy/flaky.yaml — applied to every repository on the golden path
detect:
  rerun_on_failure: 1             # one automatic rerun, never more
  flaky_if: fails_then_passes_on_same_commit
quarantine:
  action: remove_from_merge_gate  # still runs; result no longer blocks
  ticket_to: codeowners           # the owning team, from CODEOWNERS
  deadline_days: 14               # fixed or deleted by then
  after_deadline: fail_owner_build
report:
  scorecard: quarantined_tests    # visible count, per team

Each line answers a way quarantine goes wrong. One automatic rerun lets a flaky failure through once without teaching people to rerun until green. A quarantined test still runs, so its owners can see whether their fix worked. The ticket goes to a team, not to whoever happened to see the failure. And the deadline is enforced, because quarantined tests without one are forgotten.

The pipeline as a product

All of this needs an owner. At Encore the Developer Portal team owns CI pipelines and their templates, and Runtime owns deployment from the artifact onward. The arrangement is sensible, and for years it was understaffed. Developer Portal’s four people also owned the portal, the templates, the service registry, the scorecards and the exception ledger. Nobody decided to neglect CI; it simply never reached the top of a backlog shared with five other products, and encore-core’s build grew slower by a minute or two a year, each minute too small to justify a project. In month five of year nine, after the survey’s 54% and the Resale team’s first weeks in encore-core, Encore added two engineers to Developer Portal to work on CI, which brought the team to six.

Treating CI as a product means giving it users, an owner, a measure of whether it serves them and a way to hear from them. A hospital laboratory is a fair comparison. Clinicians do not care how its analyzers work; they care about turnaround time, from drawing a sample to having a result they can act on, and about the rate of wrong results, and a well-run laboratory publishes both. The analogy holds for the choice of measures and breaks at the source of error: a laboratory’s false results come partly from the variability of patients, while CI’s false reds come almost entirely from its own tests and machines.

Developer Portal now publishes an objective for CI in the same form as any service’s: time from push to verdict at the 50th and 90th percentiles, with the 90th at or under 10 minutes on golden-path repositories and 12 on encore-core, and a false-red rate, the share of failed runs that pass on rerun with no change, held under 1%. Server uptime is left out, because developers feel it only through those two numbers. Published objectives for CI are still rare, though Google’s build-satisfaction surveys show the measuring is done. Encore also asks about CI in its quarterly developer survey, and Chapter 6 discusses how the platform group reads the answers. My own years of platform engineering inside a bank left me convinced that the most useful thing an internal tool team can do is state in public what it promises its users.

Recap. Continuous integration is the practice of integrating daily and fixing a broken main at once; the commit stage serves it best when it answers in five to ten minutes. Dependency-graph test selection is proven wherever the build graph can be trusted, and learned selection pays at the scale where it was proven. A merge queue and an enforced quarantine keep main green as more teams commit to one repository, and someone must own the pipeline as a product, with an objective that measures what its users feel.

Exercise 4.4. Find the five slowest steps in your CI for one repository, with the median and 90th-percentile duration of each, and the name of the team that owns each one. If any step has no owner, say who should have it, and estimate its false-red rate from the last month’s reruns.

4.5 Feature Flags in Development

Trunk with flags

Trunk-based development leaves an obvious question: if everyone merges to main every day, where does work that will not be finished for weeks go? It goes to main, switched off. A feature flag, as the Resilience book’s Section 4.3 defines it, is a runtime switch that decides which behavior deployed code exhibits, and it rests on the distinction between deploying and releasing that the Architecture book’s Section 5.3 drew. Those books own flags as a tool of release, with its rollouts, kill switches and debt. This section treats flags as a tool of development: how a team keeps unfinished work on trunk without exposing it, and the code-level discipline that keeps this from becoming a mess.

Pete Hodgson’s “Feature Toggles (aka Feature Flags),” published on Fowler’s site in 2016 and revised in 2017, remains the standard account, and Google works this way at very large scale: as Potvin and Levenberg reported, its features are controlled by flags rather than branches. A flag keeps integration happening every day, so the unfinished feature is compiled, tested and merged against everyone else’s work continuously and semantic conflicts surface the day they are created. A branch defers all of that to one day at the end.

A theater building the set for its next production while the current one is still running builds it backstage, in the same building with the same crew, so that measurements, rigging points and trapdoors are checked against the real stage as the work goes on. Nobody builds it in a distant warehouse and hopes it fits on opening night. The analogy breaks in an instructive place. A set backstage takes up space but leaves the current show untouched. Flagged code sits in the same files as the running code, behind an if, and every flag adds a second path that must be kept correct and eventually removed.

Verified Resale has lived this way since its first code. From month four, each piece went to main behind a release flag owned by the team that wrote it: the listing endpoints, the resale purchase saga, the Resale module in the fan app. For the final six weeks before the month-12 launch, the complete flow ran on trunk and in production, dark, behind one release flag, resale-verified-flow, exercised end to end by Encore’s staff and by Rowmate’s certification traffic. How it was then switched on belongs to the Products book’s Chapter 10 and the Resilience book’s Section 4.3.

Kinds of flags

Hodgson distinguishes four kinds of flags, each with its own lifetime and owner. The Resilience book’s Section 4.3 gives Encore’s expiry rules; Figure 4.5 adds the owners.

Kind	What it decides	Typical life (Hodgson)	Owner at Encore	Example in Verified Resale
Release	Whether unfinished or unreleased code runs	“should generally not stick around much longer than a week or two”	The team that wrote the code	resale-verified-flow
Experiment	Which variant a user sees, for measurement	As long as the experiment needs	The team running it, via the Experimentation platform	None at launch (the Products book’s Chapter 4)
Operational	Whether an expensive or risky path runs under stress	Mostly short; kill switches for years	The team on call for the service	resale-listings-intake, which stops new listings during an incident
Permission	Which users or customers get a capability	Can last “multiple years”	Moves to entitlements	Which organizers may enable resale

Figure 4.5 — Four kinds of flags, with Hodgson’s lifetimes and Encore’s owners. The last row is a flag only until it becomes configuration.

The last row repays a closer look. A permission flag that lives for years has become a product decision about who may use what, and it belongs in the system that makes such decisions. At Encore that is the entitlement machinery Chapter 7 describes, on which organizers’ resale rules are built as per-organizer configuration, so a permission flag may exist only while a capability is being built. Experiment flags belong to the Products book’s Chapter 4.

Branch by abstraction inside a codebase

Flags hide a new feature. They do not by themselves help replace an old component that many parts of the code depend on, and that was Inventory’s problem with reissue. Chapter 1 explained it: barcode issuance derives a ticket’s identity from its order and seat, four other places in encore-core recompute it the same way, and reissue needs a ticket to receive a new identity, which the derived scheme cannot express. The technique for changing such a thing on trunk is branch by abstraction, which Paul Hammant described in 2007, crediting Stacy Curl with it. The Architecture book’s Section 3.4 introduced it at the scale of a system; here are its mechanics inside one codebase.

Introduce an abstraction in front of the old implementation. Move every caller to it, one small change at a time. Build the new implementation behind the same abstraction, switch to it with a flag, and finally remove the old implementation. At every step trunk builds and the system works. The best-known public case is Thoughtworks’ Go continuous-delivery server, whose team, as Jez Humble told it, replaced its object-relational mapping layer, iBatis, with Hibernate this way, all on trunk.

Inventory’s abstraction took this shape.

// encore-core: services/seat-inventory/issuance/TicketIdentities.kt
interface TicketIdentities {
    fun issue(order: OrderId, seat: SeatId): TicketId
    fun current(order: OrderId, seat: SeatId): TicketId   // callers stop recomputing
    fun reissue(request: ReissueRequest): ReissueOutcome   // Chapter 2's contract
}

class DerivedIdentities : TicketIdentities {               // year-one behavior
    override fun issue(order: OrderId, seat: SeatId) = TicketId.derive(order, seat)
    override fun current(order: OrderId, seat: SeatId) = TicketId.derive(order, seat)
    override fun reissue(request: ReissueRequest): ReissueOutcome =
        error("derived identities cannot change")
}
// OpaqueIdentities stores identities: existing tickets keep their derived value,
// reissued ones get a random opaque id. Chosen by flag `ticket-identity-opaque`.

The first changes did nothing visible. They introduced the interface and moved the four places that recomputed identities to call current, one small change each. Only then did Inventory build OpaqueIdentities, whose essential property is that for every existing ticket it returns exactly what the derived scheme returns, so switching the flag changes nothing for any ticket until it is reissued. For three weeks in month eight both implementations ran in production, the new one answering in shadow with every disagreement logged, the parallel run of the Architecture book’s Section 3.4; Chapter 11 reports what it found. Once the flag was on everywhere, reissue could do its work through the interface Chapter 2 designed: a new opaque identity for the buyer, the old one onto the revocation feed, and a retry with the same idempotency key treated as the same reissue, since a ticket may be resold, and reissued, more than once. Partners receive a ticket_id through the API and never parse a barcode, which rotates in any case; the one partner whose old code had parsed the identity’s prefix, Rowmate, rewrote it during sandbox certification, as Section 2.5 tells.

Testing flag combinations

A system with n flags has 2ⁿ possible configurations, and at 2,300 flags no team will test them all, nor should it try. Hodgson’s advice is the practical one: “It’s most important to test the toggle configuration which you expect to become live in production.” He adds that “Many teams also perform some tests with all toggles flipped On,” and he recommends testing the fallback, the configuration with the new toggles off, so that turning a flag off in an emergency is known to work.

Encore tests three configurations per service: production-next, the flag values the service will have after the change ships, taken from a snapshot of the flag service; all-new-on, which catches a new path that works only while another new path is off; and fallback, with the change’s own flags off. Test fixtures default to the production snapshot, because a suite that hard-codes every flag to off is quietly testing a system that no longer exists. During its six dark weeks, resale-verified-flow also had a small suite of journey tests that ran with the flag on in every build, so the hidden product was tested as continuously as the visible one.

Cleanup as part of done

The Resilience book’s Section 4.3 tells how Encore found a third of its 2,300 flags unchanged for a year, and sets out the rules that followed, along with Uber’s Piranha, which writes cleanup changes automatically. What remains here is the code side.

The change that removes a release flag should delete the flag check, the branch no longer taken and the tests that exercised it, together. Hodgson describes teams that create “a toggle removal task” with the toggle, and others that build “time bombs,” tests that fail once a flag expires. Encore uses the first and a variant of the second: a lint step in the commit stage fails on any reference to a flag the flag service does not know and warns on any reference to an expired one. A month after full release, the Resale team’s removal of resale-verified-flow deleted about 300 lines across four repositories. Inventory’s removal of ticket-identity-opaque took DerivedIdentities with it, and with that went the last place in Encore’s code that assumed a ticket’s identity could never change.

Recap. Flags let unfinished work live on trunk and integrate every day, which is why they beat long-lived branches, and each kind of flag has its own lifetime and owner, with long-lived permission flags belonging in entitlements. Branch by abstraction lets a team replace a component that many callers depend on, one reviewable change at a time, and with a parallel run it let Inventory change the year-one assumption about ticket identity without stopping the system. Test the configurations production will have, default tests to production’s values, and remove each flag with its dead branch and tests in one change.

Exercise 4.5. List the flags in one service you own, with each one’s kind, owner and removal date. Find one whose purpose is finished, and write the change that deletes it, including the dead branch and its tests. Note what you found that nobody had remembered was there.

Kata: The Two-Quarter Lead Time

Your brief: “Inkstone.” Inkstone makes a collaborative design tool used in browsers and a desktop app, sold by subscription to studios and in-house design teams. It has 250 engineers in 26 stream-aligned teams and a platform team of eight, and its code lives in 30 repositories: a large inkstone-app repository that holds the editor, the rendering engine and the collaboration server, which together account for about two-thirds of all changes, and 29 smaller service repositories. CI on inkstone-app takes 45 minutes at the median and over an hour at the 90th percentile; its tests reach the network, and about one failed run in eight passes on rerun. The median change is about 450 lines, and changes that touch the rendering engine wait a median of two working days for review, because three engineers approve almost all of them. Teams use long-lived feature branches merged at the end of each two-week sprint, and the merges regularly break main for half a day. Nobody owns CI; the platform team runs the build machines and treats everything else as the product teams’ problem. The CTO has asked for the lead time from first commit to merge on main to be halved within two quarters, and wants to know, before committing, how the company will tell whether it worked.

Deliverables:

1. Baseline — Measure lead time from first commit to merge on main for the last quarter, at the median and 75th percentile, split into its parts: time on a branch before review is requested, time in review, time in CI and time waiting to merge. Say which part is largest, and how you obtained each number.
2. Branching and flags — Propose a move from sprint-long branches to trunk-based development, with the flag practice that makes it possible. Name the kinds of flags you will allow, their owners and how each is removed, and give one branch-by-abstraction plan for a component of the rendering engine.
3. Review plan — Set a target change size, a latency target for first response and for P75 time in review, and a guardrail against rubber-stamping. Say how you will widen the circle of rendering-engine reviewers, and which changes may be shipped or shown without waiting for review.
4. CI plan — Lay out how inkstone-app’s CI gets from 45 minutes to under 10: what you would make hermetic first, how you would select tests, whether you need a remote cache or a merge queue, and how flaky tests are quarantined. Say who owns CI afterwards, how many people it needs and what objective they publish.
5. Measurement and timeline — Lay out the two quarters in order, explaining why that order, and name the measures, with guardrails, that will show the CTO whether the lead time halved without stability or review quality getting worse.

Rubric:

Where you now stand

You can now look at the development loop as a system of its own and find where a change waits on its way from a laptop to main. You can give a developer’s code the neighbours it needs without running the fleet, choose a repository shape knowing what each costs, make a build trustworthy enough to cache and to select tests from, keep review fast without making it careless, run CI as a product with an owner and an objective, and carry unfinished work on trunk behind flags that are removed when their work is done. You have also seen why none of this followed automatically from Git: cheap branches made integration easier to postpone, and the discipline of integrating daily had to be learned separately. One force now presses on every part of this loop at once. Assistants that write code can produce changes faster than any loop designed for human authors was built to absorb, and the question of what that does to review, testing and ownership is the subject of Chapter 5.

References

• Jez Humble, David Farley — Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley, 2010.
• Titus Winters, Tom Manshreck, Hyrum Wright (eds.) — Software Engineering at Google: Lessons Learned from Programming Over Time. O’Reilly, 2020 — free online.
• Caitlin Sadowski, Emma Söderberg, Luke Church, Michal Sipko, Alberto Bacchelli — Modern Code Review: A Case Study at Google. ICSE-SEIP, 2018 — free online.
• Martin Fowler — Patterns for Managing Source Code Branches. martinfowler.com, 2020 — free online.
• Alberto Bacchelli, Christian Bird — Expectations, Outcomes, and Challenges of Modern Code Review. ICSE, 2013 — free online.
• Nicole Forsgren, Jez Humble, Gene Kim — Accelerate: The Science of Lean Software and DevOps. IT Revolution, 2018; with DORA — Capabilities: Trunk-based development, dora.dev — free online.
• Pete Hodgson — Feature Toggles (aka Feature Flags). martinfowler.com, 2016, revised 2017 — free online.
• Michael E. Fagan — Design and Code Inspections to Reduce Errors in Program Development. IBM Systems Journal 15(3), pp. 182–211, 1976.
• Patrick Riggs — Move faster, wait less: Improving code review time at Meta. Engineering at Meta, 2022 — free online.
• Tyler French, Mindaugas Rukas, Xiaoyang Tan — How We Halved Go Monorepo CI Build Time. Uber Engineering, 2022 — free online.
• Rachel Potvin, Josh Levenberg — Why Google Stores Billions of Lines of Code in a Single Repository. Communications of the ACM 59(7), 2016.
• Mateusz Machalica, Alex Samylkin, Meredith Porth, Satish Chandra — Predictive Test Selection. ICSE-SEIP, 2019 — free online.
• Garrett Heel — Scaling productivity on microservices at Lyft (Part 1). Lyft Engineering, 2021 — free online.
• Google Cloud and DORA — Accelerate State of DevOps Report 2023. Google Cloud, 2023 — free online.
• Linus Torvalds — Kernel SCM saga... linux-kernel mailing list, 6 April 2005 — free online.
• Scott Chacon, Ben Straub — Pro Git, 2nd ed.. Apress, 2014 — free online.
• Vincent Driessen — A successful Git branching model. nvie.com, 2010, with a note of reflection added in 2020 — free online.