Building Software at Scale in the AI & Cloud Era

← the whole book on one page

Chapter 1: Building Software at 400 Engineers — The System That Builds the System

“Software engineering is programming integrated over time.” — Titus Winters, Tom Manshreck and Hyrum Wright

Nine engineers can build a remarkable amount of software. They sit within earshot of one another, each of them has read most of the code, and a question about why something works the way it does can be answered by turning a chair around. Four hundred engineers can build far more, and yet a curious thing happens as a company grows from the first number to the second: each change becomes harder to make. A modification that once took an afternoon now takes a fortnight, crosses three teams, waits in two queues and needs a meeting to find out who owns the code it must touch. Nobody decided that this should happen. It is the cumulative result of thousands of reasonable decisions, each made by default, about how code is organized, how changes reach production and how people are arranged around both. The question of this chapter, and of this book, is what it takes for an organization of hundreds to keep changing a large, old and valuable piece of software quickly and safely, and why that is so much harder than it was for the few who wrote its first version.

The question is as old as the name of our discipline. In October 1968, at Garmisch in the Bavarian Alps, the North Atlantic Treaty Organization (NATO) brought together more than 50 people to talk about the trouble they were having with large software systems, and its organizers gave the meeting a deliberately provocative title: software engineering. The participants were candid about how far their practice fell short of that name, and they saw, more clearly than they are usually given credit for, that the trouble grew with the number of people as much as with the number of lines. What they lacked was a method. Section 1.1 tells their story, because the next half-century of our field can be read as a long search for the practices they knew they needed, and this book is about the ones that were found and proven.

Encore is where we will watch those practices at work. It sells concert tickets across Europe, it reached 400 engineers in its seventh year, and it has stayed at about that size since. Now, in the late spring of its ninth year, it has decided to build Verified Resale: a way for fans to resell tickets they bought through Encore, at no more than face value, through authorized partners, with every resold ticket reissued under a new barcode. The Products book follows that bet as a product, from its memo to its launch and review. This book follows the building. The bet’s engineering needs are plain enough to state and hard to meet: it must change the oldest code Encore owns, a barcode issuance module written when the company had nine engineers and understood today by exactly one of them; it must coordinate 12 teams across 23 dependencies; and it must do both without slowing the rest of Encore’s teams, most of which have nothing to do with resale. Each chapter that follows takes one part of that work and teaches the practice that makes it possible.

This chapter sets the stage: the two systems every software company runs, Encore at its present size, Verified Resale as engineering first sees it, the four levers the rest of the book pulls, and what assistance from artificial intelligence (AI) changes about all of this.

1.1 Two Systems

The product and the system that builds it

Every software company runs two systems. The first is the one its customers use: at Encore, the waiting room, the seat map, the checkout and the ticket on the fan’s phone. The second is the one that produces the first. It consists of the code and its tests, the build tools and pipelines, the platform that services run on, the teams that own each piece, the review habits, the plans and the budgets, and the unwritten rules about who may change what. I will call this second system the engineering system: the code, tools, platforms, teams and practices through which an organization turns an intention into running software, and goes on changing that software for as long as it lives. This book is about the engineering system. The Architecture book was about the shape of the first system, and the Products book about deciding what it should do next; here we study the machinery that makes and remakes it.

The distinction is easy to state and easy to forget, because the second system is mostly invisible to anyone outside it. A touring concert production makes the point better than any diagram. The audience sees a show: two hours of music, light and sound in a particular hall on a particular night. Behind it travels another organization entirely, with riggers, trucks, a production manager, a stage plot, a load-in schedule and a crew that has rehearsed the order in which the stage goes up. That second organization decides how many cities the tour can play, how quickly it can move between them, and what happens when a truck breaks down in the Alps. No one buys a ticket to see it, yet it sets the limits of every show. The analogy breaks in a way that teaches. A touring crew builds the same stage every night, and its virtue is repetition. An engineering system builds a different product every day, because each change alters the thing being built, and it must also change itself as it goes, since the tools and habits that suited 70 engineers will not suit 400.

The authors of Software Engineering at Google put the same idea in a sentence that serves as this chapter’s epigraph: software engineering is “programming integrated over time.” They go on to name three differences between programming and software engineering, which are time, scale and the trade-offs at play. A program written once and thrown away needs only a programmer. A program that must be changed for years, by people who did not write it, while it carries money and reputations, needs an engineering system, and the quality of that system decides how fast and how safely the program can change.

The engineering system is sociotechnical, which is to say it is made of people and machinery that shape each other. The first sign of this is Conway’s law, which the Architecture book’s Section 1.3 introduced and its Section 5.5 turned into a design tool: the structure of a system tends to mirror the structure of the organization that builds it. The law runs in both directions through the engineering system. The way teams are drawn shapes the code, and the way the code is drawn shapes which teams must talk to one another before anything can change. Neither half can be improved for long without the other, and much of this book is about pulling on both at once.

Garmisch, 1968

In late 1967 the NATO Science Committee’s Study Group on Computer Science recommended a conference on the problems of building software, and it met at Garmisch, in Bavaria, from 7 to 11 October 1968. Its chairman was Friedrich L. Bauer of Munich. The report records that it “was attended by more than fifty people, from eleven different countries,” drawn from the users and manufacturers of computers and from the universities. It was a working conference, divided into three groups that studied the design, the production and the service of software, and among those present were Edsger Dijkstra, Alan Perlis, Peter Naur, Doug McIlroy and Brian Randell. (The Architecture book’s Chapter 13 recalls McIlroy’s argument there for mass-produced software components; we need not retell it.)

The name was chosen with intent. The report’s preface explains that “the phrase ‘software engineering’ was deliberately chosen as being provocative, in implying the need for software manufacture to be based on the types of theoretical foundations and practical disciplines, that are traditional in the established branches of engineering.” Randell later wrote that the idea of the conference, and of its title, came, he believed, from Bauer. The provocation lay in the gap between the word and the practice. Civil engineers had codes, calculations and a professional memory of bridges that fell down; software in 1968 had, by the participants’ own account, a habit of arriving late, costing far more than planned and failing in use. The figures discussed were large even by later standards. Ed David relayed that, according to T. J. Watson, IBM’s OS/360 operating system had cost over $50 million a year during its preparation, with at least 5,000 man-years invested. Alexander d’Agapeyeff observed that a European manufacturer that had employed fewer than 50 programmers in 1958 probably employed between 1,000 and 2,000 a decade later.

The participants did not agree on what to call their predicament. The report speaks of a debate over “what some members chose to call the ‘software crisis’ or the ‘software gap’.” Kolence objected that “crisis” was too emotional a word. Douglas Ross took the other side, observing that a heart attack is a crisis even when every other organ is in fine working order. Others pointed out that most computers worked tolerably well most of the time. Dijkstra welcomed the open admission of failure as the precondition for improving on it. The phrase “software crisis” became canonical only four years later, when Dijkstra, in his 1972 Turing Award lecture “The Humble Programmer,” called Garmisch “the turning point,” the conference at which “there occurred the first open admission of the software crisis.” It is worth keeping that history straight, because the word has since been repeated so often that it sounds like a finding. At Garmisch it was a contested description, offered by people who disagreed about how bad things were and agreed only that they were bad enough to need a discipline.

A common reading of Garmisch holds that its participants thought of scale as a matter of code and underrated the problem of people. The report shows the opposite. Alan Perlis told the meeting that “we must learn how to build software systems with hundreds, possibly thousands of people.” John Buxton warned that more than 20 programmers on one project was usually a disaster, and Opler said he knew how to organize communication for projects of 10 to 50 people and was sure he did not know how to do it for much larger ones. Ed David’s position paper cautioned that scale did not refer only to the size of the code. Randell cited Melvin Conway’s paper, published in Datamation that April, and proposed out loud what we now call the inverse Conway manoeuvre: since a system will tend to mirror the structure of the group that builds it, the group’s structure should be designed deliberately to produce the system we want. Garmisch, in other words, already understood that the hard part of large software lies in the many people who must change it together.

What it did not have was a method. Peter Naur and Brian Randell wrote the report in Munich in the week after the meeting, from rapporteurs’ notes and tape recordings, and published it in January 1969. A second conference followed, near Rome, from 27 to 31 October 1969, and its report, edited by Buxton and Randell, is unusually frank. The meeting “bore little resemblance to its predecessor,” the editors wrote; the sense of shared urgency had faded, and “a lack of communication between different sections of the participants became, in the editors’ opinions at least, a dominant feature.” No attempt was made to continue the series. Randell later recalled that at Garmisch it had been “fully accepted that the term software engineering expressed a need rather than a reality,” and Rome had shown how far the field still was from agreeing on how to meet that need. Figure 1.1 draws the meeting in outline and places it among the years around it.

Figure 1.1 — Garmisch in outline: three constituencies, three working groups and a report, and on the timeline below, the short distance from Conway’s paper to the meeting and the long one to the phrase “software crisis.”

I find this the most useful way to read the conference. Garmisch named the problem this book addresses: a gap between what organizations needed to build and what their methods could build, a gap that opened wherever systems and the groups building them became large. Its participants even distrusted easy answers; Ross warned that there was “no worse word than ‘breakthrough’” in discussing solutions, 18 years before Fred Brooks’s “No Silver Bullet.” The decades since have been spent finding practices that close the gap a little, testing them and keeping the ones that work. Some were found quickly and some took 40 years. Some, like version control and automated testing, became so ordinary that we forget they once had to be invented. The engineering system of a company like Encore is the accumulated answer to the question Garmisch asked, and the last chapter of this book will return to that Alpine meeting to ask how much of the gap has closed.

What scale changes

Growth from nine engineers to 400 does three things to the cost of building software, and it is worth separating them because they call for different remedies. The first is coordination. Brooks showed, in The Mythical Man-Month, that the possible paths of communication in a group grow with roughly the square of its size, and the Architecture book’s Chapter 5 works through that arithmetic for Encore’s growth to 70 engineers. At 400 people the paths among individuals are far too many to use, so organizations group people into teams and route communication through team boundaries, which turns a problem of people into a problem of structure. Coordination is the one cost we can say with confidence grows faster than headcount, because the arithmetic guarantees it.

The second cost is comprehension. When Encore had nine engineers, most of them had read most of the code. At 400 nobody has read most of the code, and nobody could; the codebase has grown past what one mind can hold, and each team understands its own part and a thin margin around it. There is no single canonical result that measures how comprehension decays with size, so I offer this as an argument from observation. It has, however, a well-studied cousin. Matthew Skelton and Manuel Pais’s Team Topologies treats a team’s cognitive load, the amount a team must hold in mind to change its software safely, as the limit on how much a team can own, and developer-experience research has made it one of the three dimensions it measures. Comprehension is the reason a change at 400 so often begins with archaeology: before anyone can modify the code, someone must find out what it does and who still knows.

The third cost is the cost of change itself. Meir Lehman’s laws of software evolution, whose first law the Architecture book’s Section 13.3 takes up, have a second law that matters more here. In Lehman’s 1980 wording, “as an evolving program is continually changed, its complexity, reflecting deteriorating structure, increases unless work is done to maintain or reduce it.” The final clause carries the weight. Complexity is the default outcome of change, and only deliberate work holds it back. Brooks’s distinction between essential and accidental complexity, which the Architecture book makes its starting point, sharpens the claim. The essential complexity of selling each seat exactly once to a fan who is not a bot does not change whether Encore has nine engineers or 400. The accidental complexity does: the extra layers, the duplicated libraries, the three ways of doing one thing, the code whose purpose nobody remembers. Accidental complexity accumulates with every change unless the engineering system removes it, and an organization of 400 makes a great many changes. That work of removal is much of what this book describes.

Comprehension and the cost of change tend to grow faster than headcount, though I cannot give a rate for either. Coordination provably does. Together they explain the curious fact this chapter began with, that adding people to a growing company makes each of its changes harder even while it makes the company able to attempt more of them.

What a good engineering system produces

If the engineering system is a product in its own right, we should be able to say what it is for, and to judge it by results instead of by activity. Three results matter most.

The first is delivery: how quickly and how safely changes reach users. The research program that began as the State of DevOps surveys, and continues as DORA (DevOps Research and Assessment), has studied this more carefully than anyone. Nicole Forsgren, Jez Humble and Gene Kim’s Accelerate (2018), drawing on survey data from 2014 to 2017 and more than 23,000 responses, reported the finding that has shaped the field since: speed and stability go together, so the organizations that deliver most often also fail least and recover fastest. DORA’s delivery metrics, long known as the “four keys,” became five in 2024. They are change lead time, deployment frequency, change failure rate, failed deployment recovery time and, newly, rework rate, which DORA groups into throughput and stability. The Architecture book’s Section 11.2 defines them and the Resilience book’s Section 4.2 shows why throughput and stability do not trade off; this book uses them in Chapter 10 to measure delivery across Encore’s teams.

The second result is the ability to change old code safely. No standard metric captures it, and it is easy to neglect because new code is where the excitement is. Yet a company’s most valuable code is usually its oldest, since it encodes years of decisions about how the business really works, and an engineering system that can only change new code has a short future. Chapters 2 and 3 are about this capacity, and Verified Resale will test it severely.

The third result is people who can do their best work. Abi Noda, Margaret-Anne Storey, Nicole Forsgren and Michaela Greiler, in “DevEx: What Actually Drives Productivity” (2023), proposed three dimensions of developer experience: feedback loops, how quickly a developer learns whether a change works; cognitive load, as above; and flow state, the ability to work without interruption on something absorbing. It is a framework for measuring, derived from earlier research, and Chapter 6 uses it as one lens on whether Encore’s platform is helping.

These measures have limits, and it is honest to state them at the start. None of them says whether the right thing was built; a team can deliver quickly and stably a feature nobody wanted, which is the Products book’s subject. All of them can be gamed once they become targets, which is Chapter 10’s warning. And none can be improved by someone who has not first looked at how work actually flows. The proven way to look is value-stream mapping: drawing the path a change takes from idea to production, with every step, handoff and wait marked and timed. It comes from lean manufacturing, where Mike Rother and John Shook’s Learning to See (1998) set out the method, and DORA lists value-stream management among the capabilities that predict delivery performance. The exercise at the end of this section asks for one.

Where this book sits in the series

This is the third of five books that follow Encore. The Architecture book shaped the system: its styles, boundaries, data, interfaces and platform. The Products book decided what to build and why, and followed Verified Resale from a memo to a launch. This book builds it, and in doing so studies the engineering system that does the building. The Resilience book hardens the system against load and failure, and the Operations book runs it on the day the load arrives. Each book leaves the others’ subjects alone. Here that means we take the architecture of Encore’s services as given, take the product decisions about Verified Resale as settled, and leave progressive delivery, service level objectives and incident response to the books that own them, pointing to them where a reader needs the thread.

Recap. Every software company runs a product and an engineering system that builds it, and the second sets the limits of the first. Garmisch named the problem of building software with many people in 1968 and lacked the methods to solve it; the field has spent the decades since finding them. Scale raises the costs of coordination, comprehension and change, and complexity rises unless deliberate work holds it back, so a good engineering system is judged by delivery, by the safety of changing old code and by the experience of the people who use it.

Exercise 1.1. Draw the value stream for one recent change in your organization, from the moment someone decided it should happen to the moment a user could see it. Mark every team it passed through, every tool, and every wait, with the time spent waiting. Then add up the hours of work and the hours of waiting separately, and compare them.

1.2 Encore at 400

Readers who know Encore from the earlier books may skim this introduction. Encore sells concert tickets across the European Union (EU), priced in euros, and also sells its platform to venues as software as a service (SaaS). Three facts define its engineering. When a famous artist’s tour goes on sale, nearly always at 10:00 on a Friday, traffic rises 500× within minutes. Every seat must be sold exactly once, never lost and never sold twice. And the bots arrive before the fans. It has four kinds of customer: fans; event organizers; partners such as venues, resale partners and a festival app, which call its public application programming interface (API) at api.encore.com; and venues that run Encore as SaaS. Each of the services named below, from the Sale Gate to Seat Inventory, is described where it first matters in the Architecture book, and the appendices of the Resilience and Operations books keep the whole company on one page. What follows is Encore seen from inside its engineering system, which none of those books looked at directly.

How Encore got here

Encore’s engineering system was never designed as a whole. It grew in three stages, and at each stage the people building it made choices by default that were reasonable for their size and became constraints at the next. Figure 1.2 sets them side by side.

Figure 1.2 — Encore’s engineering system at three sizes. The outlined row, the decision each size made by default, is the one nobody wrote down at the time.

The bottom row deserves a slower reading than the others, because it holds the engineering system’s history in compressed form. At nine people, ownership is unnecessary; everyone knows who wrote what, and anyone can change anything. At 70, the split into services (the Architecture book’s Chapter 5) gave each service a team, but some code belonged to no service in particular, and it stayed in the old repository with no clear owner. At 400, the golden path made new services easy to build well, and the old code that still worked was, sensibly, left alone. Each default was a good decision on its own day. Taken together they produced a company with excellent paths for new code and a quiet reluctance to touch old code, and Verified Resale needs to change the oldest code of all.

The fuller story of how Encore’s organization grew, including what broke at each size, belongs to Chapter 9. Here we need only the engineering system as it stands now.

The codebase

Encore’s backend code lives in two kinds of repository. The shared repository, encore-core, grew from the year-one monolith and still holds several of the services that were split from it, Seat Inventory & Ticketing and the Orders & Payments saga among them, along with the shared libraries that several services use. Newer services, created from the platform’s template since the golden path arrived, each have a repository of their own. Chapter 4 examines this arrangement and why Encore keeps it; for now it is enough to know that the oldest code and the slowest builds live in the same place.

On the frontend, the fan storefront on the web is a modular monolith built on one design system, and the organizer console is a separate application on the same design system, as the Architecture book’s Chapter 8 explained. The fan app is native, on iOS and Android, and it consists of a shell and 14 feature modules, each owned by a team: Discover, Search, Event, Waiting Room, Seat Map, Checkout, Payment Completion, My Tickets, Account, Inbox & Preferences, Help & Refunds, Presales, Festivals and Following. Verified Resale will add a fifteenth.

Inside the shared repository, inside the Seat Inventory & Ticketing service, sits the module this book will keep returning to. Barcode issuance turns a sold seat into a ticket identity and the entry credential that a venue’s scanners accept. It was written in Encore’s first year, in Kotlin like the rest of the original monolith, and it runs to about 11,000 lines. Every ticket Encore has ever sold has passed through it. Section 1.3 explains why Verified Resale cannot avoid it.

The path to production

A change at Encore travels a route that most of its engineers no longer notice. The engineer works on a short-lived branch and integrates into the main line at least daily, which is trunk-based development, one of the capabilities DORA’s research associates with high delivery performance. The change is reviewed by an engineer from the owning team, and by the owners of any other code it touches. Continuous integration (CI) builds and tests it; on the shared repository that takes 31 minutes, on most newer repositories under 10. It then follows the golden path, the supported route from a service template to production that the Architecture book’s Section 15.2 described: the template, the provisioning workflow, the registry entry and the scorecard. Deployment is progressive, through canaries at 1, 5, 25 and 100% of traffic, never during an on-sale, and unfinished features hide behind some 2,300 feature flags. The Resilience book’s Sections 4.2 and 4.3 own the mechanics of that last stretch.

The numbers are good by the standards of the industry. Encore makes about 180 production deploys on an ordinary working day. Its median lead time for changes, from commit to production, is about 26 hours. The median hides the shape of the distribution, though. Most of those hours are spent waiting for a reviewer, and for changes that cross team boundaries the wait stretches to days. A change confined to one team’s new service can reach production before lunch. A change that touches the shared repository and two teams’ code routinely takes most of a week. Verified Resale consists almost entirely of the second kind.

The teams

Encore has 40 stream-aligned teams, about 360 engineers averaging nine a team, and each owns a flow of business change: a part of the fan app, a piece of the organizer console, a partner API, a service in the purchase path. Beneath them sits a platform group of nine teams and 40 people, about 10% of engineering and at the top of the 5–10% band that the Architecture book’s Section 15.2 called a common rule of thumb. The platform teams own the runtime, the developer portal and its templates, the paths for APIs and web frontends, the event backbone, the data platform, the AI gateway, security engineering and reliability. The stream-aligned teams fall into five groups: nine for the fan storefront and app, seven for the organizer console, three for partner and public APIs, seven for the SaaS that venues buy, and 14 for the shared domains that everything depends on, from the Sale Gate and Bot Screening to Seat Inventory, Orders & Payments and Notifications. Chapter 9 explains how the map came to look like this and how Encore redraws it.

A map of teams says who is responsible. It does not say who owns which lines of code, and at 400 engineers that question cannot be answered from memory. Encore answers it the way most large engineering organizations now do, with ownership metadata: a record, kept in the repository and the service registry, that names the team responsible for each directory and service. Google’s OWNERS files, described in Software Engineering at Google, are the best-known version, and the CODEOWNERS file that most code hosts support is the commonest. An excerpt from the shared repository’s file shows both what such a record does and what it cannot do.

# CODEOWNERS — shared repository (excerpt)
/services/seat-inventory/                   @encore/inventory
/services/seat-inventory/issuance/barcode/  @encore/inventory
/services/orders/                           @encore/orders-payments
/services/orders/saga/                      @encore/orders-payments
/libs/ticket-id/                            @encore/inventory
/libs/money/                                @encore/finance-platform
/libs/http-client/                          # no owner since year 5

The file says that the Inventory team owns barcode issuance, and the review history says that one person on that team has approved every change to it for three years. The file records accountability; it cannot record understanding. The last line is a different kind of gap. A shared library that every service in the repository uses has had no owner since its author moved teams, and changes to it are reviewed by whoever happens to be changing it. Chapter 4 makes the repository’s ownership match the team map, and Chapter 8 asks how shared code like this should be run.

What engineers say hurts

Encore asks its engineers every quarter how their work is going. The survey has 10 core items that never change, so that answers can be compared across years, and a free-text question about the single thing that most slows them down. About 68% of engineers answered the most recent one. The practice of a regular, stable survey is well established. Google’s engineering productivity researchers have described a quarterly Engineering Satisfaction survey that samples about a third of the company’s engineers each quarter, and Chapter 6 discusses how to design one properly. What matters here is what Encore’s engineers said, and Figure 1.3 shows it.

Figure 1.3 — The six most common answers in Encore’s latest developer survey, with the chapter that addresses each. Fear of old code is not the commonest complaint, but it is the one Verified Resale meets first.

A physician learns early that a patient’s complaint is precious evidence and a poor diagnosis. “My chest hurts” is true and important, and it might mean a heart, a lung, a rib or anxiety; the doctor takes the symptom seriously by refusing to treat it at face value. Developer complaints work the same way. “CI is slow” is true, and the cause might be the size of the repository, the absence of caching, a test suite that runs everything for every change, or a build system that cannot tell what a change affects. “Review takes forever” might mean too few reviewers, changes that are too large, or ownership drawn so that every change crosses a team boundary. The analogy fails at one point, and the failure is useful. A patient has one body, while an engineering system’s symptoms often share a cause across several complaints. Encore’s slow review and its fear of old code turn out, in Chapter 2, to be partly the same problem, because changes to code nobody understands are hard to review as well as hard to write.

One answer in the table points to a fact we will meet again. Three teams, in the course of integrating with partners, each built their own sender for webhooks, the messages Encore pushes to a partner’s system when something happens, with different retry rules and different signing. Engineers named the waste themselves, and Chapter 6 shows the platform group finding and replacing all three.

Recap. Encore’s engineering system grew in three stages, each leaving behind a default that was sensible at the time: no owners at nine, orphaned code at 70, old code left alone at 400. It now delivers well on new code and hesitates on old code, and its own engineers name the symptoms in its survey. Ownership metadata, a stable survey and a value-stream map are the first instruments for seeing such a system as it is.

Exercise 1.2. List the five complaints you hear most often from engineers in your organization. For each, write down at least two different causes that could produce it, and name the part of the engineering system (code, tests, tools, platform, teams or plans) where each cause would live. Then check one of them.

1.3 Verified Resale Arrives in Engineering

The bet, from the engineering side

Encore already lets partners advertise resale. Since its sixth year, under an arrangement called Partner Listings, authorized partners have been able to read Encore’s event listings and post resale offers that appear as links on sold-out event pages. The ticket itself changes hands outside Encore, as the original pass is forwarded from seller to buyer, and Encore checks nothing beyond the partner’s word. That arrangement allows the same barcode to be sold to several buyers, gives organizers neither control nor a share of the fee, and lets buyers through who were never screened. The Products book tells how Encore diagnosed those weaknesses and decided, in the late spring of year nine, to replace Partner Listings with something better. In brief, the product it decided on works as follows. A fan lists a ticket they bought through Encore’s Sale Gate. The listing appears in Encore’s own app and on authorized partners, Rowmate at launch and Tessera second. Another fan, screened like any primary buyer, buys it at no more than face value plus a 10% service fee, split four points to the partner through which the buyer purchased (Encore keeps those four when the buyer uses Encore’s own app), three to the organizer and three to Encore; on a €60 ticket the buyer pays €66, and the seller receives the full €60 five days after the event. Encore then reissues the ticket: the seller’s ticket identity is revoked and a new one is issued to the buyer. Mobile tickets at Encore have used rotating barcodes since the eighth year, generated on the phone from a per-ticket secret so that a screenshot is worthless. Reissue gives the buyer a new secret, and the old ticket identity joins the revocation lists that every venue gate holds for offline validation. Organizers may tighten the resale rules for their events but never loosen them, and they can write those rules in plain language through an assistant that drafts the typed rule set for them to confirm.

To engineering, that paragraph is a list of changes to code that already exists, plus some code that does not. A new Resale service must hold listings, rules and the resale purchase flow; a new Resale team, formed in month four from people in Orders & Payments and Partner APIs, with a product trio and a staff engineer, will own it. The partner API needs new endpoints, webhooks, a sandbox and a certification process. Seat Inventory must be able to reissue a ticket. Orders & Payments needs a resale purchase saga. Billing & Payouts must meter the fee split and pay sellers. Notifications must replace passes and tell sellers their ticket has sold. The fan app needs its fifteenth module, and the gate scanners must learn to refuse the revoked identity. The plan aims for a working path end to end by month nine, before the autumn tour season, and a staged launch in France, with three organizers and Rowmate, at month 12.

The 12 teams and the 23 dependencies

When the first quarterly plan for Verified Resale was drawn in month four, it touched 12 teams. They are the new Resale team; Inventory, which owns barcode issuance; Venue Entry, which owns the gate scanners and their revocation lists; Orders & Payments; Billing & Payouts; Partner APIs; Bot Screening; Organizer Console, for the rules console; Mobile App, for the release train and the wallet ticket view; Fan Messaging, which owns Notifications; Design System; and Security Engineering, for partner credentials. Eight more teams, among them Trust & Safety, Catalog and the Finance platform team, were consulted but given no planned work. The plan listed every place where one team needed something from another before its own work could finish, and there were 23 of them. Figure 1.4 groups them by the team that must deliver.

Figure 1.4 — The 23 dependencies of Verified Resale’s first plan, grouped by the team that must deliver. Of these, 10 land in code written in Encore’s first two years.

The last column was not in the original plan. An engineer on the new Resale team added it during the first planning session, and it changed the conversation. Dependencies on recent code, such as Partner APIs or Mobile App, were the ordinary business of planning: the delivering team knew its code, could estimate the work and could say when. The 10 dependencies on code from the first two years were different. Nobody could estimate them with confidence, because nobody fully understood the code they touched, and every one of them sat on the path to the month-nine milestone. That is Encore’s finding, and I do not offer it as a general law. It is common enough, though, that a planner should expect it: the dependencies most likely to hold up a large change are those on code that the organization has, by its long-standing defaults, learned to leave alone.

Encore records each dependency as a small entry in a file kept beside the plan, so that it can be reviewed, assigned and closed like any other piece of work. The entry for the most worrying of the 23 reads like this.

# plans/verified-resale/dependencies.yaml (excerpt)
- id: VR-01
  needed_by: resale
  provided_by: inventory
  what: void a ticket identity and issue a new one in a single transaction
  code: services/seat-inventory/issuance/barcode (written year 1)
  needed_for: month-9 walking skeleton
  risk: one engineer understands this code; no meaningful tests
  status: open

A record like this is ordinary project hygiene, and its value lies in the field most plans leave out. The risk line ties a scheduling dependency to the condition of the code beneath it, which is where the schedule will actually be decided. Chapter 10 shows how Encore plans across the 12 teams, and what it does about dependencies it would rather not have.

The legacy core

Barcode issuance is about eight years old. It was written in the months when Encore had nine engineers and one monolith, by people who were also building checkout, the seat map and the first pipeline, and it shows the marks of that time: long functions, a handful of tests that check little beyond whether the code compiles, and assumptions that were true when they were made and are now held up by nothing but habit. Of its original authors, one remains at Encore. She is Lena, one of the first nine engineers, now a senior engineer on the Inventory team, and she is the person every reviewer waits for when anything in the module changes.

There is a name for this condition. The truck factor of a body of code is the smallest number of people whose departure would leave nobody who understands it. (The Architecture book’s Section 13.3 calls the same idea the bus factor; this book uses the other name throughout.) Barcode issuance has a truck factor of one. The condition is common. When Guilherme Avelino and his colleagues estimated truck factors for 133 popular open-source projects on GitHub in 2016, they found that 65% had a truck factor of two or less. I spent a decade maintaining a widely used open-source library, and the number does not surprise me; most code that many people depend on is understood in depth by very few, and those few are seldom chosen for the role; they are usually the people who happened to stay.

Why can reissue not be added beside the module, as so much of Encore’s newer work has been? The answer lies in an assumption buried in the year-one code. When barcode issuance was written, a ticket’s identity was permanent: once issued, it belonged to that seat and that order for as long as the ticket existed. The code derives the identity from the order and the seat, and four other places in the codebase, written later, rely on recomputing it the same way. When rotating barcodes were added in year eight, the team built the per-ticket secret around the existing identity without changing it, and that was the right decision for rotation, because rotation never needed an identity to change. Reissue needs exactly that. A resold ticket must stop being the seller’s and become the buyer’s, under an identity that every scanner, every revocation list and every one of those four other places will treat as new. The assumption of permanence is not confined to one function. It is spread through the module and beyond it, and reissue must change it at the root.

An architect renovating an old building faces a similar problem with load-bearing walls. A partition can be moved in an afternoon, but a wall that carries the floor above must be shored up before it is touched, and the first job is to find out which walls are which. The analogy holds well in one respect: the work of shoring up, which in software means getting the code under tests that describe what it does today, comes before any change, and Chapter 2 teaches that work. It fails in another, and the failure is worth stating. A builder can see a load-bearing wall, or at least find it on the drawings. In old code the load-bearing assumptions are invisible until something leans on them, and the drawings, if they ever existed, were never updated. Michael Feathers’s definition of legacy code as code without tests, which Chapter 2 takes up, fits this reading: until the code is under test, nobody can say which of its assumptions bear load.

The five invariants as an engineering contract

The Products book made Verified Resale a promise of fairness, and fairness at Encore has a precise form. Five rules govern the product, and they are worded the same way everywhere they appear in this book.

1. Partners never buy primary tickets.
2. Only tickets bought through the Gate, within the per-fan limit, can be listed.
3. Organizer caps and rules can only tighten.
4. Every resale buyer passes Bot Screening and the per-buyer limits.
5. Partners never see sales-velocity data.

To a product manager these are commitments. To an engineer they are invariants: properties that must hold in every state the system can reach, whatever sequence of requests arrives and whatever fails along the way. The difference matters. A commitment is kept by people paying attention; an invariant is kept by code that makes violating it impossible, and by tests that would fail if the code ever allowed it. The fifth rule shows how much care the translation needs. Resale partners already read Encore’s availability endpoints during on-sales, as the Resilience book’s Section 9.5 noted, and the rule survives only because the availability they read is the same coarsened view that every caller gets, from which velocity cannot be derived. The third rule shows where enforcement must live. The organizer rule assistant can propose any rule it likes, but the typed rule set it produces cannot express a loosening, so the platform enforces the rule, and neither the partner nor the AI can break it.

Chapter 3 turns the five invariants into tests that run on every build, and Chapter 8 asks how they shape Verified Resale as a marketplace. Here the point is simpler. The invariants are the part of the bet that engineering must guarantee absolutely, and every one of them passes through code in Figure 1.4’s oldest rows.

What could go wrong

Large changes that cross many teams fail in a small number of recognizable ways, and naming them early lets each chapter address one. Each has a well-documented public case.

The first is the rewrite that never ends. Faced with old code that nobody wants to change, it is tempting to write new code that replaces it. Netscape released version 4.0 of its browser in 1997 and then rewrote the browser from scratch; there was never a version 5.0, and version 6.0 did not ship until November 2000. Joel Spolsky, writing in April 2000 as the 6.0 beta appeared, called the decision to rewrite from scratch “the single worst strategic mistake that any software company can make.” A rewrite discards the knowledge encoded in old code along with its defects, and it must reproduce years of behavior before it can ship anything. Chapter 2 explains why Encore changes barcode issuance in place.

The second is the big-bang integration. In April 2018 the British bank TSB moved about 5.2 million customers from the systems of its former parent, Lloyds, to a new platform over a single weekend. Up to about 1.9 million customers lost access to online banking, some for weeks. The independent review by the law firm Slaughter and May, published in November 2019, found that the new platform had not been ready and its testing had been inadequate. When 12 teams’ work meets for the first time at the end, every misunderstanding surfaces at once; Chapters 3 and 4 show how Encore’s teams integrate continuously instead.

The third is the dependency that blocks for a quarter. One team, busy with its own priorities, holds up 11 others because the work they need from it never reaches the top of its backlog. Chapter 10 treats this as a matter of planning and economics, and Chapter 11 tells what happened to Verified Resale’s.

The fourth is code generated with AI assistance that nobody understands. The Architecture book’s Section 15.4 and the Resilience book’s Section 4.5 told of Encore’s Notification rewrite, in which a coding agent rebuilt a service overnight and one engineer caught a path that would have issued duplicate tickets. Verified Resale will use such tools, and Chapter 5 asks how a team owns what a machine wrote.

The fifth is a platform gap that every team patches alone. The Architecture book’s Section 15.1 described what happens when teams carry more than they can: some sink into infrastructure, some skip it, and some solve it their own way. Encore’s three webhook senders are an example already in its code, and Verified Resale, which needs partner credentials and partner webhooks, would add more if nothing changed. Chapter 6 is about the platform group’s answer.

Recap. Verified Resale reaches engineering as a set of changes to existing code across 12 teams and 23 dependencies, and the hardest of them land in the oldest code, where a truck factor of one and an assumption of permanent ticket identity stand in the way. The five fairness rules become invariants that code must enforce and tests must prove. The failure modes of such a change are well documented, and each of the following chapters is, in part, a defense against one of them.

Exercise 1.3. Take the largest change your organization plans for this year. List every team it touches and, for each, the oldest code that team must change, with the year that code was written and the number of people who understand it well. Mark any dependency where that number is one.

1.4 The Levers

An engineering system has many moving parts, but the practices that improve it fall into four groups, each acting on a different part of the system. I will call them levers: places where deliberate effort changes how fast and how safely the whole system can change software. The four are design, flow, leverage and organization, and the chapters that follow are arranged around them. Figure 1.5 lays them out.

Figure 1.5 — The four levers of an engineering system, the chapters that teach each, and the way each fails when it is used without the others.

Design: keep code changeable

The first lever acts on the code itself. Lehman’s second law says that complexity rises unless work is done to hold it back, and the design lever is that work: keeping the code’s structure coherent, refactoring as understanding grows, recording technical debt as data so that it can be weighed against other work, and building the tests without which no change to old code is safe. Chapter 2 treats the codebase and Chapter 3 the testing strategy that makes change possible. For Verified Resale this lever decides whether barcode issuance can be changed at all.

Flow: shorten the loop

The second lever acts on the path a change travels from an engineer’s editor to the main line. Small changes, fast feedback, review that keeps up and continuous integration that people can trust are DORA capabilities with measured effects on delivery, and the reason is the economics of batch size, which Donald Reinertsen worked out for product development and which Chapter 10 takes up. Chapter 4 covers the development loop and Chapter 5 the AI assistance now part of it. For Verified Resale this lever decides whether 12 teams’ changes reach the main line in hours or in days.

Leverage: build platforms

The third lever solves a problem once for many teams. An internal platform takes on what every team would otherwise build for itself, a SaaS platform does the same for tenants, and an ecosystem extends it to partners; Chapters 6, 7 and 8 take them in turn. This lever comes with a condition, and it is fair to state it now. Platforms pay off when teams adopt them and when someone measures whether they help. DORA’s 2024 report found, as the Architecture book’s Section 15.5 recounts, that teams using an internal platform at first saw their delivery throughput and stability fall, by roughly 8% and 14%, even as individuals reported feeling more productive. A platform adds steps before it removes them, and Chapter 6 is about getting through that dip.

Organization: design the builders

The fourth lever acts on people: how teams are drawn, how leaders lead, how work is planned across teams and how it is funded. It is the lever Randell pointed at in 1968 when he proposed designing the group to get the system we want, and Team Topologies gave it a modern vocabulary of team types and interaction modes. Chapter 9 covers teams and leadership, and Chapter 10 planning, forecasting and funding. For Verified Resale this lever decides whether the Resale team exists in the right shape and whether the 23 dependencies are managed or designed away.

Why all four

In 1984 Eliyahu Goldratt and Jeff Cox published The Goal, a novel about a plant manager trying to save a failing factory, and its argument has outlived the factory. Every system that turns work into output has a constraint, a step slower than the others, and the throughput of the whole system is set by that step. Improving any other step improves nothing; work simply arrives sooner at the constraint and waits there. In one of the book’s best-known scenes the manager takes his son’s scout troop on a hike and notices that the line of boys moves at the pace of its slowest member, a boy named Herbie, however fast the boys at the front walk. When he puts Herbie at the head of the line and shares out the contents of Herbie’s overloaded pack among the stronger boys, the troop reaches camp together. The remedy is to find the constraint, make the most of it, arrange everything else around it, and then, once it is relieved, look for the new constraint, because there always is one. Goldratt developed this theory of constraints for manufacturing, where it is well proven; in software delivery it is widely used as a way of reasoning, and value-stream mapping is its usual instrument. I know of no controlled study showing that it holds for software teams, so I offer it as reasoning, supported by examples.

The reasoning explains why the four levers must be pulled together. Each acts on a different part of the engineering system, and each, pulled alone, tends to move the constraint somewhere the others act. Encore’s own history has two examples. In year five the fan-app team cut its build from 40 minutes to 12, a real achievement, and its median time from opening a change to merging it did not move, because the changes now reached review sooner and sat there just as long. The flow lever had been pulled in the tooling while review practice stayed as it was, and review was the constraint. In year six the platform team of the day built a shared deployment pipeline, announced it, and found 18 months later that two teams used it. The pipeline was good. Moving to it cost each team weeks that nobody had planned or funded, so the leverage lever had been pulled without the organization lever, and the constraint moved from tooling to adoption.

The clearest public case is recent, and it involves AI. DORA’s 2024 report found that as AI adoption rose, individuals and teams reported better documentation, better code quality and faster code review, while delivery throughput fell by an estimated 1.5% and delivery stability by 7.2% for every 25% increase in adoption. DORA’s own reading is that “improving the development process does not automatically improve software delivery—at least not without proper adherence to the basics of successful software delivery, like small batch sizes and robust testing mechanisms.” That is a lever pulled alone. Writing code got faster; the work moved downstream, to review, testing and integration, and the system as a whole did not speed up. It leads directly to the last section of this chapter.

Recap. Four levers improve an engineering system: design keeps code changeable, flow shortens the loop, leverage solves problems once for many teams, and organization shapes the people who build. Every system has a constraint, and a lever pulled alone moves work to the constraint faster without relieving it. That is why this book pulls all four, and why each chapter points to the others.

Exercise 1.4. Name the last significant improvement your organization made to how it builds software. Say which of the four levers it pulled, then find out where work waited longest before the improvement and where it waits longest now. If the answer to both is the same place, say why.

1.5 What the AI Era Changes for Building

Code got cheaper; change did not

On Alphabet’s earnings call of 29 October 2024, Sundar Pichai said that “more than a quarter of all new code at Google is generated by AI, then reviewed and accepted by engineers.” The sentence has been quoted widely as evidence that writing code has become cheap, and in one sense it is. A competent engineer with a good assistant can produce a working function, a test, a migration script or a first draft of a service far faster than a few years ago, and the cost of producing code continues to fall. The Architecture book’s Section 15.4 argued what this means for architecture. This book takes the engineering consequence, which is less comfortable.

The second half of Pichai’s sentence carries that consequence: the code is “reviewed and accepted by engineers.” Writing code was never most of the cost of changing software. Understanding the code that is already there, deciding what to change, reviewing the change, testing it, integrating it with everyone else’s, deploying it safely and then owning it for years: these were always the larger part, and AI assistance has made the smallest part cheaper while leaving the larger parts where they were, or making them larger, because there is now more code to review, test and own. Dijkstra, in the same Turing lecture that made Garmisch’s crisis famous, gave its cause as machines that had “become several orders of magnitude more powerful,” so that the programs people attempted outran their ability to reason about them. The echo is hard to miss. When the means of producing software become much more powerful, the limit moves to the means of understanding it. Encore’s Notification rewrite showed this on a small scale: the agent wrote a service overnight, and its safety rested on one engineer who read it closely enough to find a path that issued duplicate tickets.

The evidence in one paragraph

The research on AI-assisted development is young and its results disagree, which is itself informative; Chapter 5 examines it in detail. Controlled experiments on small, self-contained tasks have found large speedups, and field experiments inside companies have found smaller, though still real, gains. The most careful study of experienced developers working in their own mature repositories points the other way: in METR’s (Model Evaluation and Threat Research) randomized trial, published in July 2025, 16 experienced open-source developers working on 246 real issues took 19% longer when AI tools were allowed, although they had forecast a 24% speedup and afterwards believed they had been about 20% faster. METR has since described that result as out of date and published a follow-up in February 2026, so the specific figure should not be leaned on; the gap between how fast people felt and how fast they were is the more durable lesson. At the level of delivery, DORA’s 2024 report found modest gains in organizational and team performance as AI adoption rose, alongside the falls in throughput and stability described in Section 1.4. Read together, the studies suggest that the gains are real, uneven and largest where the surrounding engineering system is ready to absorb them.

What becomes more valuable

If code is cheaper and change is not, the parts of the engineering system that turn code into safe change become more valuable. Tests that pin down behavior let a machine-written change be checked against what the system must do. Review keeps pace only when changes are small enough to review. Clear design lets an assistant’s context and a reviewer’s understanding cover the same ground, fast builds tell both of them quickly whether a change works, and platform paths make the safe way the easy way for every change, whoever or whatever wrote it. DORA’s 2025 report on AI-assisted software development, drawn from about 5,000 respondents, frames AI as an amplifier: it strengthens what an organization already does well and magnifies what it does badly. That is a finding about the engineering system, and it is the reason this book gives AI assistance one chapter of its 11. Chapter 5 covers the practice, and the Resilience book’s Section 4.5 covers the policy for machine-written changes on their way to production.

An analogy from aviation is tempting here. As cockpits gained autopilots, the pilot’s work shifted from flying to monitoring, a history the Operations book’s Chapter 3 takes up, and as assistants write more code, the engineer’s work shifts in the same way toward reading, judging and integrating. The analogy breaks where it matters most. An autopilot flies a known aircraft through a known envelope, while an assistant writes new code into a system whose envelope is years of accumulated complexity, and the engineering system is what gives the person reviewing that code a fighting chance.

The rule for this book

This book recommends only practices with evidence behind them: practices that are widely used, with research or long industry experience to show that they work. Where a practice is young, as much of AI-assisted development still is, the book labels it emerging, says so plainly, and gives the best real account of its use. Where Encore does something the evidence does not yet support, the book says that too. The Products book follows the same rule with the same labels, so that the two books, and the series, can be read as one argument.

Recap. AI assistance has made writing code cheaper and left the larger costs of changing software, understanding, review, testing, integration and ownership, where they were or heavier. The evidence so far shows real but uneven gains, a persistent gap between perceived and measured speed, and a strong dependence on the engineering system around the tools. Practices that turn code into safe change gain value as code gets cheaper, and this book recommends only those that are proven, labelling the rest.

Exercise 1.5. Find one change in your codebase from the last month that was largely written with AI assistance. Trace what checked it before it reached production: which tests, which reviewer, which pipeline stages. Then ask the reviewer how long the review took and how confident they were, and compare that with a change of similar size written by hand.

Kata: The Tariff Deadline

Your brief: “Northgrid.” Northgrid sells software to energy retailers: billing, metering, customer accounts and switching, run as SaaS for 30 retailers in four EU countries. It has 250 engineers in 28 teams, including a platform team of 12. Its billing core was written in the company’s first year, nine years ago, by a team of six; it calculates every invoice Northgrid’s customers send, about 11 million a month, and two of its original authors remain, one of whom is about to retire. The core lives in a shared repository whose CI takes 48 minutes, and changes to it wait a median of four days for review. A regulator in one of Northgrid’s countries has published a new dynamic tariff, priced by the hour from the wholesale market, that every retailer must be able to offer within 14 months. Northgrid’s first plan shows the change touching nine teams with 19 cross-team dependencies, and its last developer survey named flaky end-to-end tests, review waits and “nobody understands billing” as the top three pains. The leadership team has proposed two answers: rewrite the billing core, or buy an AI coding assistant for every engineer.

Deliverables:

1. Value-stream map — Draw the path a change to the billing core takes from idea to production, with every team, tool, handoff and wait, and estimate the time at each step. Mark where work waits longest.
2. Engineering-system profile — In one page, describe Northgrid’s engineering system as Section 1.2 described Encore’s: the codebase and its oldest parts, the path to production, the teams and their ownership metadata, and what engineers say hurts. Note which default decisions from earlier stages still shape it.
3. Dependency map with code age — List the 19 dependencies of the tariff change as Figure 1.4 lists Verified Resale’s, adding the age of the oldest code each one touches and its truck factor. Identify the dependencies most likely to stall the change.
4. Invariants — Write the rules the dynamic tariff must never break, such as an invoice that always reconciles with metered consumption, as invariants a test could check.
5. The first lever — Choose which of the four levers to pull first, and say where the constraint will move once it is pulled and which lever you will need next. Address both of leadership’s proposals directly, with evidence.

Rubric:

Where you now stand

You can now see the engineering system as a system in its own right: the code, tools, platforms, teams and plans that decide how fast and how safely a product can change. You can map how a change travels through it, read the defaults its history left behind, tie a large change’s dependencies to the condition of the code beneath them, and choose among four levers knowing that each pulled alone moves the constraint somewhere else. You have also met the change this book will follow, Verified Resale, and the module at its center, barcode issuance, with its one remaining author and its assumption that a ticket’s identity never changes. Before any of the new code can be written, that old code must be understood, put under test and made safe to change. Chapter 2 begins there, with the codebase: what complexity is, how debt accumulates and is measured, and how an organization changes code it did not write and does not fully understand.

References

• Peter Naur, Brian Randell (eds.) — Software Engineering: Report on a Conference Sponsored by the NATO Science Committee, Garmisch, Germany, 7th to 11th October 1968. NATO Scientific Affairs Division, 1969 — free online.
• J. N. Buxton, B. Randell (eds.) — Software Engineering Techniques: Report on a Conference Sponsored by the NATO Science Committee, Rome, Italy, 27th to 31st October 1969. NATO Scientific Affairs Division, 1970 — free online.
• Brian Randell — The 1968/69 NATO Software Engineering Reports. Dagstuhl Seminar 9635, History of Software Engineering, 1996 — free online.
• Edsger W. Dijkstra — The Humble Programmer. Communications of the ACM 15(10), 1972 — free online.
• Titus Winters, Tom Manshreck, Hyrum Wright — Software Engineering at Google: Lessons Learned from Programming Over Time. O’Reilly, 2020 — free online.
• Nicole Forsgren, Jez Humble, Gene Kim — Accelerate: The Science of Lean Software and DevOps. IT Revolution, 2018.
• Google Cloud and DORA — Accelerate State of DevOps Report 2024. Google Cloud, 2024 — free online.
• Abi Noda, Margaret-Anne Storey, Nicole Forsgren, Michaela Greiler — DevEx: What Actually Drives Productivity. ACM Queue 21(2), 2023 — free online.
• M. M. Lehman — Programs, Life Cycles, and Laws of Software Evolution. Proceedings of the IEEE 68(9), 1980.
• Melvin E. Conway — How Do Committees Invent?. Datamation, April 1968 — free online.
• Joel Becker, Nate Rush, Beth Barnes, David Rein — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. METR, 2025 — free online.
• Guilherme Avelino, Leonardo Passos, André Hora, Marco Tulio Valente — A Novel Approach for Estimating Truck Factors. Proceedings of the 24th IEEE International Conference on Program Comprehension, 2016 — free online.
• Matthew Skelton, Manuel Pais — Team Topologies: Organizing Business and Technology Teams for Fast Flow, 2nd ed.. IT Revolution, 2025.
• Eliyahu M. Goldratt, Jeff Cox — The Goal: A Process of Ongoing Improvement. North River Press, 1984.
• Joel Spolsky — Things You Should Never Do, Part I. Joel on Software, 2000 — free online.
• Slaughter and May — Independent Review of the TSB IT Migration. TSB Bank, 2019 — free online.