Building Software at Scale in the AI & Cloud Era

← the whole book on one page

Chapter 5: Building with AI Assistance

“If the library is well-stocked, programming has been reduced to a matter of hours, rather than weeks.” — Grace Murray Hopper, 1952

What does a machine that writes code actually change about building software with 400 engineers, and how would we know? The question sounds new, and the tools that raise it are new, but the promise behind it is one of the oldest in our field. Every generation of programmers has been told that programming is about to write itself. Each time, something real happened: the cost of producing some part of a program fell sharply. And each time, the work that remained turned out to be the harder part, the part that had been hidden behind the labor of typing instructions. The honest way to approach today’s assistants is to ask which of those two things is happening now, and in what proportion, and to let the evidence answer.

The story that frames this chapter begins in 1952, when Grace Hopper described a program that assembled other programs out of a library of routines, and her colleagues declined to believe it could work. Section 5.1 develops that story, because it teaches something the present debate keeps forgetting: the doubters of automatic programming were partly right, its champions were partly right, and the work moved up a level without disappearing. We then examine the modern evidence in detail. This chapter is the series’ home for that evidence, and where it is young we say so.

Encore is not deciding whether to adopt these tools. It adopted them years ago, piecemeal, as every company did. A coding agent rebuilt the Notification service overnight in year seven, and by the autumn of year eight a quarter of Encore’s merged changes carried an agent label, as the Resilience book’s Section 4.5 recorded. What Encore has never done is measure what the tools are worth. Verified Resale is the first large change built with them from the start. An agent migrates 60 call sites off the legacy barcode format, an assistant drafts the hand-written parts of the partner SDKs (software development kits), and a graduate engineer on the new Resale team learns the event-sourced seat ledger with an assistant at his side. Meanwhile Encore’s leadership has asked a plain question, whether the tools pay, and a small working group has answered it with a measurement plan. The policy for reviewing agent changes already exists, in the Resilience book’s Section 4.5, and this chapter does not rewrite it. Our subject is the other side of the work: what the author owes before review begins, and what the organization owes before it claims a result.

5.1 The Evidence

Automatic programming, 1952

Grace Murray Hopper learned to program on the Harvard Mark I during the Second World War. In 1949 she joined the Eckert-Mauchly Computer Corporation in Philadelphia, which Remington Rand bought the following year, and there she helped program the UNIVAC I, the first large commercial electronic computer in the United States. Programming it meant writing machine instructions by hand, and much of the writing was the same routines, recopied and readdressed for every new problem. In 1951 and 1952 Hopper and her group built a program they called the A-0 compiling routine. The word “compiler” meant something narrower then than it does now. The mathematician wrote a specification that named subroutines by their call numbers in a catalogued library, together with their arguments. A-0 fetched those subroutines, adjusted their addresses so that they fit together, and produced a complete machine-code program, which was then run. In today’s terms it was closer to a linking loader with a subroutine library than to a translator of formulas, and it was not alone: Alick Glennie’s Autocode at Manchester and Laning and Zierler’s algebraic system at MIT have competing claims to being first. A-0 was one of the earliest such systems, and it is the one that gave the word “compiler” its software meaning.

Hopper described the scheme at the Association for Computing Machinery’s national meeting in Pittsburgh in May 1952, in a paper called “The Education of a Computer”. “The programmer may return to being a mathematician,” she wrote. “He is supplied with a catalogue of subroutines.” She saw, earlier than most, where the cost of computing was going: “With the computer paid for, the cost of programming and the time consumed, comes to the notice of vice-presidents and project directors.” And she was candid about the price. Standardized subroutines, she noted, lose “a small amount of time … in performing duplicate data transfers which could be eliminated in a tailor-made routine”. Her answer was to let the UNIVAC produce the program and then have a person remove the duplication by hand. That is the 1952 form of a practice this chapter will keep returning to: let the machine generate, then have a person refine and own the result. Figure 5.1 sketches the arrangement.

Figure 5.1 — A-0’s scheme, drawn as a schematic. The mathematician names subroutines by call number; A-0 fetches them from the library and fits them together; a person removes the duplicate transfers by hand. The machine generates, and a person refines and owns.

The reception was cool. In 1987 Hopper told an interviewer, in a newsletter profile, “Nobody believed that. I had a running compiler and nobody would touch it. They told me computers could only do arithmetic.” It is reported speech, recalled 35 years later, and should be read as such, but John Backus’s own account of the period describes the same resistance. Writing in 1976, Backus recalled the programmers of the early 1950s as “members of a priesthood guarding skills and mysteries far too complex for ordinary mortals”. To them, he wrote, “it was obviously a foolish and arrogant dream to imagine that any mechanical process could possibly perform the mysterious feats of invention required to write an efficient program.” The phrase of the day for the whole enterprise was automatic programming: the hope that a machine could produce a working program from a description less detailed than the program itself. The Office of Naval Research held a symposium under that name in May 1954.

Two things about this history matter for us. First, the skeptics were not simply wrong. A-0’s programs really were less efficient than hand-tuned code, and efficiency was a serious matter on machines whose time was scarce and expensive. It took FORTRAN’s optimizing compiler, delivered in 1957, to show that generated code could be good enough that most programmers would stop checking it. The Architecture book’s Section 15.4 uses that FORTRAN moment as a passing parallel to today’s agents and shows where the parallel breaks: a compiler is deterministic and its behavior is specified, and a model is neither, so we can never come to trust the generator the way programmers came to trust FORTRAN. Our trust must rest on checks of what comes out. Second, consider what happened to the work. Fred Brooks, in “No Silver Bullet” (1986), listed automatic programming among the hopes that would not deliver an order-of-magnitude gain, and cited David Parnas’s judgment that the phrase had only ever meant programming in a language one level higher than the programmer currently had. Hopper’s library did not abolish programming. It moved programming up a level, from instructions to specifications, and the specification turned out to be where the difficult thinking had always lived.

Controlled studies of single tasks

The cleanest evidence comes from a randomized controlled trial (RCT): an experiment in which participants are assigned by chance to work with or without the tool, so that any difference in outcome can be attributed to the tool rather than to who chose to use it. Two such trials anchor the early evidence.

The first, by Sida Peng and colleagues, ran from May to June 2022, just before GitHub Copilot became generally available. The authors recruited 95 freelance developers through Upwork and randomized 45 of them to use Copilot and 50 to work without it. Everyone received the same task: implement an HTTP server in JavaScript, checked by 12 tests that participants could see but could not edit. Among those who finished, the treated group took 71.17 minutes on average and the control group 160.89, which the authors report as 55.8% faster, with a 95% confidence interval of 21% to 89%. The treated group’s success rate was seven points higher, but that difference was not statistically significant. Less experienced developers, those who spend more hours a day coding, and developers aged 25 to 44 gained the most. Asked to estimate their own gain, participants in both groups guessed about 35%, well below the 55.8% that was measured. They underestimated what the tool had done for them.

The second trial, by Elise Paradis and colleagues at Google, published in 2025, put 96 full-time Google engineers on one complex, enterprise-grade task in the summer of 2024, with or without three AI features in their internal tools. The best estimate was about 21% less time with the features, and the authors describe the confidence interval as large. Engineers who spend more hours a day on code gained more. The authors caution that Google’s internal tooling may not generalize to other companies.

Both studies are well designed, and both are narrow by construction. Each measured one bounded task with a clear definition of done, the setting in which assistance should help most, and it did. Such trials cannot tell us what happens to a codebase that lives for years, or to the team that reviews the output.

Field experiments

The best causal evidence from real work is a set of three randomized field experiments run at Microsoft, at Accenture and at an anonymous Fortune 100 company, analyzed by Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng and Tobias Salz. The working paper circulated from 2024, and the study was published in Management Science online on 27 February 2026. Each experiment was run by the company “as part of their ordinary course of business”: chance decided which developers were given GitHub Copilot, a completion assistant, and which were not. Pooled across the three experiments and 4,867 developers, access to the assistant raised the number of completed tasks by 26.08%, with a standard error of 10.3%. “Less experienced developers had higher adoption rates and greater productivity gains.”

Three cautions belong with the number. The abstract itself says that “each experiment is noisy and results vary across experiments”, so the pooled estimate is real but imprecise. “Completed tasks” were measured mainly as pull requests, which counts output and says nothing directly about whether the output was worth having. And the tool was a completion assistant, so the result describes that class of tool and says nothing directly about agents. The study’s lasting contribution is its method: it shows that a company can randomize access to a developer tool inside its ordinary operations and learn something true. Section 5.5 borrows that design.

The study that disappointed

In July 2025, METR (Model Evaluation and Threat Research) published a trial that set the rest of the evidence in a different light. Joel Becker, Nate Rush, Elizabeth Barnes and David Rein recruited 16 experienced open-source developers to work on 246 real issues in their own repositories, which averaged more than 22,000 stars and more than a million lines of code. Each issue, roughly two hours of work, was randomly assigned to be done with AI tools allowed or not; the tools were mostly Cursor Pro with Claude 3.5 and 3.7 Sonnet, and the work ran from February to June 2025. With AI allowed, tasks took 19% longer, with a confidence interval of 2% to 39% longer. Before starting, the developers expected a 24% speedup. Afterward, they believed they had been 20% faster.

I spent a decade maintaining a widely used open-source library, and the setting METR chose is one I recognize: a large, mature codebase whose maintainers carry years of unwritten context about why things are the way they are, and whose quality bar is set by thousands of users who will notice any regression. There an assistant has the least to offer and the most to get wrong. METR was careful to say what its result does not show. It does not show that AI tools fail to speed up most developers, nor that no better way of using them exists. It shows that in this demanding setting, with early-2025 tools, experts were slower and did not know it. Their perception was wrong by about 40 percentage points, in the opposite direction from Peng’s participants, who had underestimated their gain. Together, the two studies teach this section’s most useful lesson: developers’ sense of how much a tool helps them is unreliable in both directions, and no organization should decide anything on the strength of it. Figure 5.2 sets the two trials side by side.

Figure 5.2 — What developers believed against what two randomized trials measured. Peng’s participants underestimated their gain; METR’s experts believed they were faster when they were slower. Sources: Peng et al. (2023); Becker et al., METR (2025).

METR has since marked the 2025 results as out of date, and its follow-up shows how hard the measurement has become. A second study, begun in August 2025 with 57 developers (10 returning and 47 new), 143 repositories and more than 800 tasks, estimated an 18% reduction in time for returning developers (confidence interval from 38% faster to 9% slower) and 4% for new ones (from 15% faster to 9% slower). Both intervals include zero. METR called the result “an unreliable signal”: developers who did not wish to work without AI declined to take part, between 30% and 50% of participants withheld tasks they did not want to do unaided, and time-tracking broke down for developers running several agents at once. METR is redesigning the study, so the effect on experienced developers in mature code has not been reliably measured since 2025. A METR survey from May 2026 adds a final caution. Of 349 technical workers who responded, at a response rate of about 2%, the median reported that AI had made them 3× faster. Beside the 2025 trial’s gap between belief and fact, that figure shows how far perception can run from measurement, and nothing more.

What DORA sees at the team level

The trials measure individuals on tasks. DORA (DevOps Research and Assessment), the research program behind the delivery metrics most engineering organizations use, measures teams, through large annual surveys. Its 2024 report found that as AI adoption rose, delivery suffered: “As AI adoption increased, it was accompanied by an estimated decrease in delivery throughput by 1.5%, and an estimated reduction in delivery stability by 7.2%.” Those estimates are for each 25% increase in adoption, and the same increase went with better documentation quality (7.5%), code quality (3.4%) and code review speed (3.1%). The craft of the individual improved while delivery slowed and became less stable.

The 2025 report, State of AI-assisted Software Development, drew on nearly 5,000 technology professionals and more than 100 hours of qualitative data. Adoption had become “near-universal: 90%”. More than 80% of respondents believed AI had raised their productivity, and 30% reported little or no trust in the code it generated. The central finding changed sign on one measure and not on the other: “Unlike last year, we observe a positive relationship between AI adoption on both software delivery throughput and product performance. However, AI adoption does continue to have a negative relationship with software delivery stability.” Both halves must travel together. Teams that adopted more tended to ship more, and their delivery tended to be less stable. DORA framed AI as an amplifier of a team’s existing strengths and dysfunctions alike, and published an AI Capabilities Model, to which Section 5.5 returns.

DORA’s data carry their own conditions. They are self-reported, from self-selected respondents, and the findings are correlations, so they cannot say whether adoption causes instability or whether teams that adopt eagerly differ in other ways. What they offer is breadth: thousands of teams, in every kind of company. The Resilience book’s Section 4.5 has already taken the stability finding into Encore’s review policy. Sections 6.5 and 10.5 of this book point back here for the numbers.

How to read all of it

Figure 5.3 gathers the studies in one place, with the setting each one measured and the most important thing it cannot tell us.

Figure 5.3 — The evidence on AI assistance and productivity. Each study is sound within its setting, and the settings differ more than the headlines do.

Read together, the studies are less contradictory than they first appear, because they measured different situations. Gains are largest on bounded, well-specified work and on work unfamiliar to the person doing it: a fresh server in a language one half-knows, a task in a codebase one has not lived in. Gains shrink, and can reverse, where experts work in mature code under high quality bars, because there the assistant contributes least of what the expert lacks and most of what the expert must then check. Perception is unreliable in both directions, so no organization can learn the answer by asking its engineers. Team outcomes depend on the system around the tool, the tests, review, batch size and platform that must absorb a higher volume of change. And as Section 5.4 will show, the velocity gained early can be bought with complexity that slows the same team later.

Brooks’s distinction between essential and accidental complexity, introduced in the Architecture book’s Section 1.2, makes this pattern intelligible. An assistant attacks accidental complexity very well: recalling an API, writing the boilerplate around a query, translating a test from one framework to another. It does nothing for essential complexity, such as deciding what a resold ticket means to a door scanner that has no network, or which of five fairness rules a new endpoint might violate. Where most of a task is accidental, the gain is large. Where most of it is essential, the assistant’s speed in the accidental part is spent checking its guesses about the essential part. Hopper’s library worked the same way. It removed the drudgery of recopying routines and left the mathematician with the specification, which was the hard part all along.

Recap. Controlled trials show real gains on bounded tasks, and field experiments show smaller, noisy gains in real work, largest for less experienced developers. Experts in mature code were slower in the one careful trial of that setting, and believed the opposite. Team-level data show throughput rising with adoption while stability falls. The pattern follows Brooks’s distinction: assistance removes accidental complexity and leaves the essential kind exactly where it was.

Exercise 5.1. For one recent task done with an assistant, write down how long you think it took and how long it would have taken without the tool. Ask a colleague who knows the task to make the same two estimates independently. Then check the timestamps in your version control and ticket history, and compare all three numbers.

5.2 Where It Helps

Section 5.1 predicts where assistance should pay: on work whose difficulty is mostly accidental, whose correctness machinery can check, and whose result a person can review without reconstructing every decision behind it. The strongest published accounts of assistance at scale come from exactly that kind of work.

Migrations at scale

A migration, in this sense, is a change of the same kind applied at many places in a codebase: replacing a deprecated library, widening a type, moving from one testing framework to another. Section 2.5 described how large organizations make such changes, by splitting one logical change into many small shards, each reviewed and landed on its own. Migrations suit assistance unusually well, because the rule is known in advance, each instance is small, and the existing tests say whether an instance broke anything.

Google has published the most detailed account. In “How is Google Using AI for Internal Code Migrations?” (2025), Stoyan Nikolov and colleagues describe migrating identifiers in Google Ads, a codebase of more than 500 million lines, where “dozens” of ID types were defined as 32-bit integers and used in “tens of thousands of code locations across thousands of files”. The team located the code with Kythe, Google’s code-indexing system, prompted a model per change type, and sharded the results to the code’s owners. Measured character by character, comparing the model’s first output with what was finally committed, “80% of the code modifications in the landed CLs were fully AI-authored”. (A CL, or changelist, is a change submitted for review.) Engineers estimated that the total time, including review and landing, fell by about 50%. The paper is equally plain about the rest: in most cases “the human needed to revert at least some changes the model made”. A companion paper, “Migrating Code At Scale With LLMs At Google” (2025), by Celal Ziftci and several of the same authors, reports 39 distinct migrations carried out by three developers over twelve months: 595 code changes and 93,574 edits, of which 74.45% of changes and 69.46% of edits were generated by the model, again with an estimated halving of migration time. Every change passed automated checks including the build and regression tests, was inspected by its developer, and was reviewed by the code’s owners.

Two cautions keep these numbers honest. The papers share authors and describe one family of migrations within one company, so they are best read as two measurements of one program in different units (characters of landed code in one, changes and edits in the other), and they do not confirm each other independently. The time savings, too, are the engineers’ own estimates, which nobody measured under control. A second company supplies some independence. Airbnb reported in March 2025 that it had migrated nearly 3,500 React component test files from Enzyme to React Testing Library in about six weeks, against an estimate of a year and a half by hand.

Publishing offers a fair comparison. When a publisher changes its house style, a copy editor can apply the new rule across a whole backlist quickly, because the rule is explicit and each instance can be checked against the style sheet. The work goes wrong only where a sentence’s meaning depended on the old form, where changing a word mechanically would change what the author said. A good copy editor flags those sentences and leaves them to the author. The comparison holds well for migrations and fails at one point: a copy editor knows when meaning is at stake and stops, while a model applies the rule with the same confidence everywhere. Somebody has to find the sentences whose meaning will change, and that somebody is the owner of the code.

Encore’s own migration had exactly this shape. Barcode issuance, the eight-year-old core described in Chapter 2, embeds an event-code prefix in its legacy barcode format, and over the years 60 places in Encore’s code had learned to read that prefix directly; none of them asked Seat Inventory which event a ticket belonged to. Verified Resale gives every ticket an opaque ticket_id, and reissue makes the prefix unreliable, so all 60 call sites had to move to the new identity lookup. They sat in 11 repositories owned by seven teams. Ines, a staff engineer on the Inventory team, which the draw described in Section 5.5 had placed in the first cohort to receive background agents, sponsored the migration. Once the characterization tests described in Chapter 2 were in place, she ran it much as the Google papers describe: a background agent generated one change per call site, and a ladder of checks ran before any human owner saw a line of it. Figure 5.4 shows the ladder.

Figure 5.4 — The validation ladder for Encore’s barcode call-site migration, with what each rung caught. Each rung is cheaper than the one after it, so the cheap machine rungs run first and spare the expensive human ones.

Tests and boilerplate

Tests and boilerplate are the next obvious candidates, and the evidence is thinner: no controlled study has measured the quality of tests that assistants write. What we have is practitioner guidance, from Addy Osmani’s Beyond Vibe Coding and from Gene Kim and Steve Yegge’s Vibe Coding, both published in 2025, together with Airbnb’s migration, which was itself a migration of tests. The guidance converges on one condition. A test has two parts: the setup that drives the code, and the test oracle, the part that decides whether the output is correct. Generating the setup is cheap and largely safe. Generating the oracle is where the danger lies, because a test whose oracle was inferred from the code under test will faithfully confirm whatever the code does, bugs included.

So the condition for using generated tests is that a person writes or approves the oracle, and that the author of the code, human or agent, cannot quietly change it. Peng’s experiment used exactly this arrangement, with tests participants could see but not edit, and Encore’s barcode migration used it on rung 2 of Figure 5.4. The Resilience book’s Section 4.5 explains why the second half of the condition matters, with the evidence that agents faced with tests they cannot honestly pass often edit the tests instead; we take that finding as given here. What this chapter adds is the author’s habit. When an assistant drafts tests for code you are writing, read the assertions before anything else, and ask of each one whether you would have written that expectation yourself, from the specification, without looking at the code.

The partner SDKs show where boilerplate generation fits in Encore’s work. The client libraries in TypeScript, Python and Java are generated deterministically from the OpenAPI contract of the partner application programming interface (API), as any well-run API program does; that generation is ordinary tooling, with no model involved. What an assistant drafted were the hand-written parts around the generated core: usage examples, documentation, and small idiomatic helpers for retries with idempotency keys. Partners are promised 12 months’ notice of any breaking change, so the SDKs’ public surface runs through the same breaking-change differ as the API itself. In the first month the differ stopped an assistant-drafted helper that had renamed a public method to something tidier, a rename that would have broken every partner who called the old name.

Prototypes and throwaway code

Osmani distinguishes two ends of a spectrum. At one end is “vibe coding”, accepting generated code by its apparent behavior without reading it closely; at the other is AI-assisted engineering, in which a person owns the design, reviews the output and verifies it. The framing is a practitioner’s, and it lets us say plainly where the first end belongs: spikes, demos, internal one-offs, and the clickable prototypes that the Products book’s Section 3.5 uses in discovery. Code whose only job is to answer a question, and which will be discarded once the question is answered, can be written by feel, because nobody will maintain it.

The engineering rule is that throwaway code must actually be thrown away, and it is the rule most often broken, because a prototype that works is a standing temptation to ship. Section 5.4 describes the measured cost of yielding: a brief rise in velocity and a lasting rise in warnings and complexity. Encore keeps prototypes structurally apart. The Resale team’s discovery prototypes lived in a sandbox repository with no deployment pipeline to production, no access to production credentials and an expiry date in its README. When the team decided to build the fan resale module for real, it started from the prototype’s screens and lessons and wrote the code again on the golden path.

Where it does not help yet

The same pattern that predicts where assistance helps predicts where it does not. It helps least in mature hotspots, the files that change often, break often and carry years of accumulated reasons; on invariants that span services, which no single file states; and on ambiguous requirements, where the hard part is deciding what to build. Osmani calls the experience the “70% problem”: an assistant takes a task most of the way quickly, and the last 30%, the edge cases, the integration, the security and the maintainability, takes as long as it ever did, or longer, because it must now be done inside code the engineer did not write. The 70 and the 30 are a metaphor, not a measurement, and should not be quoted as data. But its shape fits METR’s 2025 trial, and the Stack Overflow Developer Survey of 2025, in which 84% of roughly 49,000 respondents used or planned to use AI tools and about two-thirds named solutions that are “almost right, but not quite” as a frustration.

Encore’s five fairness invariants show the problem concretely. Invariant 3 says that organizer caps and rules can only tighten. No single file in the codebase states it; it is enforced by the type of the rule set in the Resale service, by a check in the organizer console, and by tests in CI. An assistant adding a rule option sees only its file, and nothing there says that an option which loosens a default breaks a promise Encore has made to every organizer. The invariant lives in the system and in the heads of the people who designed it, and an assistant can respect it only once someone has written it where the assistant will read it.

Understanding unfamiliar code is a harder case to judge, and it deserves an honest label: the evidence here is practitioner experience only. No controlled study has isolated it, but two adjacent findings point in a useful direction. Peng and Paradis both found larger gains on unfamiliar tasks and stacks, and a 2026 study discussed in Section 5.5 found that engineers who used an assistant to ask conceptual questions kept their understanding, while those who delegated the work lost it. The practice that follows is modest: ask the assistant to explain the code, then verify the explanation against the code and tests before acting on it. Encore’s engineers used it this way on the barcode module, whose last remaining author had been its only reader for years. The assistant’s account of the issuance path was mostly right and once confidently wrong, about the order in which a retry wrote its two records. The characterization tests caught the error before anyone built on it.

Recap. Assistance pays best on migrations, with a human oracle on tests, and on code that is truly thrown away. It pays least on mature hotspots, cross-service invariants and ambiguous requirements, where the essential complexity sits. Explanation of unfamiliar code is useful on practitioner evidence alone, provided the explanation is checked against the code before anyone relies on it.

Exercise 5.2. Sort your team’s last month of completed work into the four categories of this section: migrations, tests and boilerplate, throwaway code, and the work where assistance does not help yet. For each item, note whether an assistant was used and whether, looking back, it paid. Compare the pattern with Figure 5.3.

5.3 Agents in the Loop

From completion to agents

The tools have changed shape quickly. Four classes are now in common use. Inline completion suggests the next few lines as the engineer types. Chat answers questions and drafts code that the engineer pastes in. Agentic editing, inside the engineer’s editor, plans a change across several files, runs the tests and revises its work while the engineer watches. And a background agent takes a task, works on it unattended in its own environment, and returns a pull request for a person to review. What changes at each step is who holds the loop. With completion, the engineer decides every line, and the tool only proposes. With a background agent, the agent makes hundreds of small decisions the engineer never sees, and the engineer meets the result as a reviewer meets a colleague’s change.

Kim and Yegge describe the work in three loops of growing scale, an inner, a middle and an outer developer loop, reaching from a single working session out to the work of teams, each with its own practices for preventing, detecting and correcting mistakes. Controls belong at every scale: a checkpoint in the editor and a canary in production are the same idea at two loop sizes. The shift is visible in published evidence, too. METR’s 2026 follow-up reported developers running several agents at once, often enough that per-task time tracking stopped working.

Aviation offers a useful comparison. Autopilots work in modes, and the pilot’s job changes with the mode, from flying the aircraft to monitoring the system that flies it. Human-factors research has long recognized mode confusion, a pilot’s mistaken belief about which mode is engaged, as a hazard in its own right, and cockpit design answers it by displaying the active mode at all times. The comparison carries over well: an engineer must always know what an agent is allowed to do on their behalf, and the tooling should make that visible. It breaks at certification. An autopilot’s modes are few, specified and tested to a standard before any passenger boards, while an agent’s behavior shifts with each model version and has no such specification, which is why the controls in the rest of this section sit around the agent instead of inside it.

Specs and context

A background agent knows only what it is given and what it can find. The engineer’s first contribution to an agent’s work is therefore written: a task specification, the repository’s rules file, and the acceptance tests that define done. Each is an old artifact with a new reader: a ticket written carefully enough for a stranger to act on, the conventions a new team member would otherwise learn in review, and tests that say what done means. The Architecture book’s Section 15.4 argued that ADRs (architecture decision records) become agent context; the artifacts here are the engineer’s, one level down.

The evidence that context matters is young but real. DORA’s AI Capabilities Model names “AI-accessible internal data” as a capability that “boosts AI’s impact on individual effectiveness and code quality”, a correlational survey finding. Google’s migration papers document the context they supplied: the set of files to change, located with Kythe, and a prompt written for each change type. John Berryman and Albert Ziegler, in Prompt Engineering for LLMs (2024), describe how GitHub Copilot assembles context under a fixed token budget, choosing what to include because not everything fits. The practical consequence for an engineer is that a good spec is short and specific. It names the files, the rule, the tests that must stay green and the things the agent must not touch, because whatever the spec leaves out, the agent fills in with a guess.

Here is the spec Ines wrote for one shard of the barcode migration, the Fan Messaging team’s:

task: replace barcode event-prefix parsing with ticket identity lookup
shard: fan-messaging (repo: notifications, 6 call sites)
locate: callers of BarcodeLegacy.eventPrefix()   # list attached, from code index
change: |
  Replace each call with ticketIdentity.eventOf(ticket_id).
  Pass ticket_id from the message envelope; never derive it from the barcode.
must_stay_green: [characterization/barcode_prefix, contract/notifications-consumers]
do_not_touch: [tests/**, build files, retry or timeout settings, any public API]
size: one change per call site; no unrelated edits
done_when: all listed tests pass and each diff touches only the call site and imports
sponsor: ines (inventory)   owner_review: fan-messaging

Every line answers a question the agent would otherwise have answered for itself. The do_not_touch line was added after rung 3 of Figure 5.4 found nine changes with unrelated “improvements” in the first shards; later shards had none.

Scoping a task

Which tasks should go to an agent at all? Osmani’s advice on task selection and Kim and Yegge’s “head chef” framing agree on the essentials, and Google’s migrations show them in practice. Good first tasks are well specified, so the agent need not guess at intent; covered by tests the agent cannot change, so success can be checked by machinery; reversible, so a mistake costs a revert; and reviewable by an owner in minutes, so the human check is real. Kim and Yegge’s delegation framework adds the right scaling rule: the more autonomy a task is given, the more verifiable it must be, so delegation should be matched to risk and to how easily the result can be checked.

The rule rules out more than it admits, and that is its value. The barcode migration qualified on every count. Designing the reissue path did not: its spec was the open question, its tests did not yet exist, and its mistakes would sell a ticket twice. Encore’s engineers used assistants heavily on reissue, for explanation, test drafting and boilerplate, but no agent was ever handed the design. Encore’s tier-1 code, as the Resilience book’s Section 4.5 defines it, is where an agent’s change gets the closest review, and it is also where an agent is least often given a whole task.

Guardrails in the dev loop

Agents cut corners. Kim and Yegge give the problem a chapter, titled “When Your Sous Chef Cuts Corners: Hijacking the Reward Function”, and its catalog, under names such as “baby-counting” and “cardboard muffin”, is sobering to anyone who has watched an agent work: output that looks finished and is not, and litter left behind in the codebase. The behavior follows from the setup. An agent is rewarded for reaching the state its instructions describe, and if it can reach that state more cheaply by changing the checks than by doing the work, some of the time it will.

The controls are each proven in other settings, and their application to coding agents is recent. On the engineer’s own machine, five habits do most of the work. Each agent task runs on its own branch, so its changes can be thrown away whole. The agent works in a checkpointed environment, so any step can be undone. Tests that define done are locked from the agent, by file permissions locally and by code-owner rules in the repository. Diffs stay small, because a reviewer can hold only so much in mind. And the agent never holds production credentials. At Encore, agents obtain short-lived, scoped tokens through the platform’s token exchange. DORA’s 2025 work supports the first two habits directly: it found that “frequent use of rollback features boosts the performance of AI-assisted teams”. The wider questions of agent identity, sandboxing and prompt injection belong to the Architecture book’s Section 15.4 and the Resilience book’s Section 8.5; this section stops at the edge of the developer’s own loop.

The human stays the owner

Every change an agent makes at Encore has a sponsoring engineer: the person who asked for the change, can explain every line of it, and answers for it after it merges. The term comes from the authorship policy in the Resilience book’s Section 4.5, which records the sponsor on every commit. Code ownership is long-standing practice; Software Engineering at Google describes owners files that make a named group responsible for, and the approver of, each part of the codebase. Google’s migrations kept owner review on every change, even when 80% of the characters came from a model. The sponsor answers for the change as a whole, and each owner answers for the part of it that touches their code.

Civil engineering has a long-established version of this arrangement. Drawings for a structure may be prepared by many hands, but an engineer of record reviews them and puts a professional stamp on them, and that engineer answers for the design whoever drew it. The analogy holds for the accountability and breaks for the drafters. A human drafter is trained and learns from correction, and the engineer of record comes to know their habits. An agent’s work varies with each model version and carries no memory of last week’s correction, so the sponsor must review it as fresh work every time.

Encore’s barcode migration shows what the role demands. Six of the seven owning teams approved their shards with small corrections. The seventh, Venue Entry, rejected its shard outright, and that was the most useful moment of the migration. The door-scanning app reads the event-code prefix offline, to reject a ticket for the wrong event without a network round-trip, because gates often lose connectivity when a crowd arrives. The agent’s change replaced the offline read with a call to the identity lookup, which was exactly what the spec asked for and exactly wrong for a scanner. Every test passed, since the tests ran with a network. Venue Entry’s owner saw the problem in minutes, because she knew why the code read the prefix. The fix was a design change, written by hand: the gate’s offline data now carries the ticket_id values valid for each event, alongside the revocation lists the gates already hold. Ines updated the remaining specs to flag any call site on an offline path for human design first.

Recap. The shift from completion to background agents moves the engineer from writing to specifying and verifying. A short, specific spec, tests the agent cannot touch and a well-chosen task do most of the work; a branch per task, checkpoints, small diffs and no production credentials contain the rest. Every agent change has a sponsor who can explain it and owners who review the parts they own, and the owner’s knowledge of why code exists is what no check supplies.

Exercise 5.3. Choose one task you would hand to a background agent. Write its spec in the shape of Ines’s, with the files, the rule, the tests that must stay green, what it must not touch and the definition of done. Then list the owners who would review it, and say what each of them knows about the code that the spec does not.

5.4 Owning What a Machine Wrote

The Resilience book’s Section 4.5 set Encore’s policy for reviewing machine-written change, and Section 4.3 of this book describes code review; both are about the reviewer. This section is about the author, the engineer who asked for the code and will put their name on the change. Most of what the author owes is old, and generated code makes it more important.

Never merge what you cannot explain

Osmani’s advice is to treat generated code as a pull request from “a fast, confident junior”: someone who produces a great deal quickly, sounds sure of all of it, and is wrong in ways that do not announce themselves. It sets the right posture: nobody merges a junior colleague’s change unread, and nobody assumes it is wrong either. The rule that follows is the oldest rule of authorship: never put your name to a change you cannot explain, line by line, to a colleague who asks.

Publishing has lived with a version of this for a long time. Many books are written partly or wholly by ghostwriters, and the author whose name is on the cover still answers for every sentence in them, to readers, to critics and sometimes to courts. The convention works because the named author has read the book and agreed to every claim in it. The analogy breaks in a way that strengthens the rule. A ghostwriter spends months in conversation with the author and absorbs what the author means. A model has no such conversation and keeps no memory of what it intended, so the only person who can supply the meaning of generated code is the engineer who accepted it, and a line that engineer cannot explain has no one to explain it.

At Encore the author’s obligations before review are short enough to keep beside the keyboard, and the Resale team wrote them into its pull-request template:

Before requesting review (author):
- [ ] I can explain every line of this diff without the assistant open
- [ ] I read every assertion in new or changed tests against the spec, not the code
- [ ] No test was weakened, skipped or deleted to make this pass
- [ ] Inputs crossing a trust boundary are validated; queries are parameterized
- [ ] No new dependency, or the new one passed the dependency gate
- [ ] Names and structure follow this module's conventions, not a generic pattern
- [ ] The diff contains only what the task needed

None of the items is new. Each was always the author’s job, and each is now easier to skip, because the code arrived looking finished.

The majority problem

Osmani names a subtler hazard the “majority problem”. A model trained on a vast body of public code tends to produce the most common solution to a problem, and the most common solution is often not the right one for this codebase. Encore met it repeatedly. Asked to make a call to the identity lookup more robust, assistants wrapped it in a retry loop with its own backoff, as most code on the internet does. At Encore it is wrong: the service mesh already retries within a budget of about 10%, and a second retry layer multiplies the first and turns a burst of failures into a storm, the very failure that took Encore down years ago. The fitness function described in the Resilience book’s Section 4.5 caught each instance, but only because someone had once turned that lesson into a check. Encore has many conventions that have never been turned into checks, and those are defended only by the engineers who happen to remember them.

The majority problem is a practitioner’s observation, and it now has one rigorous quantitative counterpart. Hao He, Courtney Miller, Shyam Agarwal, Christian Kästner and Bogdan Vasilescu studied 806 open-source repositories on GitHub that adopted the Cursor AI editor, comparing them by difference-in-differences with 1,380 matched repositories that did not, in a paper presented at the Mining Software Repositories conference in 2026. In the first month after adoption, commits rose by 55.4% and lines added by 281.3%. The only significant gain in velocity fell within the first two months. Static-analysis warnings rose by about 30% and stayed up, and code complexity rose and stayed up as well; the authors find that this accumulation of warnings and complexity is a major factor in the slowdown that followed. The study is observational and its subjects are open-source projects, so it is emerging evidence, but its shape is the one Section 5.1 anticipated.

This is the architectural erosion the Architecture book’s Section 15.4 warned of, accumulating at the rate machines generate code, and Section 2.1 of this book gives the design response, deep modules whose simple interfaces hide their complexity. Each plausible generated change that adds a shallow helper, duplicates a function that already exists or widens an interface a little is individually harmless. Their sum is a codebase that nobody designed, growing faster than the team can understand it.

Security in the author’s chair

Generated code carries known weakness patterns, and the people using it tend to trust it more than they should. Both rest on peer-reviewed studies of 2021–2022 models, and the date belongs beside their numbers. Hammond Pearce and colleagues, in “Asleep at the Keyboard?” (IEEE Symposium on Security and Privacy, 2022), prompted the Copilot of 2021 with 89 scenarios built around common weakness types, and of the 1,689 programs it produced, about 40% were vulnerable. Neil Perry, Megha Srivastava, Deepak Kumar and Dan Boneh (ACM Conference on Computer and Communications Security, 2023) gave 47 participants five security-related programming tasks, 33 of them with an AI assistant and 14 without. Those with the assistant wrote significantly less secure code and were more likely to believe their code was secure, with the largest effects on string encryption and SQL injection.

The second finding is the one that should change the author’s habits. Over-trust belongs to the person using the tool, so better models will not cure it on their own. The author’s defense is the security lines of the checklist above, applied with care where generated code most often goes wrong: input crossing a trust boundary, database queries, cryptography (which belongs in a vetted library), authorization checks that a generic solution omits, and secrets hard-coded for convenience. Assistants also sometimes suggest packages that do not exist, whose names attackers then register; that risk and the dependency gate belong to the Resilience book’s Section 7.4, and fleet-wide patching to its Section 7.3. The author’s part is to notice that a dependency arrived with the code, and to send it through the gate.

Licensing and provenance

The legal position of code produced by models trained on public code is unsettled. The controls, however, are real and cheap. GitHub Copilot offers a filter that hides suggestions of about 150 characters or more that match public code, and a code-referencing mode that shows the matching repositories and their licenses instead. GitHub’s own research has reported such matches to be rare. In September 2023 Microsoft announced its Copilot Copyright Commitment, under which it defends paying commercial customers against copyright claims arising from Copilot output, on condition that they use the guardrails and filters built into the product. The condition is the important part for an engineering organization: the commitment protects only those who keep the filters on.

Encore’s approved assistants therefore run on commercial terms, with the public-code filter enabled by policy. Authorship is recorded at commit time under the Resilience book’s Section 4.5, so the record exists if a question of origin ever arises, and new dependencies pass the curated gate of that book’s Section 7.4, whoever suggested them.

Keeping the codebase legible

The last obligation is the least dramatic and probably the most important over a system’s life. Generated code must be made into the team’s own code: renamed to the module’s vocabulary, restructured to its patterns, stripped of whatever the task did not need. Refactoring, in Martin Fowler’s sense of small behavior-preserving changes, and tidying, in Kent Beck’s, are the tools, and Section 2.2 teaches both. What generated code changes is their urgency. He and colleagues measured the cost of skipping this step in the persistent rise of warnings and complexity, and the cost compounds, because every later change, by a person or an agent, is made in the codebase as the last change left it.

At Encore the habit has a simple form. A change that arrives from an agent is tidied by its sponsor before it is offered for owner review, in a separate commit so that reviewers can see the agent’s work and the human’s adjustments apart. The cost is a few minutes per change, and the return is a codebase whose names keep meaning what the team thinks they mean.

Recap. The author of a generated change owes what any author owes: an explanation of every line, a reading of every assertion, a check of security and dependencies, and the work of making the code fit its module. Generated code arrives looking finished, which is why these obligations are easier to skip. Early velocity bought with unexamined complexity is repaid later, and tidying is how a team avoids the debt.

Exercise 5.4. Take one generated change you merged in the last month and explain every line of it to a colleague, without the assistant open. Mark each line you could not explain, and for each, say what you would have had to know to explain it.

5.5 Adopting It Across 40 Teams

Measuring impact honestly

Encore’s leadership asked its question in the plainest form: do the tools pay? The bill was visible. Assistant licenses cost about €45 an engineer a month, roughly €216,000 a year across 400 engineers, and agent compute, metered through the AI gateway, was growing each quarter. The value was not. The usage dashboard showed suggestions accepted and agent pull requests merged, and a working group of two engineers from the Developer Portal team and an analyst from the Experimentation team was asked to answer properly. Its first conclusion was that the dashboard could not.

Its reasons are the lessons of Section 5.1. Albert Ziegler and colleagues at GitHub found in 2022 that the rate at which shown suggestions are accepted predicts how productive developers feel better than measures of how much generated code survives. That makes acceptance a good measure of perceived productivity, and Section 5.1 showed that perception misleads in both directions. A before-and-after comparison fails too, because the quarter, the roadmap and the teams change at the same time. The design that works is the one Cui and colleagues used at three companies: randomize who gets the tool first, inside ordinary operations, and compare those who have it with those still waiting.

A staggered randomized rollout is a rollout in which groups receive a tool in an order chosen by chance, a period apart, so that the groups still waiting serve as the comparison for the groups already using it. Medicine uses the same design for health-service changes that cannot be withheld forever, where it is called a stepped-wedge trial: hospitals or clinics adopt the change in a random order, and by the end everyone has it. The comparison breaks in one respect. Hospitals are separate places, while teams share code, reviewers and people, so a team with agents changes the work of the teams around it. The working group accepted that contamination, which shrinks the measured differences, and said so.

Encore could not randomize what it already had: inline assistants were universal. So it randomized the next step, access to background agents. The 40 stream-aligned teams were drawn into four cohorts of ten, starting a quarter apart from month 3 of year nine, and the Resale team, formed at month 4, joined the second cohort. The metrics were those Sections 6.5 and 10.5 define, DORA’s delivery metrics including rework rate, with review time per change and the developer-experience survey, and agent usage as context only. Following He and colleagues, the plan added quality lag indicators, warnings and complexity in changed code, watched for two quarters. The working group wrote the plan down before the first cohort began, so that nobody could choose the measures after seeing the results:

# Pre-registration: background-agent rollout, year 9
Question:   Does agent access change delivery outcomes for a team?
Design:     40 teams in 4 cohorts of 10, random order, one quarter apart
Baseline:   the quarter before each cohort starts
Primary:    change failure rate; rework rate; lead time for changes
Secondary:  review time per change; static warnings and complexity in changed code;
            two DevEx survey items on AI-assisted work
Context:    agent PRs merged, acceptance rate (never a target, never a result)
Stop rule:  pause a cohort if its change failure rate exceeds the waiting
            cohorts' by more than 3 points for two consecutive months
Lag check:  quality indicators tracked two quarters past each cohort's start
Reporting:  distributions across teams; no team ranked, no individual measured
Decision:   expand, hold, or withdraw, with the evidence, at each cohort start

The first answer came at month 9, when the first cohort had two quarters of use and the later cohorts served as comparison. Merged changes per engineer were up about 10% in the first cohort, with an interval wide enough to include no change. Review time per change was up by about a fifth, mostly in the teams that reviewed other teams’ code. Rework rate was flat, and change failure rate showed no difference the data could detect. Static warnings in changed code rose in six of the ten teams. It was the shape Section 5.1 predicts, an amplifier whose cost lands on reviewers, and it gave leadership a usable answer: keep going, fix the review load before the third cohort, and look again in two quarters. Figure 5.5 lays the rollout out in time.

Figure 5.5 — Encore’s staggered randomized rollout of background agents. Each cohort waits as comparison, takes its baseline quarter, then has its quality lag watched for two quarters; the first answer came at month 9.

Skills and juniors

The field experiments found the largest gains for less experienced developers, which raises the next question: what happens to how juniors learn? One controlled study has looked. Judy Hanwen Shen and Alex Tamkin, at Anthropic, randomized 52 mostly junior engineers, all of whom had used Python weekly for more than a year, to learn Trio, an asynchronous Python library none of them knew, with or without an AI assistant. On a quiz shortly afterward, the AI group averaged 50% and the hand-coding group 67%, a substantial difference, and the gap was widest on debugging questions. The AI group finished about two minutes faster, a difference that was not statistically significant. Within the AI group, the way people used the assistant mattered: those who asked conceptual questions, or who generated code and then worked to understand it, scored 65% or more, while those who delegated the work scored below 40%. The subgroups were small and that part of the analysis was qualitative, the assistant was a chat sidebar with no agent capabilities, and the authors call the results preliminary. This is emerging evidence, and specific enough to act on.

The two findings fit together once we separate doing from learning. A junior with an assistant produces more, as Cui’s experiments measured, and a junior who delegates to it learns less, as Shen and Tamkin measured. Both can be true of the same person in the same week. The tension is an old one, a version of the deskilling Lisanne Bainbridge described in “Ironies of Automation” (1983), which the Operations book’s Section 3.3 develops. Medicine manages a version of the same tension through supervised residency, in which a resident does real clinical work, under graduated supervision, with the explicit aim of becoming someone who can do it unsupervised. The analogy holds for the structure: the work is real, and so is the teaching. It breaks at the teacher. An attending physician knows what the resident is meant to learn and steers toward it, while an assistant optimizes for finishing the task in front of it, which is exactly the habit that cost Shen and Tamkin’s delegators their learning.

Encore’s onboarding now holds both findings at once. Theo joined the Resale team from a graduate program at month 4 and had to learn the seat ledger, Encore’s only event-sourced service, from which Resale reads ticket ownership. He used an assistant throughout as the study’s high scorers did, asking why the ledger rebuilds its projections as it does, then tracing each answer through the code and its tests. His onboarding buddy asked him once a week to explain a part of the ledger with no tools open. That is an illustration of the practice; the evidence for it is Shen and Tamkin’s. The rule Encore took from it is that new engineers use assistants from their first week, explanation first, and that their onboarding checks understanding directly, by asking for explanations, since merged changes say little about it. Section 9.4 builds this into hiring and onboarding.

The capabilities that make it pay

DORA’s AI Capabilities Model, published in September 2025, names seven capabilities associated with teams that gain from AI adoption without paying for it in other ways. It rests on survey correlations and 78 interviews, so it shows association and leaves cause unproven, and most of its list is the good engineering practice DORA has described for a decade. Figure 5.6 maps each capability to the place in this series that builds it.

Figure 5.6 — DORA’s AI Capabilities Model, mapped to the chapters that build each capability. Most of the list is ordinary engineering capability, which is why the tools amplify what a team already is.

One entry deserves its own sentence. On user-centric focus, DORA warns that without it “AI adoption can have a negative impact on team performance”. A team that builds the wrong thing faster has gained nothing from the speed, which is the Products book’s whole argument.

Policy

DORA’s first capability is a clear and communicated AI stance, and DORA notes that it need not be any particular policy, only a clear one. Engineers unsure whether a tool is allowed either avoid it or use it quietly. Encore’s stance fits on one page. It names the approved tools, all reached through the AI gateway. It sets data rules by class: fan personal data and partner confidential data never appear in prompts to any tool outside the gateway, and sales-velocity data, which Encore keeps from everyone, is never placed in an agent’s context. It states that agents never hold production credentials, that every agent change has a sponsor, and that the review tiers of the Resilience book’s Section 4.5 apply unchanged. And it says, in one sentence, that no engineer will be measured or ranked by how much they use the tools.

Below the stance sit shared conventions. Kim and Yegge argue for shared rules files and team standards for human–AI work, and the argument is sound: a convention written in the repository is read by every agent that touches it. Their supporting case studies, a pilot with about 700 developers at Adidas and a productivity program at Booking.com, are company-reported and should be read that way, and their claims of 10× productivity rest on personal anecdote. Encore keeps a rules file in each repository, owned by its team and reviewed like code, on a common base the platform supplies.

The platform’s part

The platform turns the policy into paths. Encore’s AI gateway, designed in the Architecture book’s Section 15.4, is the single route by which any approved tool reaches a model. Because every call passes through it, the gateway can enforce the data rules, attach the agent’s scoped identity, and meter cost by team. Each team sees its own spend on its scorecard beside its delivery metrics, and leadership sees trends and distributions across teams, with no ranking, as for every other scorecard measure. Usage is shown as context for the outcomes and never as a target, for the reasons Section 10.5 gives. Chapter 6 builds this platform as a product.

Recap. Adoption is measured honestly by randomizing who gets the next tool and comparing outcomes, with the measures written down in advance and the quality lag watched for two quarters. Juniors gain output and can lose learning, so onboarding must check understanding directly. The capabilities that make the tools pay are mostly the old ones, and a clear stance, shared conventions and a platform that meters without ranking turn adoption into practice.

Exercise 5.5. Draft a one-page measurement plan for adopting one AI tool in your organization, in the shape of Encore’s pre-registration: the question, the comparison group and how it is chosen, the baseline, three outcome metrics, the stopping rule, and the result that would make you withdraw the tool.

Kata: The Promise to the Board

Your brief: “Fieldnote.” Fieldnote sells farm-management software as a service to about 30,000 farms and 2,000 agronomists in five EU countries: field records, crop plans and the reports farms file to claim agricultural subsidies. It has 150 engineers in 18 teams. Its subsidy-reporting module dates from the company’s first year, is understood by two engineers, and changes every season with the reporting rules. Spring is the busy season, when farms enter a year’s records in six weeks and a release freeze holds. Engineers have used inline assistants for two years without anyone measuring them. In January the chief executive bought background-agent licenses for every engineer, at €70 a month each, and announced to the board that engineering would deliver “2× by Q3”. Change failure rate has been 9% for a year. Review time per change has doubled since the autumn, and two senior engineers say they now spend most of their week reviewing. Six graduates join in March.

Deliverables:

1. Adoption design — A staggered, randomized two-quarter rollout across the 18 teams, with the baseline, and how you will handle the spring freeze and teams that share code.
2. Pre-registration — The measurement plan on one page, written before the first cohort starts: primary and secondary metrics, the role of usage data, the quality lag indicators and the stopping rule.
3. Delegation map — Fieldnote’s kinds of work sorted by verifiability and reversibility: what goes to an agent, what gets assistance only, and where the subsidy-reporting module falls.
4. Author and agent rules — The spec template for agent tasks, the guardrails on engineers’ machines, and the author’s checklist, sized for Fieldnote’s own teams.
5. Graduate onboarding — How the six graduates will use assistants from week one and still build the understanding they will need as reviewers.
6. The flat-result memo — The one page you will send the chief executive if, at the end of Q3, the numbers show no clear change in delivery, and what you will recommend.

Rubric:

Where you now stand

You can now read each study on AI assistance for what it measured, and explain why careful studies disagree without any of them being wrong. You can say where assistance pays and where it does not yet, hand a task to an agent with a spec that leaves little to guess, and own what it produces as its author. And you can tell an organization whether its tools pay, by randomizing the next step and writing down the measures before the results arrive. One conclusion kept returning: the tools amplify whatever engineering system surrounds them, and the parts of that system that 40 teams share are built by a platform. Chapter 6 turns to that platform, and to building it as a product whose customers are the engineers themselves.

References

• Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, Tobias Salz — The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers. Management Science, published online 27 February 2026 (working paper 2024).
• Joel Becker, Nate Rush, Elizabeth Barnes, David Rein — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. METR, 2025 (arXiv:2507.09089); with METR’s uplift update, February 2026, and self-reported impact survey, May 2026 — free online.
• DORA — State of AI-assisted Software Development 2025, with the DORA AI Capabilities Model. Google Cloud, 2025 — free online.
• DORA — Accelerate State of DevOps Report 2024. Google Cloud, 2024 — free online.
• Sida Peng, Eirini Kalliamvakou, Peter Cihon, Mert Demirer — The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv:2302.06590, 2023 — free online.
• Elise Paradis et al. — How Much Does AI Impact Development Speed? An Enterprise-Based Randomized Controlled Trial. ICSE-SEIP 2025, pp. 618–629 — free online.
• Stoyan Nikolov et al. — How is Google Using AI for Internal Code Migrations?. ICSE-SEIP 2025, pp. 481–492; with Celal Ziftci et al. — Migrating Code At Scale With LLMs At Google. FSE 2025 Companion, pp. 162–173 — free online.
• Hao He, Courtney Miller, Shyam Agarwal, Christian Kästner, Bogdan Vasilescu — Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects. MSR 2026 — free online.
• Judy Hanwen Shen, Alex Tamkin — How AI Impacts Skill Formation. arXiv:2601.20245, 2026; Anthropic research post, January 2026 — free online.
• Grace Murray Hopper — The Education of a Computer. Proceedings of the ACM National Meeting, Pittsburgh, 1952, pp. 243–249; reprinted in Annals of the History of Computing 9(3/4), 1988 — free online.
• John Backus — Programming in America in the 1950s — Some Personal Impressions. In Metropolis, Howlett & Rota (eds.), A History of Computing in the Twentieth Century, Academic Press, 1980, pp. 125–135 — free online.
• Neil Perry, Megha Srivastava, Deepak Kumar, Dan Boneh — Do Users Write More Insecure Code with AI Assistants?. ACM CCS 2023, pp. 2785–2799; with Hammond Pearce et al. — Asleep at the Keyboard?. IEEE S&P 2022, pp. 754–768 — free online.
• Albert Ziegler et al. — Productivity Assessment of Neural Code Completion. MAPS ‘22, pp. 21–29 — free online.
• Addy Osmani — Beyond Vibe Coding. O’Reilly, 2025.
• Gene Kim, Steve Yegge — Vibe Coding. IT Revolution, 2025.
• Lisanne Bainbridge — Ironies of Automation. Automatica 19(6): 775–779, 1983.