Software Architecture in the AI & Cloud Era

← the whole book on one page

Chapter 9: Data Architecture — From Distributed Data to Data Mesh

Code is what a system does; data is what it knows. Split the doing, and the knowing demands a constitution.

Decomposition promises teams their freedom, and data is where that promise gets audited. In Chapter 5 we gave every Encore service a private database, and the payoff was real: the Catalog team can change its schema on a Tuesday afternoon without asking anyone. But three old comforts went quietly out the door with that decision. The first is the cross-domain JOIN, the single query that could stitch an order to its concert and its fan because all three lived in one database. The second is the transaction that spans everything, the guarantee that a set of changes in different parts of the business either all happened or none did. The third is the analyst’s “one big database where everything lives,” the place you could point a reporting tool at and simply ask. Encore’s teams now ship independently, and that is the gain. The cost shows up in places like the finance report, which used to be one query against one database and has become a distributed-systems problem with a deadline attached: the numbers it needs now live in several services, owned by several teams, each changing at its own pace.

It helps to remember what we gave up, and why it once seemed so natural. In 1970 Edgar Codd published the paper that introduced the relational model, and its title spoke of “large shared data banks.” For the decades that followed, the shared database sat at the quiet center of enterprise systems, with every application reading and writing the same tables. That was a genuine triumph of abstraction: programs could ask for data by what it meant rather than by where it was stored, and every program saw the same answer. Yet, as Chapter 5 found, sharing tables also created a coupling so deep that no team could change its schema without negotiating with every other team that touched it. Renaming a column became a diplomatic event. So we did not drift away from the shared database by accident; we traded one shared truth for many owned ones, deliberately, to win independence. The honest thing now is to count what that trade costs and pay it with our eyes open, rather than discover the bill one broken report at a time.

The chapter takes the cost in two halves. The operational half concerns the data that running services use to do their work, such as orders, seats and payments. Here the questions are who owns which data, how a service can answer a question that spans several owners without coupling them back together, and what takes the place of the transactions we lost. The analytical half concerns the data that people use to understand the business after the fact, such as sales trends, on-sale performance and the finance report itself. Here the questions are why the classic central warehouse pipeline, where one data team pulls everything into one place, breaks down as the organization grows, and what the data mesh does about it. The two halves use different tools, but I have come to believe they share a root. In each, the hard part is deciding who is responsible for a piece of data and its meaning; once that is settled, choosing the technology is comparatively easy. Each is an ownership problem before it is a technology problem.

9.1 Data Ownership in Distributed Systems

One writer per table

We begin with the one rule on which everything else in this chapter rests. In Chapter 3 an aggregate guarded its own invariants, and nothing outside it could reach in and change its state. Carry that discipline from a single codebase out to the whole fleet of services and you arrive at the baseline rule: every table has exactly one owning service, and only the owner writes to it. Reads are a different matter. Other services may need to see the data, and Section 9.2 shows several ways they can negotiate that. Writes are not negotiable. The reason is that a table with two writers is really two services sharing a contract that nobody has written down. Both depend on what each column means, which values are legal and which rows may exist, so neither can change any of that without the other’s consent. In practice the two services must then be tested together and deployed together, which quietly undoes the independence that splitting them was meant to buy.

A land registry is a useful picture of what “owner” means here. Banks, surveyors and tax offices all keep copies of what the registry says about a plot of land, and they consult those copies every day. Yet only the registry can record a transfer of ownership. A bank’s copy may be out of date, and nobody is confused about which version wins, because the registry’s authority comes from being the one place where a change becomes true. An owning service plays the same role for its table: other services may hold copies, but a seat is held, or a sale is recorded, only when the owner says so. The analogy breaks on tempo. A registry changes slowly and with ceremony, through signatures, stamps and waiting periods, while a database table may change many times a second. The principle survives the change of speed, though: there is still exactly one place where each change becomes true.

Not every piece of data is shaped the same way, and Software Architecture: The Hard Parts, by Neal Ford, Mark Richards, Pramod Sadalage and Zhamak Dehghani (2021), names three ownership shapes. The first, single ownership, is the easy one: one service owns a concept outright, as Catalog owns events, and nobody else has a claim to it. Most data falls here. The second, common ownership, covers data that every service needs to contribute to or read, such as audit records. Here the temptation is to create one shared table that everyone writes to, and that is exactly the mistake to avoid, because a table that belongs to everyone in effect belongs to no one: nobody can change its schema, and nobody answers for its quality. There are two better choices. We can give the data one owning service and have everyone else call its API, or we can have each service publish events that the owning service collects into a store it alone controls. Either way, audit records end up with a single owner, even though every service contributes to them.

The third shape, joint ownership, is the hard case: two services each have a legitimate claim on the same concept. Consider who owns a sale at Encore. Orders runs the purchase workflow, from the moment a fan’s hold becomes an order to the moment payment succeeds. Finance runs the ledger service, which was split out of Chapter 5’s Orders & Payments as the accounting work grew, and Finance must record that same sale for revenue, refunds and tax. Both teams have good reasons to want to write the sale. The usual resolution is a split by meaning. We notice that the word “sale” refers to two different things. To Orders, the sale as process is a workflow with states such as pending, paid and canceled. To Finance, the sale as accounting fact is an immutable entry with amounts and dates. Orders owns the first, and Finance owns the second, which it populates by listening for SaleCompleted events. One word now has two models and two owners, each writing only its own table. This is Chapter 3’s bounded-context lesson applied to storage: the same term may mean different things in different contexts, and each context deserves its own model. So when two teams quarrel over a noun, it is worth asking whether the noun means two things. It usually does.

Once ownership is settled, each owner is free to choose the kind of database that suits its data. The choice should follow the access pattern, meaning which questions the service asks of its data and how often, rather than whatever technology happens to be fashionable. The table below summarizes what each family of stores is good for and what it makes painful:

Encore’s choices follow from this table. Orders and the Finance ledger run on relational databases, because that is where the invariants live: an order must not be paid twice, and the books must balance, and a relational database enforces such rules inside a transaction. Seat Inventory keeps its holds in a key-value store. A hold is a fan’s temporary claim on a seat with a five-minute time-to-live (TTL), after which it expires on its own; holds are created and looked up by key in huge numbers during an on-sale, and are not queried in any other way, which is exactly what a key-value store is built for. Encore does not adopt a third engine until a real query needs one. The common failure in this area is not choosing the wrong database but running more kinds of database than anyone can operate well. Each one needs backups, upgrades, monitoring and someone on call who understands it, so every database a team adopts is a pet it must feed for years.

Consistency without distributed transactions

Owning data separately raises a hard question: if a business process touches several owners, how do their data stay in agreement? Chapter 5 answered part of it with sagas, which run a workflow that spans owners as a series of local steps, each with a compensating step to undo it if a later one fails. Beneath sagas sit three consistency patterns, which we can list in rising order of decoupling, meaning how little each owner needs to know about the others or wait for them. Background sync is batch reconciliation: at intervals, a job compares the owners’ data and corrects any differences. Encore’s nightly finance close works this way, matching the day’s orders against the ledger. It is unfashionable, and for the right flow it is embarrassingly effective, because it is simple and cheap, and it can catch errors that slipped past every other mechanism. Orchestrated consistency is the saga’s explicit process, in which a coordinator tells each owner what to do next and tracks the outcome. Event-based consistency is the most decoupled: each owner publishes facts about what has happened, and other owners react to them in their own time. This is Encore’s default way of propagating changes. The events go out through Chapter 6’s transactional outbox, in which the service writes the state change and the event to its own database in a single local transaction, so that a change can never be committed without its event, or an event recorded without its change; a relay then publishes the recorded events.

What Encore refuses is two-phase commit (2PC) across services. To see why, it helps to remember what the transaction is and where it came from. The transaction, as Jim Gray and his colleagues developed it in the 1970s, is one of our field’s great abstractions. It lets a programmer pretend that a set of changes happens all at once or not at all, so the program never has to reason about a half-finished state. Inside one database that pretense is cheap, because a single engine controls all the data, the locks and the log. Two-phase commit stretches the pretense across several databases. A coordinator first asks every participant to prepare, which means each one locks the affected rows and promises it can commit. Only when every participant has said yes does the coordinator tell them all to commit. The catch is in the waiting. The whole transaction can move only as fast as the slowest participant, and it succeeds only if every participant and the coordinator are up, so each service’s availability becomes tied to all the others’. Worse, if the coordinator fails after participants have prepared, they cannot decide on their own whether to commit or abort, so they sit blocked with their locks held until it returns. For Encore, the price is too high. Its services deploy independently, so at any moment some participant may be in the middle of a restart, and Chapter 4 showed what network latencies between services cost. Locking seats and orders across that network during an on-sale is the wrong trade. Pat Helland made the broader case in “Life beyond Distributed Transactions” (2007): systems that must scale almost without limit end up built from entities that are fully transactional inside, and that talk to one another only through messages. Encore follows that shape. Atomic transactions stay inside one owner’s database, and messages carry everything that crosses from one owner to another.

Eventual consistency is a business conversation, not an apology. Giving up distributed transactions means that some data will be briefly out of date, and that is acceptable only if the business has agreed how out of date. A sentence such as “The dashboard trails reality by up to a minute” is a requirement statement: the business can weigh what that staleness costs against what it would cost to remove. So each flow that crosses owners gets a chosen point on Chapter 4’s consistency spectrum, signed off by the stakeholder who lives with the result, instead of guessed at by the engineer who happens to write the code.

Recap. Every table has one writer, and that writer is the place where its changes become true, as the registry is for land. Data that every service needs still gets a single owner, and data that two services both claim is split by meaning into two models with two owners, joined by events. Sagas, events and humble batch reconciliation keep owners in agreement in place of cross-service 2PC, whose locks and blocking tie every service to the slowest and least available of its peers. How stale each flow may be is agreed with the business, flow by flow, instead of assumed by whoever writes the code.

Exercise 9.1. Find a table, or a business concept, in a system you know that has two writers. Split it by meaning: write down the two models it really contains, the service or team that should own each one, and the event that would carry changes from one to the other.

9.2 Distributed Data Access

Querying across services

Section 9.1 gave every table one writer. That settles who may change data, but it leaves a harder question open: how do we ask about data that lives with several owners? Consider a fan opening the account page on Encore’s site. The page shows the fan’s orders, their tickets and the status of any refund, and each of those facts belongs to a different service. In a single shared database this would be one query with a couple of JOINs. Now there are three owners, one screen, and a question that none of the three services owns. Someone has to assemble the answer, and there are four respectable ways to do it. Each has a bill attached, so it is worth reading the table below by its last column as much as its first.

Figure 9.1 — The access menu: composition for screens, replicated read models for query-heavy needs, CDC for owners that don’t publish events. The missing row is reading another service’s tables directly. It is missing because today’s harmless read becomes tomorrow’s join, and once other teams depend on the owner’s tables, the owner can no longer change its schema.

The four rows look different, but the last three share one idea, and it is the idea worth carrying away. The query moves to a copy shaped for it, and contracts feed the copy. Instead of reaching into three owners’ databases at the moment the fan clicks, we keep a separate store arranged around the question being asked, and we fill it only through channels the owners have promised to keep stable: their published events, or a CDC stream whose schema they publish and version as deliberately as an API. That second condition is what keeps the copy from turning into a back door. CDC without a published schema is simply reading another service’s tables with extra steps, because every rename in the owner’s database ripples straight into your store.

A library catalog is a good picture of such a copy. The catalog is not the books; it is a set of facts about the books, arranged for the questions readers actually ask (by author, by title, by subject), and kept current as books arrive and leave. It may trail the shelves a little: a book returned an hour ago might still show as on loan. Readers accept that small lag gladly, because the alternative is walking the stacks by hand for every question. The analogy also shows where the work goes. Someone has to keep the catalog up to date, and that job belongs to whoever wants the catalog, not to the books.

Encore chose the replicated read model for its account page. The Fan-BFF subscribes to the event streams of the three owning services and maintains its own projection, a table holding exactly the rows and columns the page needs. When a fan opens the page, it loads with one local query instead of three remote calls. Because the Fan-BFF reads events rather than tables, the owners remain free to change their internal schemas as they like, provided the events keep their contract. The price is freshness: a refund approved a moment ago may take a few seconds to show. Product looked at that trade and signed off on “seconds” of staleness as the requirement, in the spirit of Section 9.1’s business conversation. The price also includes work. The Fan-BFF team now runs the projection, rebuilding it when its shape changes and watching its lag. That cost falls on the team that wanted the query, which is exactly where it belongs; the owners are not asked to bend their models for one screen.

There is one sanctioned exception to “only the owner touches its tables.” The Hard Parts names it a data domain: a schema that a few closely related services all read and write. This is joint ownership chosen on purpose, with the coupling admitted rather than hidden. The consequence follows directly. If several services share a schema, a change to that schema must land in all of them at once, so in practice they deploy together. That is tolerable only when the services already change in step for business reasons, so keep a data domain to a small cluster that would move together anyway, and never let it grow into the shared database by another name.

Copies and contracts let readers stay out of an owner’s tables, but the owner still has to change those tables from time to time, and it must do so while traffic keeps flowing. Picture engineers replacing a bridge on a road that cannot close. They do not demolish the old span first. They build the new span beside it, move traffic across in stages, and tear down the old one only after the new one has carried real loads and proved itself. At every step until the demolition, they could send traffic back the way it came.

Zero-downtime database migration follows the same plan. It applies Chapter 5’s expand–contract discipline to the owner’s own storage, in four phases, and every phase except the last can be undone:

1. Expand. Add the new shape, such as new columns or a new table, alongside the old one. Old code does not know the new shape exists and simply ignores it, so nothing breaks.
2. Dual-write. Deploy new code that writes every change to both the old shape and the new one. Old readers carry on reading the old shape, which is still complete and correct.
3. Backfill. Copy historical rows into the new shape in throttled batches, so the migration never starves live traffic, and verify each batch against the old data before moving on. When it finishes, the new shape holds everything the old one does.
4. Contract. Switch reads to the new shape, then wait and watch through a full business cycle (long enough for every kind of business activity that touches the data, such as on-sales and refunds, to run against the new shape). Only then drop the old shape.

The last step is the one-way door. Until the old shape is dropped, a problem in the new one costs nothing worse than switching reads back; once the old data is gone, there is nothing to switch back to. That is why the watching period comes before the drop, not after it.

Encore used exactly this sequence when regulation required splitting price into base_price and fees (Chapter 7’s regulatory change). The new columns appeared first; the code then wrote both the old total and the two new fields; past orders were backfilled with their split in batches; and only after reads had run on the new fields through a full cycle was price removed. The whole migration took three weeks and happened without incident. That is the mark of a good migration: by the time it is finished, nobody remembers it happened.

Recap. A question that spans owners is answered from a copy shaped for that question and fed through contracts. Use API composition for shallow joins across two or three owners, replicated read models (projections) for deep or heavy queries, and CDC for owners that don’t publish events, provided the change stream has a published schema. Reading another service’s tables directly couples you to its schema and freezes it. Owners change their own schemas by expand–contract in four phases (expand, dual-write, backfill, contract), and every step is reversible until the old shape is dropped.

Exercise 9.2. Take your product’s heaviest cross-domain screen, the one that needs data from the most owners. Which row of Figure 9.1 serves it today (or is it quietly reading another service’s tables)? Which row should serve it, given how fresh the data must be and how many owners are involved? Finally, name the contract that would feed the copy: which events or which published change stream, and from which owners.

9.3 Analytical Architectures

Why central data pipelines break at scale

So far this chapter has dealt with operational data: the records services read and write while fans buy tickets. Analytics asks a different kind of question. It wants to know how sales for a tour compared with last year’s, which venues sell out fastest, or how much revenue refunds cost the business last quarter. These questions span every domain and look back across months of history, and the field has built three generations of storage to answer them.

The warehouse, shaped in the 1990s by Bill Inmon and, with dimensional modeling, by Ralph Kimball, holds data that has been cleaned and structured before it arrives. It is governed, and it is queried in SQL. The lake came next and took the opposite bet: store everything raw and cheaply, in whatever shape it arrives, and apply a schema only when someone reads it (“schema on read”). The lakehouse, today’s default, brings the warehouse’s discipline (schemas, transactions, governance) to the lake’s cheap storage through open table formats. In practice that means Apache Iceberg or Delta Lake tables listed in a shared catalog, which most warehouse query engines can read in place without copying the data. Any of the three could hold Encore’s data perfectly well. That is a useful clue, because it tells us where not to look when analytics starts to fail.

What breaks is not the storage but the operating model wrapped around it, meaning who moves the data, who understands it, and who answers the questions. In the classic arrangement a central data team extracts data from every domain’s databases through pipelines. The central team builds and maintains those pipelines, but it does not control their sources. Picture what that means at Encore. The Orders team renames a column in its own database, a perfectly reasonable change to a table it owns. That night the pipeline that copies orders into the warehouse fails, or worse, quietly loads nulls. The Orders team never sees the failure, because the pipeline isn’t theirs; the central team finds out the next morning, when finance asks why yesterday’s revenue reads zero. Meanwhile every new question, such as “how do refunds vary by venue?”, joins a queue behind a team that knows the tables intimately but knows little about how ticketing actually works. The central team has become what Chapter 5 called the coordination machine: a monolith made of extract, transform, load (ETL) jobs, with a single backlog through which every analytical question in the company must pass.

We have seen this shape before, more than once. The data-processing departments of the mainframe era kept a queue of report requests, and a manager who wanted a new report could wait weeks for a programmer to write it. The integration teams of the service-oriented architecture (SOA) years, whose story Chapter 2 told, kept a queue of changes to the enterprise service bus. Each time, a central group owned the plumbing that ran between domains it did not understand, and each time the queue grew faster than the group could. The technology changed from generation to generation, but the bottleneck stayed in the same place: wherever the plumbing belonged to one group and the meaning of the data belonged to another.

Before we reach for anything as ambitious as a mesh, two simpler repairs address most of the damage. The first is owner-published feeds. Instead of the central team scraping each domain’s tables, analytics consumes the events that owners already publish on Chapter 6’s backbone. Orders publishes SaleCompleted because it has decided what a completed sale means, and it treats that event as a promise to its consumers; its internal tables can change shape without breaking anyone downstream. These feeds need not be real-time. Stream them where freshness genuinely matters, such as a live dashboard during a big on-sale, and let much of the rest stay in batch. A monthly finance report gains nothing from data that is seconds old, and there is no virtue in paying for freshness that nobody uses.

The second repair is data contracts: a published schema for every analytical feed, together with a guarantee about which kinds of change are allowed (adding an optional field, say) and which are not (removing or renaming one). A contract changes when and where a breakage is discovered. Without one, “your change broke finance’s dashboard” is a Friday-afternoon surprise, found by the wrong team after the damage is done. With one, the producer’s continuous integration (CI) build checks each schema change against the contract, so the breaking change fails on the producer’s own build, while the author can still reconsider it, add the new field alongside the old one instead of renaming, or warn consumers first. For CI to check a contract, the contract must be machine-readable rather than a page on a wiki; the Open Data Contract Standard (ODCS, from the Linux Foundation’s Bitol project) is one such format. The deeper effect is that contracts move the work of noticing a breakage to the team that knows the schema best, its owner. Section 9.4 follows that principle to its conclusion.

Recap. The lakehouse is the sensible storage default, and storage is not where analytics fails. The failure is organizational: a central team runs pipelines coupled to schemas it does not own, and every question waits in that one team’s backlog. Two repairs come first. Owner-published feeds replace table scraping, streamed only where freshness justifies the cost, and data contracts checked in the producer’s CI catch breaking changes before they ship.

Exercise 9.3. Pick one dashboard at your work and trace it back to its data sources. Write down every source table or feed it depends on, and count how many independent schema changes could break it. Then answer: when one of them breaks it, who finds out first, the person who made the change or the dashboard’s owner?

9.4 Data Mesh

Domain ownership for analytical data

Section 9.3 ended with a principle: move breakage, and with it responsibility, to the team that knows the schema. The data mesh, which Zhamak Dehghani described in 2019, follows that principle to its conclusion. What strikes me about her four principles is how little in them is new to a reader of this book. Each one takes an idea we have already met in the operational world and carries it into analytics. Let us take them one at a time and see where each came from.

Domain ownership says that each domain team owns its analytical data as well as its operational data. Orders already owns the services and tables that record ticket sales, so Orders also owns ticket-sales, the analytical view of those sales that the rest of the company reads. No central team reaches into Orders’ database to build that view on Orders’ behalf. This is Conway’s law (Chapter 3) carried into analytics: the organization is already split into domain teams, so the analytical data is split along the same lines, so that the people who understand a sale are the people who publish the facts about it.

Data as a product says that a published dataset deserves the same care as a published service. Section 6.5 already gave Encore’s event streams an owner, a contract and service-level objectives (SLOs). A data product keeps all three and adds two more things: it can be discovered, and it is documented. An analyst who needs sales figures should be able to search a catalog, find ticket-sales, read what each field means and how fresh the data is promised to be, and start using it without asking around. Those additions matter because a product nobody can find, or nobody can understand once found, is not yet a product. It is just a table with good intentions.

Self-service platform says that publishing a data product should be as easy as deploying a service. If every domain had to build its own storage, pipelines, catalog entries and access controls, ownership would become a burden that each team carried separately, and most teams would quietly hand it back to a central group. So a shared platform supplies the lakehouse plumbing once, and each domain uses it. This is Chapter 15’s thesis, that a platform turns repeated hard work into a paved default, arriving a few chapters early.

Federated computational governance says that a few rules must hold for every data product, and that the platform, not a committee, enforces them. The rules cover privacy, identifiers and quality metadata. The word “computational” is the important one: the rules are written as code in the platform, checked automatically every time a product is published, instead of living in a governance committee’s memos that teams may or may not read. A product that exposes an untagged personal field, for instance, fails its publish step much as a service fails a build. This is Chapter 5’s paved road, laid this time for data.

“Federated” is the right word for that last principle, and the chapter’s epigraph leans on it. In a political federation, the member states govern themselves on most matters, but they accept a small body of law that binds them all, such as a common currency or a shared bill of rights. The mesh asks the same of domains. Each governs its own data, choosing its own pipelines, storage layout and release pace, under a short constitution that covers only what must be common. At Encore that could mean every data product identifies a fan in the same way, so finance can join ticket-sales with refunds without guesswork; every field holding personal data is tagged, so access and erasure rules can find it; and every product declares how fresh its data is. The platform enforces those clauses. The analogy also tells us when a federation fails. A constitution works only when it is short enough to obey and enforced by something sturdier than goodwill. Make it long, and domains stop reading it. Leave it to goodwill, and the first deadline erodes it.

The unit that all four principles produce is the data product, and it packages much more than a table. Figure 9.2 shows the ticket-sales product owned by the Orders team. Operational events flow in from the Chapter 6 backbone. Transformation code, which Orders writes and owns, turns them into two output ports: a stream of sales events and a table of daily aggregates. A contract with SLOs, lineage and documentation describes both ports. Consumers such as dashboards, data science and revenue reporting subscribe to those ports, and never to the tables behind them.

Figure 9.2 — Anatomy of a data product: the ticket-sales product, owned by the Orders team, keeps its transformation code, its two output ports (a sales-event stream and a daily-aggregate table) and its contract together under one owner. Consumers subscribe to the ports, never to someone’s tables, and the data product replaces “the warehouse” as the unit of analytical architecture.

The data product is information hiding carried into analytics. Parnas’s criterion, which we met in Chapter 1, asked each module to hide a design decision that is likely to change, so that the change stays inside the module instead of rippling through its callers. The data product applies that same criterion to data. It hides its storage, its pipelines and its internal schema behind its ports and its contract, and those hidden things are exactly the decisions most likely to change. Suppose Orders decides to rebuild the daily-aggregate table from the event stream instead of from a nightly batch, or to rename and restructure the columns it uses internally. Neither change needs permission from the dashboards or the forecasting model, because neither of them can see the inside. They see only the ports, and the ports still deliver the same fields at the promised freshness. Only the promise must hold still. Everything behind it is free to move.

The mesh was designed to solve a large organization’s problem, and its machinery is sized for one: many domains, a central data team that has become a bottleneck, and too many pipelines for any single group to understand. Encore, with nine teams, is not there. So it adopts the principles and leaves most of the machinery for later. Concretely, Encore writes contracts for its analytical feeds, treats those feeds as products with owners, and builds two or three real data products, ticket-sales among them. It does not yet build the full federated apparatus, with its formal governance structures and a large catalog of products. That waits, just as microservices did in Chapter 5, until two things are true: the coordination pain has been measured instead of merely predicted, and the prerequisites are in place, above all a self-service platform of the kind Section 9.5 describes.

The industry has since tested the full model, and the results support that caution. Gartner’s 2022 Hype Cycle for Data Management rated data mesh “obsolete before plateau,” Gartner’s label for an idea it expects to fade before it ever reaches mainstream, productive use. Many adopters kept the principles while scaling back the full operating model. I read that as a verdict on the operating model more than on the ideas. On that reading, domain ownership, contracts and product thinking keep their value; what disappoints is the heavy organizational apparatus, imposed before an organization needs it, because machinery built for a very large organization feels like bureaucracy to nine teams.

Recap. Data mesh applies the book’s decomposition logic to analytics through four principles: domain ownership, data treated as a product, a self-service platform, and governance enforced as code by that platform. Its unit is the data product, which keeps code, ports and contract together under one owner and hides its insides as a well-designed module does, so the owner can change them freely while the promise to consumers holds. Adopt the principles at any scale, and build the full machinery only when coordination pain has been measured.

Exercise 9.4. Pick the data people at your company ask about most often, and sketch Figure 9.2 for it. Name its owning team, its output ports (for example a stream and a table), the main promises in its contract (fields, freshness, quality), and the first three consumers who would subscribe. Then name the meeting, committee or approval process that this data product would make unnecessary, and explain why.

9.5 The Self-Service Data Platform

The data mesh stands or falls on its platform. Section 9.4 asked every domain team to own and publish its analytical data, and that request is only reasonable if publishing is easy. If each domain had to build its own storage, pipelines, catalog entries and privacy controls by hand, then Orders, Finance and every other team would spend their time on plumbing instead of on their business, and most of them would quietly decline. The platform’s job is to take that plumbing off their hands. Concretely, a domain team should be able to do four things without filing a ticket with anyone:

• Declare a product. One request produces the scaffold for the product’s code, an entry in the registry of data products and a listing in the catalog, so other teams can find it from day one.
• Publish ports. From a single definition, the platform produces both a stream (for consumers who want each change as it happens) and a table (for consumers who want to query history). The team describes its data once and does not maintain two delivery mechanisms that can drift apart.
• Inherit governance. The platform detects personally identifiable information (PII), such as a fan’s name, email address or payment details, and applies the retention and access policies that go with it. The domain team does not have to remember those rules; it gets them by publishing through the platform.
• Observe. Dashboards for freshness (how recently the data was updated), quality (whether it meets its contract) and lineage (where it came from and who consumes it) come for free, without the team instrumenting anything.

Notice what these four have in common: each turns a task that a careful team might do into one that every team does by default.

Governance is the hardest of the four, because it must reach every copy of the data, not just the original. Section 9.2 makes copies on purpose, such as the Fan-BFF’s replicated read model of a fan’s account, and the analytical side adds more: lakehouse tables, and the ports each data product publishes. Now suppose a fan asks Encore to erase their personal data. Deleting the record in the owning service is not enough, because the fan’s name and email also sit in a projection, in a stream’s retained history and in a table someone built last quarter. The erasure has to propagate to every port, projection and table. The platform can do this in one of two ways. It can issue deletes through the table format, since open table formats such as Iceberg and Delta Lake support row-level deletes. Or it can use Chapter 6’s crypto-shredding, where each fan’s personal data is encrypted under that fan’s own key, so destroying the key makes every copy unreadable at once, wherever it lives. Either way, the point is that the platform does it, not each consumer from memory. A privacy rule that binds the original record but not its copies is hardly a rule at all, because the copies are where the data spreads.

This is Chapter 15’s golden-path pattern applied to data. A golden path is a supported, well-paved route through the platform: if a team follows it, the correct behavior (here, catalog listing, PII handling, erasure, monitoring) comes built in, and doing the right thing becomes the easiest thing. The payoff shows up in time. Without the platform, every new data product repeats the same plumbing work. With it, the plumbing is already there, so by the time a domain builds its third data product, the work costs a sprint instead of a quarter.

AI depends on this platform too, and heavily. Chapter 12’s feature pipelines (which prepare the inputs a model learns from) and retrieval corpora (the documents a model looks things up in) are, underneath, just consumers of data products. Encore’s demand-forecasting model, for example, subscribes to the ticket-sales product, the same way any analytics dashboard would. The support assistant uses retrieval-augmented generation (RAG): before answering a fan’s question, it retrieves relevant past cases and passes them to the language model as context. Its corpus is built from the support-cases product, so it inherits that product’s contract, its lineage and its access rules. If a support case is restricted, the assistant cannot surface it to the wrong person; if the case format changes, the contract says so in advance.

Teams that skip this discipline do not escape it; they meet it later, at a higher price. The usual form is a model trained on a one-off data extract that nobody owns: the data goes stale, no one notices because no one is responsible for it, and no access rules came along with the copy, so the model may have learned from data it should never have seen. Untangling that after the model is in production costs far more than publishing the data properly in the first place. Trust runs down a chain: a model is no more trustworthy than its data, and the data is no more trustworthy than the ownership behind it.

Recap. The self-service platform makes correct data handling the cheapest option for every domain: declare a product, publish its ports, inherit governance (including erasure that reaches every copy) and observe it, all without tickets. AI systems are consumers of data products, so their quality is bounded by the quality and ownership of those products.

Exercise 9.5. At your company, list every manual step between “a team has valuable data” and “another team uses it safely”: requests, approvals, pipeline work, access grants, privacy reviews. Mark each step as either platform backlog (something the platform should automate) or a cost you will keep paying on every new data product.

Kata: The Freight Ledger

Your brief: “Cargo.” A freight forwarder, 14 years old, whose data still lives where it was born: one Oracle schema of 1,100 tables. A 22-person central data team keeps the analytics running through 400 nightly ETL jobs, and it is drowning, for the reason Section 9.3 described: each job depends on tables the team does not own and cannot stop from changing. The business has four domains: bookings, customs, fleet and billing. The fresh pain comes from three directions at once. Billing disputes need booking, customs and GPS history joined within minutes, but today the joined picture arrives the next morning, after the nightly jobs have run. A new machine learning (ML) team wants two years of clean movement history to train on. Customs data is restricted per country, so who may see a customs record depends on the country it concerns (compare Chapter 7’s Atlas, whose shippers see customs status only where they are the importer of record). And the CFO has banned “another warehouse rewrite”; the word another tells you the company has tried that road before.

The brief is built so that the obvious answer fails. Buying a new storage engine is exactly the rewrite the CFO has banned, and, as Section 9.3 argued, storage was never where the failure lay. What remains is the work of this chapter: decide who owns which data, serve the joined-up questions from Figure 9.1’s menu, feed analytics through contracts, package the most-wanted data as products, and move there gradually enough that the business keeps getting value along the way.

Deliverables:

1. Ownership map: the four domains’ data, each with a named owning team. Find three places where two domains both have a legitimate claim on the same data (a shipment, say, is both something booked and something cleared through customs), and resolve each by splitting it by meaning, as Section 9.1 split Encore’s sale into a process owned by Orders and an accounting fact owned by Finance.
2. Operational access design: how the billing-dispute screen gets its joined view of bookings, customs and GPS history, chosen from Figure 9.1’s menu. State its declared staleness (“up to N minutes behind”) and name the business stakeholder who signs it off, because staleness is a business decision, not a guess by the engineer.
3. Two data products: Figure 9.2’s anatomy (owner, code, ports, contract) for shipment-movements and customs-status. Include the per-country access rule as platform-enforced policy, a rule the platform applies to every port and every copy, not one each consuming team must remember to honor.
4. Migration path: how to get from 400 ETL jobs to contracts and products without a big-bang cutover. Strangle the old system piece by piece, in the manner of Chapter 3’s strangler fig: say which two pipelines you replace first, and why, and where you first introduce data contracts checked in the producer’s CI build.
5. One architecture decision record (ADR): “why the central data team becomes a platform team.” Include the losses, not only the gains. The team stops building pipelines for other people’s data and starts building the platform that lets domains publish their own, and something is given up in that trade (control over the pipelines, perhaps, or the deep knowledge of the tables that came with running them). Name what is lost honestly.

Rubric:

Where you now stand

Data now has owners, contracts, products and a platform to serve them. Yet this chapter leaned repeatedly on policies enforced as code, for privacy, erasure and per-country access, without explaining the security discipline that makes such policies trustworthy. The careful reader will have noticed that debt. Chapter 10 pays it: it draws the trust boundaries, threat-models the flows we have built across nine chapters, and treats security as an architectural characteristic, checked continuously by fitness functions, rather than as a questionnaire filled in at the end.

References

• Adam Bellemare — Building an Event-Driven Data Mesh. O’Reilly, 2023.
• Jacek Majchrzak, Sven Balnojan, Marian Siwiak — Data Mesh in Action. Manning, 2023.
• Zhamak Dehghani — Data Mesh: Delivering Data-Driven Value at Scale. O’Reilly, 2022.
• Zhamak Dehghani — How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh. martinfowler.com, 2019 — free online.
• Ian Gorton — Foundations of Scalable Systems. O’Reilly, 2022.
• Joe Reis, Matt Housley — Fundamentals of Data Engineering. O’Reilly, 2022.
• Neal Ford, Mark Richards, Pramod Sadalage, Zhamak Dehghani — Software Architecture: The Hard Parts. O’Reilly, 2021.
• Bitol — Open Data Contract Standard, v3. Linux Foundation AI & Data — free online.
• Ralph Kimball, Margy Ross — The Data Warehouse Toolkit, 3rd ed.. Wiley, 2013.