Software Architecture in the AI & Cloud Era

← the whole book on one page

Chapter 6: Event-Driven Architecture and Streaming Systems

A call asks permission; an event states a fact. Systems built on facts age better.

Every chapter so far has leaned on a quiet promise: that one service can learn what happened in another without having to ask. We have used that promise again and again without stopping to examine it. In Chapter 1’s trade-off, purchase facts reached Sales Analytics from outside the purchase path, so checkout never had to wait on a dashboard. In Chapter 3, SeatSold carried the news of a sale across bounded contexts, so that each context could react in its own language. In Chapter 5, the same events drove the sagas that coordinate a purchase across several services. In each case the arrangement was the same: one piece of software announced a fact, and whoever cared about that fact reacted to it. That arrangement has a name, event-driven architecture, and this chapter finally looks at it directly. It is not an exotic style. Most of today’s payment, fraud-detection and data-pipeline infrastructure runs on it, including the pipelines that feed Chapter 12’s AI systems.

The style earns a chapter of its own because events fail differently from calls, and in a way that is harder to notice. When a call is slow, someone is waiting for it: a timeout fires, a latency graph climbs, a fan stares at a spinner, and the trouble is plain to see. When an event topology (the web of who publishes which events and who consumes them) is badly designed, nothing waits and nothing visibly breaks. Instead, an effect is quietly applied twice, so a fan receives two confirmation emails or a sale is counted twice on an organizer’s dashboard. Or two facts arrive in the wrong order, so a projection shows 14B as available after it was sold. Or a workflow stalls halfway, and nobody can trace which service was supposed to act next. The lesson is that replacing calls with events does not remove complexity; it relocates it. The complexity leaves the call stack, where debuggers and tracers can see it, and settles in the space between services, where our tools mostly cannot. And there is one more reason for care. A caller knows the service it calls, but an event’s producer usually has no idea who its consumers are; they are strangers who subscribed on their own. An event is therefore a promise made to people the producer will never meet, which is why it deserves the same care we give to designing an API.

6.1 Event-Driven Fundamentals

Events, commands, and the direction of authority

We begin with two kinds of message that look alike on the wire and mean very different things. A command asks a specific receiver to do something, and the receiver is free to refuse. “Reserve seat 14B” is a command: Seat Inventory may carry it out, or it may answer that 14B is already held by another fan, or that the on-sale has not opened yet. An event states that something has already happened. It is written in the past tense, it belongs to the service that produced it, and there is nothing left to refuse: SeatSold tells the world that seat 14B now has an owner, and no service that reads it can make that untrue. The distinction sounds merely grammatical, a matter of tense, but here grammar encodes authority, by which I mean the right to decide what happens next. With a command, the sender holds that right: it has an intent, and it picks the one service that should carry it out. With an event, the right passes to each consumer: the producer only reports the fact, and every service that hears it chooses its own reaction. Notifications may email the fan a confirmation, analytics may add the sale to a chart, and Seat Inventory, which reported the sale, has a say in neither.

An everyday comparison makes the difference concrete: a command is a letter, and an event is a newspaper. A letter is addressed to one person and waits for a reply, so if the recipient says no, the sender learns of it and must decide what to do instead. A newspaper is printed for readers its editors will never meet, and each reader decides alone whether a story matters to them. The analogy is worth following until it breaks, because it breaks at an instructive point. A newspaper can quietly amend yesterday’s story, but a published event cannot be recalled, because its consumers have already read it and acted on it. Suppose a fan buys seat 14B and is later refunded. The SeatSold event is not erased; instead a later event records the refund, and every consumer reacts to that new fact in turn: Notifications sends a refund email, analytics subtracts the sale. A mistaken or overtaken fact is answered only by a later fact that compensates for it, which is exactly the move Chapter 5’s sagas make: they never roll back, they compensate.

This shift of authority has a striking consequence: Seat Inventory doesn’t even know Notifications exists. It publishes SeatSold and moves on, and whoever has subscribed receives it. So when the organizer-payouts service ships next quarter, it simply subscribes to SeatSold, and Seat Inventory does not change a line of code, redeploy, or hold a meeting. This is Parnas’s criterion from Chapter 1 carried to the scale of a whole system. Parnas asked each module to hide its likely-to-change design decisions from the rest; here the producer hides not only its implementation but the very existence of its consumers, and the list of consumers is exactly the kind of thing that keeps changing. Every decoupling benefit of events comes from that ignorance, and so does the debugging pain we will meet in Section 6.4. The two are the same property seen from opposite sides. A producer that knows nothing of its consumers can never be broken by them: a crashed Notifications service cannot slow down a sale. But it can never tell you what they did either: when a fan writes in to say the confirmation email never came, Seat Inventory’s logs show only that the event went out, and the trail stops there.

What goes inside an event

Once we have decided to publish an event, we must decide what to put in it, and this payload is the most consequential small decision in the style, because it decides whether the decoupling we have just admired is real or only apparent. Martin Fowler named the first two options, “event notification” and “event-carried state transfer”; delta events are a common third:

Figure 6.1 — Three payload philosophies, with what each buys and what it bills. Default to event-carried state with a curated payload, meaning only the fields consumers actually need. An event schema is an API, and Chapter 5’s expand–contract rules apply unchanged.

It helps to see one sale written all three ways. As a notification, SeatSold carries little more than an ID, and any consumer that wants to know which seat, for which concert, at what price, must ask Seat Inventory. With event-carried state, the event itself carries the seat, the concert, the price and the order it belongs to, so Notifications can write the confirmation email and analytics can update its chart without asking anyone anything. As a delta, the event says only what changed, perhaps that the seat’s status moved from held to sold, and a consumer that wants the full picture must have followed every earlier change to that seat. Each shape moves work to a different place: the notification moves it back to the producer, the delta moves it into each consumer’s memory of history, and event-carried state does the work once, when the event is written.

The notification pattern deserves a closer look, because it fails subtly. An event that says only “order 881 changed” looks perfectly decoupled on a diagram: the producer announces, the consumers listen, and no arrow points backward. Yet every consumer must then call the producer to learn what changed, and they all call at once, the moment the event arrives. Now picture an on-sale. Seats sell in a rush, each sale produces an event, and each event sends every consumer back to Seat Inventory for details, so its read traffic grows with every new subscriber, at exactly the moment Seat Inventory can least afford it. And if Seat Inventory slows down, every consumer slows with it. That is synchronous coupling: the consumers cannot do their work unless the producer answers right now. The coupling was never removed, only hidden behind an asynchronous first step.

Payload is not the only thing that makes an event a contract between strangers; two more rules complete the picture. The first is that events describe the domain, the business as fans and organizers understand it, never the producer’s implementation. SeatSold names something that happened at a concert; SeatRowUpdated names something that happened in a database table. A row update leaks the shape of that table to every consumer, and table shapes are precisely the kind of volatile decision Parnas would have us hide. The day Seat Inventory splits its seats table in two, or moves to a different database, every consumer of SeatRowUpdated breaks, while consumers of SeatSold notice nothing, because seats are still being sold.

The second rule is that schemas evolve additively: a producer may add new optional fields, but it does not remove a field or change what one means while anyone may still depend on it. Good intentions are not enough to keep that promise, because a producer’s team cannot see its consumers, so the rule is enforced by a machine. A schema registry is a shared service that stores every version of every event’s schema. In continuous integration (CI), each proposed change to an event is checked against the registered versions, and the build fails if the change would break existing readers, for instance by deleting the price field that analytics relies on. That way no producer can break a stranger by accident: the check catches the mistake before it ships, even though the producer’s team has never heard of the consumer it would have hurt.

Recap. A command asks one receiver to act and may be refused, so the sender stays in charge; an event declares a fact that has already happened and leaves the reaction to each of its consumers, who may be strangers to the producer. A published event cannot be recalled, only answered by a later, compensating event. The payload decides how real that decoupling is: notifications hide synchronous coupling, deltas push history onto consumers, and event-carried state with curated fields is the default. Event schemas are APIs: they name domain facts rather than tables, and evolve under expand–contract rules checked by a schema registry in CI.

Exercise 6.1. Take one message in a system you know and decide whether it is a command dressed as an event, such as SendEmailRequested: past-tense or event-like in name, but really an instruction to one particular receiver. If it is, write down who is actually in charge of what happens next, and propose the event that would describe what really happened in the business (for example, the sale or sign-up that made someone want the email sent).

6.2 Event Streams as State

The log: a database turned inside out

Section 6.1 treated events as messages in flight, facts on their way from a producer to whoever cares. This section asks what happens if we keep them. The difference starts with two structures that look alike from a distance. A queue hands each message to a consumer and then forgets it: once the message is acknowledged, it is gone, and nobody can read it again. A log, the abstraction behind Kafka, keeps it. A log is an append-only, ordered record. New entries go on the end, nothing in the middle is ever rewritten, and every entry keeps its position. Each consumer holds its own cursor, a bookmark saying how far it has read, so ten consumers can read the same log at ten different speeds without disturbing one another. And because nothing is discarded, any consumer can move its cursor back and read again; that is what we mean by replayable.

The idea is older than Kafka. Databases have long kept such a log internally, often called a write-ahead log: before changing a table, the database first appends a note of the change, so that after a crash it can read the notes back and rebuild whatever it had not finished writing. For decades that log was a private detail, hidden beneath the tables everyone actually queried. Jay Kreps and Martin Kleppmann saw that it could be turned outward and shared by a whole system, so that the log becomes the primary record and the tables become views derived from it. They call this the database turned inside out. The consequences are practical. A service that Encore builds years from now can arrive, set its cursor to the beginning, and replay every sale that ever happened. An analytics team that discovers a flaw in how it counted no-shows can reprocess last season’s data with corrected logic instead of shrugging at a bad number. The stream stops being mere plumbing that moves facts from one place to another and becomes a record of the business itself.

Replay has a limit, though, and the limit is set by what you are willing to store. How far back a consumer can rewind is a retention decision, and retention comes with a storage bill. Kafka keeps seven days of history by default; anything older is deleted. When longer history is worth paying for, there are several ways to get it. Tiered or object storage moves old segments of the log onto cheaper storage. Compacted topics keep only the latest event for each key, which suits streams where you need current state rather than the full story, such as “the latest price for each ticket tier”. A lakehouse sink, such as Iceberg tables, copies the stream into analytical storage that can hold years of it. One more thing has changed in recent years: the line between queue and log, once sharp, has blurred. Pulsar, and Kafka’s newer share groups (billed as “queues for Kafka”), let consumers take work from a log one message at a time, queue-style, while the log underneath still keeps everything.

Follow the idea of the log far enough and you arrive at event sourcing, where the events are not a record of the state; they are the state. Consider seat 14B. A conventional Seat Inventory would keep a row that says “14B: sold” and overwrite it each time the seat’s status changed, so the earlier statuses would be lost. An event-sourced Seat Inventory stores no such row. It stores the ledger of everything that happened to the seat: Reserved(14B), then perhaps Expired(14B) when a fan’s hold timed out, then Reserved(14B) again for a second fan, then Sold(14B). To answer “what is the state of 14B now?” the service replays those events in order and arrives at “sold”. Replaying a long history on every request would be slow, so the service periodically saves a snapshot, the state as of some event, and replays only the events that came after it.

This sounds novel, but it is one of the oldest ideas in commerce. A bank does not store your balance as a number it overwrites; it keeps a journal of deposits and withdrawals, and your balance is simply their sum. If the bank and you disagree, you both go back to the journal. Luca Pacioli described double-entry bookkeeping in 1494, some five centuries before software gave the ledger a pattern name. Accountants learned long ago that a ledger can settle a dispute and a bare total cannot, and event sourcing is software learning the same lesson.

The ledger lives in an event store, and it is worth being precise about why a message broker cannot do the job. The two serve opposite masters. A broker exists for readers: it takes a fact and fans it out to every consumer that has subscribed. An event store exists for writers: it guards the history of one aggregate, in Chapter 3’s sense the cluster of data whose rules must hold together (here, one seat), against two writers changing it at the same moment.

The guarding works through an expected-version check. Walk through the moment two fans try to buy 14B within milliseconds of each other. Each request reads the seat’s stream and sees, say, three events, so each concludes that 14B is free and that the stream is at version 3. Each then tries to append Sold(14B) with the condition “only if the stream is still at version 3”. The first append succeeds, and the stream moves to version 4. The second append arrives still expecting version 3, finds that the stream has moved on, and is rejected. That fan’s request can then reread the stream, see that 14B is sold, and offer another seat. Nobody is sold the same seat twice, and nobody needed a lock held across the whole purchase. A purpose-built database such as KurrentDB (formerly EventStoreDB) provides this check, and so does a plain Postgres table with a uniqueness constraint on stream and version. Once an event is safely appended, the event store publishes it to the broker, and from there the broker does what it is good at, telling everyone else.

Why go to this trouble? Because Encore’s ticket sales are audited and sometimes disputed, and a ledger answers questions a state table never can. When exactly did 14B sell? After how many holds that expired or failed? In what order did requests arrive during the rush of an on-sale? A row reading “14B: sold” has forgotten all of that. The price is real, though. Engineers must learn append-only thinking: you never fix a mistake by editing history, only by appending a correcting event. Event schemas will change over the years, so old events must be upcast, translated into the current shape as they are read, because the old ones can never be rewritten. And a ledger is a poor thing to query directly for “which seats in section B are still free?”, so current-state queries need a separate path. Providing that path is the job of Command Query Responsibility Segregation (CQRS).

CQRS

CQRS splits a system into two models with different jobs. The write model takes commands and protects the rules: it is made of aggregates guarding their invariants, as in Chapter 3, so it is the part that says “no, 14B is already sold”. The read models answer questions. Each is a projection: a table or index built from the events, and shaped for one kind of question. The ancestry is worth knowing. Bertrand Meyer’s command–query separation, from Object-Oriented Software Construction (1988), asked that a method either change state or answer a question, never both, because a method that does both makes code hard to reason about: you cannot ask it anything without risking a change. Greg Young’s CQRS lifts that discipline from the single method to the whole model. Once the two jobs are separated, each side can be shaped for its own work instead of compromising between them.

For Encore, this means one write model for seats and as many read models as there are kinds of question. The seat map that fans see during an on-sale wants the current status of every seat in a venue, laid out by section and row, and it must be fast. The organizer’s dashboard wants sales per hour and per price tier. The dispute team wants the full history of a single seat. Instead of one schema contorted to serve all three, each gets a projection built for it, fed from the same stream of events.

Figure 6.2 — CQRS fed by an event log: one write model appends events, and read models are projected from the log. The log is the system of record; a projection with the wrong shape is rebuilt by replaying the log through new code. The lag between a write and its appearance in a projection is Chapter 4’s replication lag in domain terms, so set a target for it and alert when it is missed.

This leads to a property that few read schemas have ever had: projections are disposable. In a conventional system, the tables you query are the data, and changing their shape means a careful migration. Here a projection is only an opinion about the log, one particular way of summarizing it. If the opinion turns out to be wrong, because a bug miscounted sales or because the organizer dashboard now needs a breakdown by city, we throw the projection away and form a new opinion from the same evidence: write the new projection code, replay the log through it from the beginning, and switch readers over when it has caught up. The log is untouched throughout.

Where CQRS pays. None of this is free. Event sourcing with CQRS means two models to maintain, infrastructure to build and run the projections, and tooling to replay and rebuild them. Encore uses it for Seat Inventory, because Seat Inventory is audit-critical (sales get disputed), contended (thousands of fans want the same seats at once), and queried in many shapes (the seat map, the dashboards, the dispute team), and it uses it nowhere else. Using it everywhere would repeat Chapter 3’s mistake with tactical domain-driven design (DDD): applying a heavyweight pattern uniformly, where most of the system needed nothing so elaborate.

The transactional outbox. Every service that publishes events, event-sourced or not, faces a classic bug. Suppose Orders & Payments saves a new order to its database and then publishes OrderPlaced to the broker. If the service crashes between those two steps, the order exists but no one hears about it: no confirmation email, no update to analytics. Reverse the steps and the opposite failure appears, an announced order that was never saved. The root of the problem is that the database and the broker are two separate systems, and two systems cannot be updated atomically, all or nothing, without a distributed transaction spanning both, which is slow and fragile. The outbox declines to try. Instead, the service writes the state change and the outgoing event together, in one local transaction against its own database: the order goes into the orders table, and the OrderPlaced event goes into an outbox table beside it. Either both are saved or neither is. A separate relay then reads the outbox table and publishes each event to the broker. If the relay crashes after publishing but before marking an event as sent, it will publish that event again when it restarts, so delivery is at least once, and consumers must still be idempotent, able to receive the same event twice with no extra effect. The general lesson outlives the pattern: when two actions cannot be made atomic, make one of them a durable record of the intention to perform the other.

Change data capture (CDC) comes at the same problem from the database’s side. Every database keeps its own internal log of changes, the one we met at the start of this section, and CDC reads that log and turns each change into a message. This lets systems that have never heard of events produce them anyway, which matters for older systems no one wants to modify. CDC is often the outbox relay itself: rather than polling the outbox table, the relay watches the database log for new outbox rows. The catch is that raw table CDC emits row changes, which are exactly the SeatRowUpdated style of event that Section 6.1 warned against, describing a table rather than the domain. If consumers subscribe to those raw changes, every column of a legacy schema becomes something strangers depend on, and the table can never again be changed freely. So curate CDC output into proper domain events, such as SeatSold, before anyone subscribes, or the legacy schema becomes a public contract by accident.

The right to be forgotten vs. the ledger. Event sourcing collides with privacy law in one sharp place. A fan may demand that Encore erase their personal data under the General Data Protection Regulation (GDPR) or a similar law. Yet the whole value of the ledger is that it never forgets, and its events are never edited or deleted. Both obligations are real, and three design moves reconcile them.

Crypto-shredding encrypts each person’s personal fields with a key that belongs to that person alone, a per-subject key. To erase the person, Encore deletes the key. The events stay in the log, intact and in order, but the personal fields inside them are now ciphertext that no one can ever read again. Keeping personally identifiable information (PII) out of the payload avoids the problem at the source: events carry a stable pseudonymous ID instead of a name or email address, and the personal data behind that ID lives in a single ordinary store that can be changed and erased. The ledger records forever that fan-8812 bought seat 14B; who fan-8812 is stays deletable. Retention windows suit streams that never needed permanent history in the first place: if a stream keeps only thirty days of events, personal data in it simply ages out.

Encore combines the first two moves. The audit ledger survives intact for disputes, and the person can still disappear from it. The hard part is that erasure must reach every copy, not just the ledger. CQRS projections, analytics tables, caches, backups, and machine-learning training sets all hold personal data that has already been decrypted, so deleting the key does nothing to them. Projections are the easy case, because they are disposable: rebuild them, and the subject’s data is gone. Backups and training datasets need their own expiry period, or must keep personal data encrypted under the same per-subject key so that deleting it reaches them too. All this makes key management critical infrastructure. If the store holding the per-subject keys were lost, every fan’s personal data would become unreadable at once, an accidental mass erasure.

Recap. A log keeps what a queue discards, so new consumers can replay history and old data can be reprocessed with better logic. Seat Inventory takes the idea to its end with event sourcing: its ledger of seat events, guarded by an event store’s expected-version check, is the source of truth, and CQRS derives one disposable projection for each kind of question. The transactional outbox commits a state change and its event in one local transaction, and a relay (often CDC) publishes the event at least once, so consumers must be idempotent.

Exercise 6.2. Name one entity in a system you know whose history answers questions its current state cannot, the way 14B’s ledger tells a dispute team when and in what order the seat sold. Write down the questions that only the history can answer, then list the events its ledger would contain, named in the past tense.

6.3 Stream Processing

Partitions, ordering, and delivery guarantees

A single log on a single machine can only absorb so many writes, so at scale the log must be split into partitions: separate, independent logs, each of which can live on a different broker and be read by a different consumer. The split buys throughput, but it costs something we had taken for granted. Order is guaranteed within a partition only. Two records in the same partition are always read in the order they were written; two records in different partitions have no defined order relative to each other at all. Picture a multilane highway. Cars in one lane cannot overtake one another, so their order holds, while the lanes together carry far more traffic than one road ever could. What decides which lane a record joins is its partition key: records with the same key always land in the same partition, so they keep their order. That makes the choice of key a domain decision rather than a technical one, because it amounts to saying which things in the business must happen in sequence and which may happen side by side.

Encore keys its streams by concert, using the event_id column from Section 4.2, where “event” has its ticketing sense of a show on sale rather than a message. Consider what that buys. Every Reserved, Expired and Sold record for one concert travels in one lane, so a consumer always sees a seat reserved before it sees the same seat sold, never the reverse. Meanwhile thousands of other on-sales run in their own lanes, each strictly ordered within itself and none waiting on the others. To spread the work across machines, consumers join a consumer group: the group divides the partitions among its instances, so that each partition is read by exactly one instance at a time. Add instances and the partitions are shared out more thinly, which is how the system scales horizontally while every key keeps its order.

The highway also shows the price. A slow truck holds up its whole lane, and no amount of extra road helps the cars stuck behind it. Because Encore keys by concert, all the traffic for one on-sale flows through one partition, and so through one consumer instance. That is Chapter 4’s hot-shard trade-off appearing again, this time in the stream: when a single concert draws most of the demand, its partition becomes the bottleneck, and adding instances to the group does nothing for it, since an extra instance can only take over some other partition. The tour everyone wants gets one lane, however wide the road.

Now to delivery. Between independent systems, exactly-once delivery does not exist. A message can be lost on the way, or its acknowledgment can be lost on the way back, and the sender cannot tell these two cases apart. Its only safe move is to send again, which means the receiver may get the message twice; delivery is therefore at-least-once. This is the old two generals problem, and it is a property of communication itself, not a gap that better tools will one day close. Two generals on opposite hills must agree on a time to attack, and their only channel is a messenger who may be captured on the way. However many messages and confirmations they exchange, the sender of the last one never knows whether it arrived. A sender that hears no acknowledgment must risk one of two outcomes: a lost message if it stays silent, or a duplicate if it sends again.

Well-built systems choose the duplicate, and then make duplicates harmless. The result is an exactly-once effect: at-least-once delivery combined with idempotent consumers, meaning consumers for which processing the same message twice leaves the same result as processing it once. Chapter 4 introduced idempotency as a duty; in a streaming system it becomes compulsory. Suppose the Sold(14B) record for a concert is delivered twice. An idempotent Seat Inventory consumer recognizes that seat 14B is already sold under that order, does nothing the second time, and the fan still holds exactly one ticket.

You will hear that Kafka transactions and Flink provide “exactly-once semantics”, and within limits they do. Inside their own transactional boundary, they commit a consumer’s input position, its state and its output records together, so a failure and retry inside that boundary never produces a visible duplicate. That guarantee is real, but it covers only what the platform itself controls. It ends the moment the pipeline calls something outside the boundary, such as an external payment service provider (PSP). Kafka cannot roll back a card charge, so the call to the PSP must carry its own idempotency key, and the exactly-once effect has to be rebuilt by hand at that edge.

Poison messages. Ordering has a dark side: a record that waits in line behind another record must wait for it. A poison message is a record the consumer cannot process at all, perhaps because it is malformed or because it triggers a bug, so it fails on every attempt. Since order holds within a partition, that one record blocks every record behind it, which for Encore means every later record for that on-sale. There is no single right response, so each stream must decide in advance. Most streams retry a few times, then set the record aside in a dead-letter queue (or a retry topic) for later inspection and move on to the next record. That keeps the stream flowing, but it gives up strict order for that key, because later records are now processed before the parked one. Seat Inventory cannot make that trade. If it skipped a Sold record, the seat would still look available and could be sold a second time, so it halts the partition and pages a human instead. Whichever choice a stream makes, the signal that something is wrong is the same: consumer lag, the distance between the newest record written to a partition and the last record the consumer has processed. Lag that keeps growing on one partition shows a stuck consumer before fans notice that their purchases have stopped going through, which is why lag is the core health signal of any streaming system.

Time and state in stream processing

So far we have treated the stream as a pipe that carries records from one service to another. Stream processors go further: they compute while data flows, rather than collecting data and analyzing it afterward in a batch. For Encore the difference is practical. A fraud score computed tonight can explain why a bot bought 200 tickets; a fraud score computed in the second between checkout and payment can stop the purchase. Computing on data in motion brings two problems that batch processing mostly hides, and both are worth slowing down for: time and state.

Time. As Lamport showed (Chapter 4), a distributed system has no single “now” on which all its parts agree. Stream processing feels this directly, because every record carries two times. Event time is when the thing actually happened, such as the moment a fan tapped “buy” on a phone. Processing time is when the stream processor finally sees the record. The two can drift far apart: a phone on overloaded stadium Wi-Fi may hold a purchase for a minute before it gets through, and a mobile app may retry a request long after the fan’s first attempt. A question such as “how many purchases happened between 19:00 and 19:10?” is a question about event time, so the processor groups records into windows by event time. The catch is knowing when a window is complete. At 19:10 by the processor’s clock, some purchases made at 19:09 may still be stuck in the stadium.

An astronomer faces a similar gap, because the light from a star left it long ago, but she can correct for it exactly, since light travels at a known speed. A purchase delayed in a crowded stadium has no such constant; its delay depends on the crowd, the network and the phone. So a stream processor cannot calculate when a window is complete; it has to decide. A watermark is that decision: a declaration that the processor will now assume all records up to a certain event time have arrived, and will close the windows before it. Encore might, for instance, declare that it waits 2 minutes for stragglers before closing a 10-minute window. Some records will arrive later still, so the processor also needs a late-data policy: drop them, add them to an already-published result, or send them somewhere for separate handling. Wait longer and results are more complete but slower to arrive; wait less and they are faster but miss more. A watermark is therefore a judgment rather than a measurement, and it should be recorded like any other architectural decision, with its reasoning, so that the next team knows why 2 minutes was chosen and when to revisit it.

State. Many useful checks must remember the past. A bot rule such as “more than five purchases in 10 minutes” needs to know, for each buyer, how many purchases it has already seen in the current window. Fetching that count from a remote database on every record would be far too slow at on-sale volumes, so stateful stream processors keep their state locally, beside the computation. That state then needs three properties to be safe. It is partitioned the same way as the stream, so each instance holds only the state for the keys it reads. It is changelogged: every update is also written to a durable log, so the state survives the loss of the machine. And it is rebuildable: when an instance fails, its replacement restores the state by replaying the changelog, and when the logic changes, for instance when the rule becomes “more than five purchases in 5 minutes”, the state can be recomputed from the input stream itself.

That last property points to one of the quiet gifts of the log. Because the log is replayable (Section 6.2), Encore can reprocess. Suppose the fraud team writes version 2 of its fraud logic. Rather than switching it on during a live on-sale and hoping, Encore can run version 2 over last month’s on-sales and compare its verdicts with the ones version 1 actually produced: which fans it would have blocked that version 1 let through, and which genuine fans it would have wrongly stopped. Only when the comparison looks right does version 2 go live. A new idea gets tested against real history before it is trusted with real fans.

Recap. Partitioning lets a log scale, but order holds only within a partition, so the partition key sets the ordering scope and we choose it from the domain, accepting that a hot key becomes a hot partition. Exactly-once delivery is impossible between independent systems; what we can build is an exactly-once effect, from at-least-once delivery and idempotent consumers, and platform guarantees stop at the platform’s edge. Each stream needs a declared policy for poison messages, and consumer lag tells us when one is stuck. Event-time windows need watermarks and a late-data policy, stateful processors need state that can be rebuilt, and a replayable log makes reprocessing routine.

Exercise 6.3. Take the busiest data flow in a system you know. Write down its natural partition key, and then, separately, the ordering the business actually requires: which records must be processed in sequence, and which may be processed in any order. Are the two compatible today? If not, say what goes wrong, such as a hot partition or an ordering the key cannot guarantee.

6.4 Workflows and Process Automation

Workflow engines for long-running processes

Choreography has served us well so far in this chapter, but its weakness shows on the day a support agent asks “where is order #881 stuck?” Recall how a choreographed workflow works: there is no central coordinator, only five services each reacting to the events of the others. The refund is real, but it exists nowhere as a single thing. Its progress is smeared across the logs and databases of five services, so the question has no owner who can answer it and no screen where anyone can look it up. To find order #881, someone must reconstruct the story by hand, service by service. Nature offers a lovely picture of the trade. A murmuration of starlings wheels across the evening sky with no one in charge; each bird simply follows its neighbors, and the flock’s shape emerges from those local reactions. Yet when we land aircraft at a busy airport, we do not trust emergence. We hand the landing sequence to air traffic control, which knows where every aircraft is and why each one is waiting. The difference is accountability. A flock never has to explain a lost bird, but Encore’s support team must explain a lost refund, to the fan who is waiting for the money.

So for processes that run a long time and that someone must answer for, Encore chooses process orchestration. Think of refunds that need an approval step, organizer payouts with dunning (the scheduled retries and reminders that chase a failed payment), and in general anything that involves humans and deadlines. Here Chapter 5’s saga orchestrator grows into a full workflow engine. The process is modeled explicitly: its steps, branches and waits are written down as one definition, rather than implied by which service happens to listen to which event. That model is versioned like any other code, it is monitored so that order #881 has a status anyone can see, and it can wait 40 days for a reply without tying up a thread. The engine does this by saving the process’s state and setting a durable timer; nothing sits in memory counting the days, so a crash or a redeploy in the meantime loses nothing.

These engines are Chapter 5’s durable-execution engines, such as Temporal, Restate, Camunda and AWS Step Functions. They differ in style. Some, like Step Functions and Camunda, have us describe the process as a model (a state machine or a diagram); others, like Temporal and Restate, are code-first, letting us write the workflow as ordinary code in a general-purpose language. What they share is the promise that a workflow survives crashes, and that promise is worth understanding, because it carries a constraint.

Code-first engines keep their promise by recording and replaying. As a workflow runs, the engine writes down the outcome of every step in a history: “called the PSP, got a refund ID back,” “timer fired.” If the worker running the workflow dies, another worker picks it up and runs the workflow code again from the beginning. This time, instead of actually repeating each step, the engine hands back the recorded result, so the code fast-forwards to exactly where it stopped. That only works if the code makes the same decisions on the second run as on the first, which means workflow code must be deterministic. It may not read the clock, generate a random number or call a service directly, because each of those could give a different answer on replay. Instead it asks the engine for the time, the random value or the service call, and the engine records the answer. Determinism, then, is the price of durability.

Replay also explains why versioning is the hard part. Changing a workflow that nobody is running is easy. The difficulty is changing a process while thousands of instances are mid-flight. Suppose Encore adds a fraud check before the refund is paid, while some refunds started last week are still waiting in review. If those old instances replay their recorded history against the new code, the code expects a fraud-check step the history never recorded, and the replay no longer matches. Engines therefore let old instances finish on the path they started on while new instances take the new path, and someone has to design and test both paths through the transition.

There is a historical warning here, and Chapter 2’s story of service-oriented architecture (SOA) supplies it. In the 2000s, SOA’s process engines were meant to coordinate services, but they often became central brains instead. Rule by rule, the business logic drifted out of the services and into the engine, until the services were thin shells and every change had to pass through one team and one tool. The microservices movement drew the lesson and summed it up as “smart endpoints and dumb pipes,” the phrase James Lewis and Martin Fowler popularized in 2014. A workflow engine is not a dumb pipe, but it can respect the same boundary. It should own a process’s sequence and its waiting, deciding what happens next and when, and leave the business rules, deciding what is allowed, in the services it calls. In Encore’s refund, the engine knows that the ticket must be revoked before money moves. The rules behind each step, such as which refunds qualify for automatic approval or whether a returned seat may go back on sale, belong to the services that own refunds, seats and sales; the engine only branches on their answers.

Figure 6.3 — Encore’s refund process as an explicit model. It shows what choreography hides: the human review step and its service-level agreement (SLA), the ticket revoked before any money moves, the PSP refund call that retries and its exit to manual resolution when retries run out, and the business rule applied at re-listing. The engine emits an event at every transition, so the process is orchestrated inside and choreographed at its edges.

Look at what the figure makes visible. A refund under 80 euros, requested more than 48 hours before the show, is approved automatically; every other request waits for a person in finance operations, who has an SLA of 24 hours to decide. The ticket is invalidated before the PSP is asked to return the money, so a fan can never hold both a valid ticket and a refund. When the PSP call fails, the engine retries it, and when retries run out, the process does not vanish; it lands in front of a human for manual resolution. Each of these facts would be scattered across services in a choreographed design, and here each is a box or an arrow. Even the business rule at the end is visible as a branch: a seat returned more than 48 hours before the show goes back on sale, and one returned later is retired. Notice, too, that orchestration does not abolish events. Every transition is published, so other services can still react to a refund being approved or closed without the engine knowing they exist. The process is orchestrated inside and choreographed at its edges.

Chapter 5’s dividing line therefore still holds. We choreograph the propagation of facts: when a seat is sold, whoever cares may react, and no one needs to be in charge. We orchestrate processes with names, owners and deadlines, because someone must be able to answer for them. Two anti-patterns arise when teams cross that line. The first is the event chain monolith: six services that must fire in exact sequence for anything to work. It looks decoupled, since every hop is an asynchronous event, but the services are as tightly bound as the steps of a single function call; the coupling is merely asynchronous. Change one link and the chain breaks, which is the very brittleness events were supposed to remove. The second is event spaghetti: so many services reacting to so many events that nobody can say what happens after SeatSold without grepping four repositories. Both have the same cure. Every named process gets an explicit model, as the refund has in Figure 6.3. And every event goes into an event catalog, a browsable list of what each event means, who produces it and who consumes it, with its contract written in AsyncAPI, which does for events what OpenAPI does for HTTP APIs. My conviction is simple: if a process has a name in the business, it deserves a model in the code.

Recap. When nobody can answer “where is it stuck?”, the workflow needs an explicit model, run by an engine that makes each instance visible, versions the process safely and can wait for weeks without holding resources. Durable engines achieve this by recording and replaying, so code-first workflow code must be deterministic. The engine owns the sequence and the waiting, while the business rules stay in the services. Aim for orchestrated cores, event-driven edges, and an event catalog that makes the whole topology knowable.

Exercise 6.4. Name one process at your work that has a business name (“onboarding,” “settlement”) but no explicit model, only code reacting to code. Sketch its equivalent of Figure 6.3: the states, the transitions, the steps where people intervene and the exits when something fails. What does the sketch reveal that nobody could see before?

6.5 Streams as Products

So far this chapter has looked at streams from the engineer’s side: how to design their events, keep their logs, partition them, and run processes over them. There is another way to look at a stream, from the side of the team that wants to use it. A stream can also be run as a product, something another team chooses to use because it is worth using, the way they would choose an API. For that to work, the stream needs the qualities we already expect of a good API. It must be discoverable, so a team that needs the data can find it without asking around. It must be documented, so they know what each event means. It must be schema-governed, so its shape changes only in the compatible ways Section 6.1 described. And it must be backed by a service-level objective (SLO), a published target for how well it behaves, so consumers know what they can depend on. A team then subscribes to the stream as it would call an API: it reads the documentation, attaches, and builds, without the producer having to do anything.

Encore’s ticket-sales.v2 stream is run this way. It has a named owner, the team that answers when something goes wrong. It has a contract describing every event it carries. It has a lag SLO: a promise about how far behind the live sale a consumer reading the stream can expect to be, measured by the consumer lag that Section 6.3 called the core health signal. And it has three consumers whose teams the producer has never met. That last detail is the real test of a product. Nobody on the producing team had to be in a room with those consumers, because the owner, the contract and the SLO answered the questions a meeting would have answered.

The payoff shows up in Chapter 12, when a data-science team builds demand forecasting. Without a stream product, that work would start with meetings: who has the sales data, what do its columns mean, can someone write a nightly export job, and who fixes the export when it breaks? With one, the team finds ticket-sales.v2 in the catalog, reads its contract, and subscribes. Because the stream is a replayable log (Section 6.2), the team can also read back through whatever history the stream retains, not only watch new sales arrive. The producer may not even notice a fourth consumer has appeared, which is exactly the point.

Two open specifications keep streams like this from being tied to one broker or one cloud. CloudEvents standardizes the envelope: the common fields every event carries around its payload, such as its type, its source, its ID and its time. AsyncAPI, the OpenAPI equivalent for events that Section 6.4 introduced, describes the contract: which channels exist and what messages flow on each. With the envelope and the contract written in vendor-neutral terms, Encore could move ticket-sales.v2 to a different broker or a different cloud, and its consumers would still recognize every event and still have the same document describing it.

The idea is older than it looks. Its ancestor is the Unix pipe of Chapter 2, which let small programs be joined together without either program knowing anything about the other. A program wrote text to its output, the next program read it, and nobody negotiated. A stream run as a product is that same composability, scaled up from programs on one machine to teams across an organization. What changes with scale is ownership. In a pipeline typed at one terminal, both ends belong to the same person, so if the output format changes, the person who changed it is also the person who suffers. Once the two ends belong to different teams, that shared fate disappears. The owner and the contract are what we add to put it back: someone accountable for the stream, and a written promise about its shape that the consumers can rely on.

James Urquhart calls this world of subscribable, composable streams flow: an economy in which data moves between teams, and eventually between organizations, by subscription rather than by integration project. The idea will return twice in this book. Chapter 9’s data mesh applies it to analytical data, treating datasets as products with owners and contracts. Chapter 15’s platform goes further and serves streams the way it serves compute, as something a team asks for and receives rather than builds for itself.

Recap. A stream becomes a product when it has an owner, a contract, and an SLO, and when other teams can discover it and subscribe on their own. Those three things answer the questions that would otherwise need coordination meetings and export jobs, so subscribing replaces negotiating. The idea is the Unix pipe’s composability at the scale of an organization, with ownership added because the two ends now belong to different teams. Data mesh (Chapter 9) and the platform (Chapter 15) build on it.

Exercise 6.5. Pick the most valuable internal data your team produces, and imagine it flowed tomorrow as a product-grade stream with an owner, a contract, and an SLO. Name the team that would subscribe first, and list the requests they would stop sending you: the exports, the ad-hoc queries, the meetings to explain what a field means.

Kata: The Event Backbone

This kata leaves Encore behind and hands you a younger company with the same underlying problem: a business that is really a stream of things happening, held together by code that keeps asking whether anything has happened yet. Your task is to give it an event backbone, using the tools of this chapter: events as contracts, logs that keep history, streams processed as they flow, and named processes with an engine that owns them.

Your brief: “Loop.” A food-delivery startup with 30 engineers. Each order moves from placed to accepted by the restaurant, then to courier assigned, picked up, and delivered; at any point it may instead leave by one of five failure exits. Today the system runs on synchronous calls between services, plus a nightly CSV export to analytics. Couriers’ apps poll the server every 5 seconds to learn whether anything has changed, and the restaurant’s tablet misses updates. The business now wants four new things: live order tracking for customers, courier surge pricing (which needs demand counted over 10-minute windows), a refund process with a human review step, and a fraud team that wants to “see everything, later.”

Deliverables:

1. Event catalog — the domain events Loop publishes, each with its payload philosophy (Figure 6.1: a bare notification, the relevant state, or only the delta), and the rules for evolving each schema without breaking its consumers.
2. Topology — whether each stream lives in a message broker or a replayable log; the partition key for each, with the ordering it guarantees and why that ordering is the one the business needs; the consumer groups that read each stream; and where CQRS projections serve the tracking screens.
3. Process model — the refund workflow drawn to the standard of Figure 6.3: an orchestration engine owning the process at its core, emitting and receiving events at its edges.
4. Delivery guarantees — for courier assignment specifically: how at-least-once delivery plays out, which mechanism makes processing idempotent, and what the courier actually sees on screen when a duplicate arrives.
5. One architecture decision record (ADR) — the migration of courier apps from polling to push, including what is lost along the way.

Rubric:

A word on that last row, because it is the easiest to skip. Polling is wasteful, but it has a quiet virtue: every 5 seconds the app asks again, so a lost answer heals itself on the next request. Push removes the waste and the virtue together. A notification dropped while the courier rides through a tunnel is simply gone, unless the design gives the app a way to ask “what did I miss?” when it reconnects. A good ADR names that loss and pays for it explicitly.

Where you now stand

You can now design events as contracts that other teams can rely on, keep history in logs so that new consumers can replay it, process streams as the events flow rather than in a nightly batch, and give each named process an engine that can account for where every instance stands. The next chapter turns to the boundary where your system meets everyone else’s: the API, which you must publish, version, defend, and operate.

References

• Adam Bellemare — Building Event-Driven Microservices, 2nd ed.. O’Reilly, 2025.
• Mark Richards, Neal Ford — Fundamentals of Software Architecture, 2nd ed.. O’Reilly, 2025.
• Bernd Rücker — Practical Process Automation. O’Reilly, 2021.
• Gwen Shapira, Todd Palino, Rajini Sivaram, Krit Petty — Kafka: The Definitive Guide, 2nd ed.. O’Reilly, 2021.
• James Urquhart — Flow Architectures. O’Reilly, 2021.
• Tyler Akidau, Slava Chernyak, Reuven Lax — Streaming Systems. O’Reilly, 2018.
• Ben Stopford — Designing Event-Driven Systems. Confluent/O’Reilly, 2018 — free online.
• Martin Fowler — “What do you mean by ‘Event-Driven’?”. martinfowler.com, 2017 — free online.
• Jay Kreps — “The Log: What every software engineer should know about real-time data’s unifying abstraction”. LinkedIn Engineering, 2013 — free online.
• Gregor Hohpe, Bobby Woolf — Enterprise Integration Patterns. Addison-Wesley, 2003.