Operating Production Systems in the AI & Cloud Era

← the whole book on one page

Chapter 4: Observability — Seeing What the System Is Doing

The data will answer almost any question, but only the ones we think to ask.

In the last days of August 1854, cholera broke out around Broad Street in Soho. Within ten days some five hundred people were dead, nearly all of them living within a few hundred yards of one another. The explanation most physicians and officials accepted at the time was miasma: the belief that disease rose from the foul air of a crowded city, and so spread wherever the air was bad. John Snow, a London physician, had argued since 1849 that cholera traveled in water instead. He had made that argument before and had not persuaded anyone, so this time he did not argue. He gathered evidence. He obtained the register of deaths, went to find out where each victim had drawn their water, and marked every death as a small bar against its address on a map of the district.

The map made the pattern visible. The deaths clustered around the public pump on Broad Street, and the exceptions turned out to be the most persuasive evidence of all, because each one had a reason that only the water theory could explain. The workhouse on Poland Street, close to the pump, had its own well and very few deaths. The men of the nearby brewery drank beer rather than pump water and were spared. A widow in Hampstead, miles away and far outside the cluster, liked the taste of Broad Street water so much that she had a bottle carried to her every day, and she died of it. On 7 September Snow put his case to the parish Board of Guardians, and the next day the pump’s handle was removed. It would be tidy to say that this ended the outbreak, but it does not quite hold: the outbreak was already waning by then, and official opinion stayed unconvinced for years. The final piece came from an unexpected direction. Henry Whitehead, a curate who set out to disprove Snow by interviewing his parishioners one by one, ended up finding the source himself: a baby’s soiled napkins, rinsed into a cesspool a few feet from the well.

What strikes me most about the story is that Snow had no better instruments than the miasma theorists. He had no microscope that showed him the cause and no data his opponents lacked. His advantage was a different question, where did each victim get their water?, which the prevailing theory did not think worth asking, and a way of arranging the data, one mark per death at its exact address, that let the data answer it. That combination is what this chapter is about. The Architecture book borrowed the word observability from Rudolf Kálmán’s control theory. Here we use it in its working, everyday sense: observability is the ability to ask a production system a question nobody anticipated and get an answer without shipping new code. The last clause matters. If answering a new question means adding a log line, deploying it and waiting for the problem to happen again, then we are not observing the system; we are running an experiment on it, slowly, while fans wait.

Chapter 3 ended with people supervising automation, and supervision is only as good as what the supervisor can see. Encore is not short of things to look at. It has telemetry everywhere: metrics from every service, spans from the service mesh that trace requests as they hop between services, logs that make up much of a telemetry bill the Resilience book weighs in its chapter on cost, and dashboards with more panels than anyone has ever looked at in one sitting. Volume, then, is not the problem, any more than the miasma theorists lacked counts of the dead. The real question for Encore is the one Snow faced: when something new goes wrong, something no dashboard was built to show, can its engineers find out why before its fans notice? The sections that follow work toward an answer, beginning with the difference between checking for failures we expected and investigating the ones we did not.

4.1 From Monitoring to Observability

Known questions and unknown ones

Two words are often used as if they meant the same thing, and the first task of this chapter is to pull them apart. Monitoring is the practice of checking a system against conditions we decided in advance were worth checking. Is the error rate above 1%? Did the synthetic buyer, the scripted fan that tries to buy a ticket, complete its purchase? Each monitor is a question written down before the trouble arrives, and the system is asked that question over and over until one day the answer changes. Monitoring is indispensable, and nothing in this chapter argues for less of it. But it has a limit built into its definition: a monitor can only encode a failure that someone has already imagined. Observability, as the opening of this chapter defined it, is the complementary property: a system is observable when we can understand its internal state from its outputs well enough to answer questions that nobody thought of in advance. Monitoring tells us that a known thing has gone wrong; observability lets us find out what is going on when the thing that has gone wrong is new.

The distinction matters more now than it once did, because the number of ways a system can fail has grown faster than our ability to imagine them. A monolith running on three servers failed in ways an experienced operator could list: a full disk, an exhausted connection pool, a slow query, a server that had stopped answering. A good operator could write a monitor for most of them, and the list did not change much from year to year. Encore’s purchase path is a different kind of object. A single ticket purchase crosses a serverless waiting room (the Sale Gate), a bot screen, an event-sourced inventory sharded four ways (Seat Inventory), a saga orchestrator (Orders & Payments) and an external payment service provider (PSP), and it does so across two regions and a service mesh. (Chapter 1’s map of Encore shows where each piece sits.) Each of those parts fails in its own ways, and the parts can also fail in combination: a slow shard that only matters when the saga retries, or a mesh timeout that only bites in one region. Nobody can list the ways those parts combine, and so nobody can write a monitor for each of them in advance.

Peng Huang and his colleagues at Microsoft gave the characteristic failure of such systems a name in a 2017 paper drawn from their experience with Azure. They called it gray failure: a partial failure that some observers see and others do not. Its signature is what they called differential observability. The system’s own health checks report that all is well, while some of its users are suffering. The service is not down, so nothing that tests whether it is up will notice; it is merely wrong, for some people, some of the time. A year later Cindy Sridharan’s short book Distributed Systems Observability took the idea and made it the practitioner’s case for observability. The reasoning is worth following step by step. A monitor is, in the end, an average or a threshold: the error rate across all requests, the latency at the 99th percentile, a check that either passes or fails. Averages and thresholds answer well when something is down, because then everyone is affected and the average moves. They answer poorly when something is wrong for some people, because those people are hidden inside the average. If one fan in fifty cannot pay, the overall success rate barely shifts, and the dashboard stays green while those fans give up and leave. We will meet exactly such a failure at Encore in Section 4.5.

An analogy may help to fix the difference, provided we also notice where it fails. Monitoring is like the warning lights on a car’s dashboard: oil pressure, engine temperature, a light for each condition the manufacturer anticipated. Observability is like the mechanic’s workshop, with its tools for opening the engine and examining whatever is there. When the car makes a noise that matches no light, adding more lights will not help, because no one knew to build a light for that noise; what we need is the ability to look inside. The analogy breaks in a way that makes the case stronger. A car’s engine is the same engine on every visit to the workshop, so a mechanic’s experience accumulates, and last year’s diagnosis is likely to apply this year. Encore’s system changes dozens of times a day, with each deploy, configuration change and scaling decision. The machine we are diagnosing tonight is not quite the machine we diagnosed last week, and so our experience is a less reliable guide, and the ability to ask fresh questions of the system as it now is becomes correspondingly more valuable.

What observability is made of

If observability is the ability to ask new questions, we should ask what makes that ability possible. The answer lies less in tools than in the shape of the telemetry itself, the data a system emits about what it is doing. Three properties of that data matter.

The first is that the data must be high in cardinality. The cardinality of a field is the number of distinct values it can take. A field such as the HTTP method has low cardinality, a handful of values; a field such as an order ID, an event_id or a build hash has high cardinality, potentially millions of values. High-cardinality fields matter because many failures are specific to one of those values: one concert’s on-sale, one build that went out at the wrong moment, one fan’s order that is stuck. If the data does not record which order or which build a request belonged to, we cannot ask about it later. The second property is that the data must be high in dimensionality, which is to say that each event carries many fields, not just a few. This matters because we do not know in advance which field, or which combination of fields, will explain why the failing requests differ from the healthy ones. Perhaps it is the region; perhaps it is the region together with the client version and the payment method. Every field we leave out is a hypothesis we will be unable to test. The third property is that the data must be explorable: kept in a form that lets us slice, group and compare by any field after the fact, without having decided in advance which views we would want. Detailed data that can only be read through pre-built dashboards gives us back the limits of monitoring.

John Snow, whose work on the Broad Street outbreak opened this chapter, had data with exactly these properties. His register held one row per person who died, with an address, a date and, after his door-to-door inquiries, the source from which that person drew water. That is high cardinality (individual addresses, not districts), high dimensionality (several facts about each death) and explorability (he could arrange the rows on a map and look for a pattern nobody had proposed). The miasma theorists worked from a summary instead: deaths per district per week. That summary was not wrong, but in producing it someone had thrown away the address, and with the address went any chance of seeing the pump. This is the general lesson. Aggregation, the act of reducing many individual records to a count or an average, is a decision about which questions will never be asked again, and we usually make that decision before we know what the questions are.

None of this is free, and the price depends on how the data is stored. Prometheus, the open-source metrics system, stores a separate time series for every combination of label values. A label with a few values, such as region, adds a few series; a label whose value is a fan ID would add one series for every fan, multiplied by every combination of the other labels, and that would ruin it, overwhelming its memory and storage long before anyone asked a question. Prometheus is built for aggregates, and it is very good at them. Columnar event stores take the opposite approach. Facebook’s Scuba, described in 2013, stores each event as a row with as many fields as it carries and is built to scan and group those rows quickly, which makes questions about individual high-cardinality values affordable. Neither design is better in general; each makes a different set of questions cheap. Encore keeps both: metrics in Prometheus for the known questions that monitoring asks continually and cheaply, and wide events in a columnar store for the questions nobody has thought of yet. The rest of this chapter is largely about how to produce that second kind of data and how to use it.

Recap. Monitoring checks conditions we chose in advance, and it remains necessary. Observability lets us ask about conditions nobody anticipated, such as gray failures, which hide from averages because they harm only some users. That ability comes from telemetry that is wide (many fields per event), high in cardinality and explorable after the fact.

Exercise 4.1. List the last five production problems in a system you know. For each, note whether an existing monitor detected it, and whether the cause was found using pre-built views or by asking a question those views could not answer.

4.2 Signals and Instrumentation with OpenTelemetry

Braided signals

For about a decade the industry described telemetry as “three pillars”: metrics, logs and traces. Each pillar had its own pipeline, its own store, often its own vendor and its own team, and the picture seemed natural enough, since the three signals do look different. A metric is a number aggregated over time, a log line is a record of something that happened, and a trace is the record of one request’s path through many services. The metaphor did harm all the same, because pillars stand apart, and the value of telemetry lies in the connections between the signals. An engineer who sees a latency spike on a metric graph wants the very requests that caused it, and then the log lines those requests wrote. If the three signals live in three systems that share nothing, each step of that journey becomes a search by timestamp and guesswork, and at 3 a.m. guesswork is where investigations stall.

A braid is the better image. A braided rope holds weight because its strands are wound together, so that a load carried by one is shared by the others. What winds telemetry’s strands together is shared context: the same trace ID, a unique identifier given to a request when it enters the system and carried with it through every service it touches, and the same names for the same things everywhere. The image has its limit, as images do. A rope’s strands are interchangeable, whereas telemetry’s are not, and each signal still answers best the kind of question it was built for: metrics for “how much, and how fast, right now”, logs for “what exactly happened here”, traces for “where did this request spend its time”. The braid tells us how the signals should relate, and it does not tell us that one could stand in for another.

Two humble mechanisms make the braid real, and I have come to believe they are the cheapest observability a team can buy. The first is to write the trace ID and the span ID (the identifier of the current unit of work within the trace) into every structured log line. Finding the logs of a slow request then stops being a search and becomes a lookup: we take the trace ID from the trace and ask the log store for every line that carries it. The second is the exemplar, a trace ID attached to an individual sample recorded in a metric’s histogram. A latency graph built from that histogram can then offer, right beside its spike, a link to an actual request that was slow, so that the engineer moves from “p99 latency doubled at 14:02” to one concrete example of it in a single click. Neither mechanism is clever, and neither is expensive. Encore’s service template does both by default, which means no team has to remember to.

None of this would be practical if every vendor and every library spoke its own dialect of trace IDs and attributes, and for most of the last decade they did. OpenTelemetry (OTel) is the vendor-neutral standard that ended much of that fragmentation. It formed in 2019 from the merger of two earlier projects, OpenTracing and OpenCensus, which had been solving overlapping problems in incompatible ways. Its central design decision is a separation of concerns. Code is instrumented against an application programming interface (API) that says what to record, while a software development kit (SDK), configured at deployment, decides where the data goes and how it is processed. A team can therefore instrument its service once and change backends later without touching the code. OpenTelemetry’s stable signals are traces, metrics and logs; profiles, which record where a program spends its CPU and memory, are arriving as a fourth, and are in alpha in 2026.

Semantic conventions as a shared vocabulary

A shared trace ID joins the signals of one request. To join the signals of many services, they also need shared names. OpenTelemetry’s semantic conventions are agreed names and meanings for common attributes. Every resource carries service.name and service.version; every HTTP span carries http.request.method and http.response.status_code; and the conventions continue in the same spirit through databases, messaging and, still in development, calls to generative AI models. The conventions fix both the name and the meaning, so that http.response.status_code is always the numeric status the server returned, recorded in the same way by every service that follows them.

The cost of not having such a vocabulary is easy to underestimate in calm times and painfully obvious in an incident. Suppose Orders records the HTTP status as status, Inventory as http_code and the Sale Gate as response.status. Then “show me every 5xx on the purchase path” is three different queries, each written by someone who first has to discover what the other team called the field. In the middle of an incident nobody writes all three. They write the one they know, see nothing alarming in it, and move on. An orchestra offers a fair comparison: its players can sight-read together because the notation is shared, and a single player who wrote music in a private notation would be silent at the first rehearsal however well they played. The comparison breaks in one useful place. An orchestra’s notation was settled centuries ago and changes slowly, while a company’s telemetry vocabulary grows every week as teams add services and fields, and so it needs an owner and a process, where music has only tradition.

Encore gives its vocabulary both. The Reliability team owns a namespace that extends the standard conventions, kept in a registry that continuous integration (CI) checks with OTel’s Weaver tool, so that a service emitting an unregistered or misspelled attribute fails its build before the mistake reaches production. OpenTelemetry asks companies to keep their own names under a unique prefix, so that they never collide with future standard ones, and every Encore field therefore lives under encore.: encore.event_id, encore.tenant_id, encore.shard, encore.cell, encore.country, encore.payment_method, and encore.on_sale. That last one is worth a moment, because the sampler below depends on it. The golden-path service template stamps encore.on_sale on every span, set to true during a scheduled on-sale and false otherwise, so that any query or policy can tell an on-sale request from an ordinary one without consulting a calendar.

This is the old idea of information hiding, applied in a new place. Information hiding asked each module to expose a stable contract and keep its internals private, so that other modules could depend on the contract without knowing the internals. The registry puts telemetry names behind such a contract. One team’s query works against another team’s service because both depend on the registry and neither depends on the other’s private habits of naming.

The Collector and the economics of sampling

Once services emit consistent telemetry, something has to carry it to the stores where it will be queried, and decide on the way how much of it to keep. Between the services and the backends sits the OpenTelemetry Collector, a program built as a pipeline of three kinds of component: receivers take data in, processors transform, filter or batch it, and exporters send it on. Encore runs the Collector in two tiers. The first tier is a set of agents, one on every node and one beside each serverless function. The agents batch what they receive, forward it, and redact the sensitive values they recognize. The second tier is a gateway: a shared pool of Collectors that strips whatever personal data got past the agents, routes security events to their own separate account, and decides which traces to keep. Every pipeline in both tiers begins with a memory limiter, a processor that starts refusing data when the Collector’s memory nears a set limit, so that a flood degrades into dropped telemetry rather than crashed Collectors. Encore needs it because an on-sale multiplies telemetry by the same 500× as it multiplies traffic.

That multiplication is the reason sampling exists. Recording every span of every request during an on-sale would cost a great deal and tell us very little, since most of those requests are identical successes. Sampling is the decision to keep some traces and discard the rest, and the important question is when that decision is made. Head sampling decides at the start of a request whether its trace will be recorded, usually by a random draw. It is cheap, because a request that loses the draw is never recorded at all, and it is blind, because at the start nobody knows whether this request will be the one that fails. Tail sampling decides after the trace is complete, when its errors and its total duration are known. That makes it far better at keeping the interesting traces, at a price: the sampler must hold every span of every trace in memory until it decides, and all the spans of one trace must reach the same sampler instance, since an instance that has seen half a trace cannot judge it. Encore meets the second requirement with a load-balancing exporter in the agents, which sends each span to a gateway instance chosen by its trace ID, so that every span of a trace lands in the same place. The OpenTelemetry project itself is candid that the tail sampler is stateful and hard to operate, and Encore’s experience, below, bears that out.

Encore’s tail-sampling policy keeps three kinds of trace: every trace with an error, every checkout slower than 2 s, and a baseline sample of the rest. The baseline is 10% normally and 1% during an on-sale. The reason for the lower rate is arithmetic. At 1,200 admissions a second for one event, a 10% baseline would keep 120 traces a second, almost all of them identical successes; at 1%, it keeps 12 a second, which is still more than enough to show what normal looks like.

Expressing that baseline turns out to be subtler than it sounds, because of how the sampler combines policies. It keeps a trace if any policy matches it. Suppose we wrote the obvious thing: one policy that keeps 10% of traces, and another that keeps 1% of on-sale traces. During an on-sale, each trace would be offered to both, and the 10% policy would keep it whenever its draw succeeded, whatever the 1% policy decided. The effective rate would stay at 10%, and the 1% policy would change nothing. The baseline must instead be split into two policies that cannot both apply to the same trace, each an and of a condition and a probability: “on-sale and 1%”, and “not on-sale and 10%”. This is exactly why the template stamps encore.on_sale on every span.

tail_sampling:
  decision_wait: 60s                                 # outlasts a bank-redirect callback
  decision_cache: { sampled_cache_size: 1000000, non_sampled_cache_size: 1000000 }  # late spans follow the decision
  policies:
    - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
    - { name: slow, type: latency, latency: { threshold_ms: 2000 } }
    - name: baseline-on-sale
      type: and
      and: { and_sub_policy: [
        { name: on-sale, type: boolean_attribute, boolean_attribute: { key: encore.on_sale, value: true } },
        { name: one-pct, type: probabilistic, probabilistic: { sampling_percentage: 1 } } ] }
    - name: baseline-off-sale
      type: and
      and: { and_sub_policy: [
        { name: off-sale, type: boolean_attribute, boolean_attribute: { key: encore.on_sale, value: false } },
        { name: ten-pct, type: probabilistic, probabilistic: { sampling_percentage: 10 } } ] }

The policies are where the reasoning above lives, but the first two lines are where Encore learned something. A tail sampler cannot know when a trace is truly complete, since there is no message that says “this request is finished”, so it waits a fixed time and then judges whatever it has. That time is decision_wait, measured from the trace’s first span. Until the March incident of Section 4.5 the wait was the default 30 s. That afternoon’s slowest checkouts spent a median of 38 s waiting for a bank’s callback, so they were judged before they had finished: the sampler saw a trace that was neither failed nor yet slow, and some of those traces were kept only in fragments while others were lost entirely. The traces that mattered most were the ones the sampler was worst placed to judge. The wait is now 60 s, which outlasts the callbacks, and it costs memory in proportion, because the gateway must now hold every in-flight trace for twice as long.

The second line, the decision cache, handles the spans that arrive after the verdict. The sampler remembers, for a large number of recent traces, whether it kept or dropped them, so that a span arriving late is kept or dropped to match the rest of its trace instead of turning up as an orphan. This helps but does not solve everything. A saga that pauses for minutes, longer than any reasonable wait, is still judged on partial evidence, which is why Section 4.3 carries the workflow ID on every one of its spans so that its pieces can be found regardless.

One more consequence of sampling is easy to forget when reading the results. A trace kept by the 1% policy stands for a hundred like it that were discarded, and a trace kept by the 10% policy stands for ten. Counts over sampled traces must therefore be weighted by sample rate. If an on-sale hour left 2,000 baseline traces showing card payments, the true number of such payments was about 200,000, and comparing that raw 2,000 with an ordinary hour’s raw count, kept at ten times the rate, would mislead badly. Errors and slow checkouts, kept at 100%, need no weighting, which is one more reason to keep them all.

All of this has a price, and Encore treats the price as a design input. Its telemetry costs about €40,000 a month, roughly 2 cents of every ticket’s 12 cents of variable cost. Retention is chosen signal by signal. Wide events, the one-record-per-request telemetry that Section 4.5 puts to work, are kept unsampled for 14 days, a period that also limits how long fan identifiers are retained. Sampled traces are kept for 30 days. Metrics are kept for 13 months, which is long enough to compare this year’s festival on-sale with last year’s. Each period answers a question about how far back someone will need to look, and for what.

Section 4.1 observed that aggregation is a decision about which questions will never be asked again. Sampling is the same decision made with a budget. The policy above is an attempt to make that decision well: to keep every row Snow would have wanted, the errors, the slow checkouts and the odd cases at the edges, and to keep only a sample of the ordinary requests in which nothing went wrong. Figure 4.1 draws the whole pipeline, from the agents to the gateway tier to the stores.

Figure 4.1 — Encore’s telemetry pipeline. Agents only batch and forward; the gateway tier redacts, routes and, in the tail sampler, decides what the bill will be.

Recap. Signals earn their value through the context they share, and two inexpensive mechanisms make that context concrete: trace IDs written into every log line, and exemplars that link a metric’s spike to a real request. Semantic conventions, extended by a registry that one team owns and CI enforces, let one team’s query work against another team’s service. Tail sampling keeps the interesting traces and discards the ordinary ones, but only if its policies combine as intended, it waits long enough for slow requests to finish, and its counts are weighted by sample rate afterward.

Exercise 4.2. Pick three services you depend on and find what each calls the HTTP status, the tenant or customer, and the deployed version. How many distinct names did you find for each, and which incident query would have failed, or quietly returned too little, because of them?

4.3 Distributed Tracing in Practice

One fan’s purchase

The braid of Section 4.2 needs a thread running through it, and that thread is the trace. A trace is the record of one request’s journey through a system, kept as a tree of spans, where each span is a timed unit of work with a name, a parent and a set of attributes. The root span is the request as the outside world saw it; its children are the calls made on its behalf, and their children the calls those made in turn. The idea is older than the tooling most of us now use for it. Google’s Dapper paper of 2010 described tracing at scale, and Zipkin, Jaeger and now OpenTelemetry brought it into open source, where ordinary teams could use it. It is worth pausing on how modest the idea is: a trace is a careful answer to the question “what happened to this request?”, and in a system of many services that question was once answerable only by guesswork and a great deal of searching through logs. Here is one fan’s purchase at Encore, as its trace records it:

Figure 4.2 — One fan’s purchase as a trace. The PSP takes 640 ms of its 800 ms share of the 3 s checkout budget; ticket issuance runs asynchronously and is joined to the purchase by a span link.

Read from top to bottom, the table tells a small story. The request arrives at the API gateway carrying the concert and the tenant it concerns. The Sale Gate confirms in 12 ms that the fan’s admission token is valid. Bot Screening scores the request and lets it through. Seat Inventory, on shard 2, places a hold on two seats. Then the purchase saga in Orders takes over, and most of its time goes to a single call: the PSP authorizing the card, which uses 640 ms of the 800 ms that the 3 s checkout budget sets aside for it. Once payment is authorized, Seat Inventory sells the held seats and appends SeatSold. The last row stands apart from the others, and we will come back to it. Drawn to scale, as in Figure 4.3, the trace shows at a glance where the checkout’s 1,140 ms went.

Figure 4.3 — The trace of Figure 4.2 as a waterfall, against the 3 s budget. The PSP call is most of the checkout; ticket issuance, four seconds later, hangs off the purchase by a link.

A trace like this looks as though the system simply produced it, the way a camera produces a photograph. It did not. Three things in it took deliberate work, and each is a place where real traces commonly go wrong: the context had to be passed from hop to hop, someone had to decide what deserved a span, and the asynchronous work had to be joined to the purchase without distorting it. We take them in turn.

Context propagation

A trace exists only because each hop passes the trace context, the trace’s ID and the ID of the current span, on to the next. When Orders calls out, the outgoing request carries those few bytes; the receiving side reads them and records its own span as a child of the caller’s. Over HTTP and gRPC the context travels in the traceparent header, which the World Wide Web Consortium (W3C) standardized as Trace Context, and OpenTelemetry reads and writes that header automatically.

The picture that comes to mind is a relay race, in which a baton passes from hand to hand and a dropped baton ends the run. It is a useful picture, but it misleads about where the baton gets dropped. Between services the handover is the easy part, because the libraries do it. The trouble is inside a service, and there, unlike on a running track, a dropped baton makes no sound. If Orders hands work to a fresh thread and that code makes an outbound call without passing the context along, the call starts a brand-new trace. The original trace ends early, a second, orphaned trace begins somewhere else, and nothing records that the two belong together: the trace breaks in two. The service mesh cannot rescue us. It sees traffic entering and leaving a service, but it cannot see which inbound request caused which outbound call, and so it cannot carry context through the service on our behalf. As Yuri Shkuro, who built Jaeger at Uber, makes clear, the hard part of tracing is getting every hop, through every thread pool and callback, to carry a few bytes of context.

Encore’s answer is to make the common mistake fail early, while it is still cheap. The service template ships an integration test that fails the build if any outbound span lacks a parent, because a missing parent is exactly the symptom of context lost on a fresh thread. The test cannot reach every thread pool a team might write, so it is a net with holes in it, but it catches the common slip before production, where the first sign of it would otherwise be a broken trace in the middle of an incident.

The same context can carry baggage: key-value pairs that ride along with the trace from hop to hop, so that a value known at the entrance is available deep inside the system without every service passing it on explicitly. Encore puts encore.event_id and encore.tenant_id in baggage at the gateway. Baggage, however, is only carried, not recorded; OpenTelemetry does not copy it onto spans by itself, so a span far downstream would know which concert it served and still not say so. The template therefore includes a baggage span processor that copies those two values onto every span, and that is what lets us ask, later, for every span belonging to one event or one tenant. Baggage also brings a responsibility. Every hop can read it and nothing protects it from alteration on the way, so it holds identifiers, never secrets or personal data. And because it travels on outbound calls by default, it would otherwise leave Encore with each request to the PSP, an outside party with no need for Encore’s internal identifiers. Encore’s egress proxy strips baggage before any call leaves for the PSP.

Span design

Once context flows, the next question is what deserves a span at all. It is the tracing version of what deserves a function: a span should mark a unit of work that someone might reason about on its own, with a beginning, an end and a reason to exist. Too few spans and the trace is one long bar that explains nothing; too many and it becomes a wall of slivers in which the important one is lost.

Encore follows common practice. It records a span for every network call, every queue publish and consume, every lock or lease, and every decision that changes the request’s path: a bot verdict, a cache miss, a retry. Those are the places where time goes and where behavior forks, so they are the places an engineer will want to see. Everything else becomes an attribute on an existing span, and the distinction matters because the two do different jobs. Spans show the shape of the work and where the time went. Attributes are what we will later group by, and they are how the high-cardinality fields of Section 4.1, such as encore.event_id, encore.shard and encore.payment_method, turn one trace into evidence that can be compared with a million others. When someone asks during an incident whether slow checkouts cluster on one shard or one payment method, it is the attributes that answer.

Queues, sagas and asynchronous work

Now the last row of Figure 4.2, which is the hard case. The SeatSold event leaves Seat Inventory on the ticket-sales.v2 stream, and Notification consumes it four seconds later to issue the ticket. How should that work appear? Both obvious answers are wrong. Recorded as a child of the checkout span, ticket.issue would stretch the trace to about five seconds, and the checkout would appear to take that long, when the fan in fact finished paying in a little over one; a latency measure that the checkout’s service level objective (SLO) depends on would be falsified by work done after the fan had left. Recorded as an unconnected trace, it would keep the latency honest but break the story: when a fan tells Support “I paid and never got my ticket”, nobody could follow that fan’s purchase into the delivery that failed.

OpenTelemetry’s answer is the span link, a reference from one span to another that records a causal connection without claiming the other as its parent. The producer writes the trace context into the event’s headers. The consumer, when it picks the event up, starts its own trace, whose first span links to the producer’s span instead of naming it as a parent. Each trace keeps its own honest duration, and the link lets us walk from one to the other.

Air travel offers a fair picture of the arrangement. A checked bag on a connecting journey carries a tag tied to the passenger’s booking, yet it travels on its own schedule, and nobody would add its hours in a sorting hall to the passenger’s flight time. When the bag goes missing, the tag is what traces it back to its owner. The picture breaks in one instructive place: a baggage system is built to follow its tags, whereas tracing backends differ in how well they let us walk a link. The same device joins a fan’s minutes in the waiting room to the purchase that follows, which is why the admission.verify row of Figure 4.2 points to the queue wait in a linked trace rather than counting it in the checkout. Links are stable in the specification, so instrumenting with them is safe, though how pleasant they are to use depends on the backend.

Sagas raise a related difficulty. A Temporal saga may pause for a PSP callback and resume minutes later on a different worker, and context held only in memory does not survive that journey. Encore therefore records each activity as its own span through Temporal’s tracing interceptors and carries the workflow ID on all of them. The reason is the tail sampler of Section 4.2, which judges a trace once its decision wait has passed: a saga that resumes after that point arrives to find its trace already judged, perhaps kept in fragments. The workflow ID is a second way in. Even when the trace has been judged early, a query for that ID gathers every activity of the purchase in one place.

Recap. A trace is only as complete as its weakest propagation point, which usually lies inside a service rather than between services, so Encore tests propagation in the build, copies baggage onto spans deliberately and strips it at the edge. Spans mark the units of work where time is spent and paths fork, and attributes carry the detail we will later query. Span links join asynchronous hops, from the waiting room to the purchase and from the purchase to the ticket, while keeping each trace’s latency honest, and a workflow ID keeps a long-running saga findable when its trace is judged early.

Exercise 4.3. Follow one real request in a system you know, from its entry point to its last asynchronous consequence. Mark each place where the trace breaks or misstates how long the work took, and name the fix for each: propagation, a span link or an attribute.

4.4 Dashboards, Alerts, and Paging Discipline

The previous sections were about gathering telemetry well: shared context, sensible sampling, traces that hold together across every hop. Gathering is only half of the work, though. Telemetry helps nobody until it reaches a person, and it reaches people in two ways. A dashboard waits for someone to come and look; an alert goes and finds them, sometimes in the middle of the night. Both deserve as much care as the pipeline behind them, and in most organizations they receive far less.

Dashboards that argue

In 1858 Florence Nightingale drew the Crimean War’s British dead as wedge-shaped monthly diagrams in which preventable disease dwarfed wounds. She did not ask her readers to work through columns of mortality figures. The shape of the diagram made the argument for them: far more soldiers were dying of disease that could have been prevented than of wounds. It was a chart with a thesis, built for people who would never read the tables, and it belongs to the same decade, and the same spirit, as Snow’s map.

A good dashboard has a thesis too, and I have come to believe that this is the single most useful test of one: can you say, in a sentence, what question it answers? Encore’s on-sale dashboard answers one question, “are fans getting through and buying?” Everything on its top row follows from that question, and each panel shows a measurement beside the target it should be judged against:

• admissions per second, against the admission rate the Gate is configured to allow;
• checkout success, against the on-sale-window SLO, which asks that 99% of admitted fans finish without an error page in the sale’s first 60 minutes;
• p99 checkout latency, against the 3 s budget fans are promised.

An engineer who glances at that row during an on-sale learns, in a few seconds, whether the sale is healthy and, if it is not, which part of the promise is breaking. Only beneath that row does the dashboard descend into detail, and it does so with two well-known recipes that divide the work cleanly. For each service it shows rate, errors and duration, which is Tom Wilkie’s RED method: how many requests arrive, how many fail and how long they take. For each resource, such as a database’s connection pool or a node’s CPU, it shows utilization, saturation and errors, which is Brendan Gregg’s USE method: how busy the resource is, how much work is queued waiting for it, and whether it is failing. RED describes the experience of whoever calls a service; USE describes the strain on whatever a service consumes. Read together, they let the engineer move from “checkout is slow” to “this service is slow” to “because this pool is saturated” without leaving the page.

Most dashboards are not built like this. They accumulate. Each past incident leaves behind a few graphs that someone added in the heat of the moment, nobody removes them afterward, and over the years the result becomes a wall of panels with no question and no owner, a museum of old emergencies. Such a wall does harm during a new incident, because it invites the eye to wander among dozens of lines, any of which might be moving for reasons that have nothing to do with today’s trouble. Encore therefore asks each dashboard to state its question and name its owner, and it deletes dashboards nobody has opened for 90 days. The 90 is Encore’s own choice of number, not an industry standard, and there is one exception: a dashboard linked from a runbook survives. A runbook may be needed rarely but urgently, and a dashboard that sits unopened for a season can still be exactly what someone needs at 3 a.m. during the one incident it was built for.

What deserves a page

Dashboards wait to be consulted. Alerts do not, and that difference raises the stakes. The mechanics of alerting on burn rate, the speed at which a service is spending its error budget, belong with the objectives themselves, and the Resilience book builds them in its chapter on service level objectives; what remains here is the discipline of deciding what is allowed to interrupt a person at all. An alert is a request for a human being’s attention, which is the one resource in the system that no autoscaler can add. A page, the kind of alert that summons someone immediately, whatever the hour, should therefore pass three tests: it is urgent, it affects users or is about to, and it needs human judgment to resolve. If it fails the first test, it can wait for morning. If it fails the second, nobody is being harmed. If it fails the third, a machine should be handling it.

The Architecture book stated the rule as paging on symptoms, with narrow exceptions. A symptom is something users feel, such as failed or slow checkouts; a cause is a condition inside the system that may or may not produce a symptom, such as a busy CPU or a lost replica. Symptoms make the better default, because they are what we actually care about and because one symptom covers a hundred possible causes, including the ones nobody thought to alert on. The narrow exception is this: a cause earns a page when it predicts a symptom more reliably, and earlier, than the symptom itself could be measured. In that case, waiting for the symptom means waiting until the harm has already begun.

Three examples at Encore show where the line falls.

The seat ledger losing one of its three locations during an on-sale qualifies for a page. Each shard of the ledger keeps five voting replicas spread across three locations, and it can commit a seat write only while a majority, three of the five, can agree. Losing any one location still leaves a majority, so the ledger keeps committing writes and a symptom-based alert may well stay silent. But the margin is gone: if the lost location held two replicas, only three remain, and one more loss would leave no majority to commit a seat write, and selling would stop. The cause predicts the symptom with near certainty and well before fans could feel it, which is exactly what the exception is for.

A certificate within a day of expiry qualifies too, but only as a backstop. Expiry is entirely predictable, so the real alert is a ticket three weeks earlier, raised during working hours when someone can renew the certificate calmly. If the page fires, it means that ticket was ignored; the page exists to catch the failure of the first, gentler alert.

CPU at 85%, by contrast, predicts no symptom on its own. A service may run at 85% all afternoon and serve every fan perfectly well, or degrade at 60% for some unrelated reason. Because the number does not reliably foretell harm, it does not page. It belongs on the dashboard, where it can help answer a question once someone is already asking one.

Figure 4.4 sorts Encore’s signals by where they go. Reading down the destination column is instructive in itself, because only some of the rows wake anyone.

Figure 4.4 — Where signals go. Only the pages wake anyone; the rest wait for daylight or for a question.

The burn-rate figures in the first row are worth translating. A burn rate of 14.4 means the checkout budget is being spent 14.4 times faster than the pace that would just use it up over the 28-day window; if that continued, the whole window’s budget would be gone in under two days. A burn rate of 6 would exhaust it in under five. Both are emergencies, if slow ones, and both page. The slow burn in the second row differs in kind: spending 40% of the seat-map budget in 10 days is a pace that runs out a few days before the window ends, which deserves a team’s attention this week but not anyone’s sleep tonight. The business anomaly in the fifth row is real information, but its right audience is the team that owns the event and its forecast, in their channel, and not an on-call engineer who can do nothing about weak demand for a show.

The last row guards against failures in observability itself, which are the hardest kind for observability to see. Every other alert in the table depends on telemetry arriving. A Collector that has quietly stopped forwarding raises no errors of its own; the dashboards downstream simply go flat, and flat dashboards look very much like health. A system that has gone blind can look exactly like a system that is calm. Encore guards against this in two ways. The first is a device the Prometheus community calls a dead man’s switch: an alert that is designed to fire all the time, so that it continuously delivers a heartbeat to the paging service, which is configured to raise the alarm when the heartbeat stops. The logic is the inverse of an ordinary alert, since here silence is the signal. Because the paging service sits outside the telemetry pipeline, it can notice a silence the pipeline cannot report about itself, which is why the table says “from outside.” The second guard is narrower: Encore alerts on the Collectors’ own counts of refused and dropped spans, so that a pipeline that is still running but losing data is caught before it goes entirely dark.

Alert fatigue

Every page that fails the three tests carries a cost, and lost sleep is only the visible part of it. Hospitals learned this cost before software did. Monitors that beep at every transient reading teach nurses, quite rationally, that beeps do not mean danger, and once that lesson is learned, the beep that does mean danger is met with the same tired glance as all the others. The Joint Commission, which accredits American hospitals, issued a formal warning about alarm fatigue in 2013, citing estimates that 85% to 99% of alarms needed no clinical intervention. An engineer paged nightly for causes that resolve themselves learns the same lesson for the same good reasons, because the pager has taught them, through repeated experience, that it is usually wrong. The danger is not that the engineer becomes careless. It is that the engineer becomes accurate about a pager that no longer deserves trust.

The parallel breaks at an instructive point, and the break favors software. A hospital cannot delete the monitor on a patient’s heart, because the patient’s condition really can change in seconds and the monitor may be the only way to know. We can delete an alert, and nothing obliges us to keep listening to a signal that has never told us anything useful. Most teams delete far too few, partly because each alert was added for a reason someone once found persuasive, and partly because removing one feels like taking a risk while keeping it feels free. Keeping it is not free. The cost is paid in the attention of every engineer it wakes, and in the slow erosion of their trust in every other page.

When Encore’s Orders team reviewed its alerts last year, the numbers told the story plainly. Of its 130 alert rules, 40 could page someone. Over a quarter, only 11 of those 40 had led to any human action, and only two had caught something before a fan noticed. The rest had woken people for problems that resolved themselves or that nobody could act on. The team kept 12 pages, each with a link to a runbook saying what to do when it fires, and demoted the rest to tickets or dashboards. The demotions removed no information, since every demoted signal is still recorded and still visible. What changed is that those signals no longer have the right to interrupt a person.

A one-time cleanup decays, because alerts accumulate just as dashboards do. So the Reliability team now reviews every rotation’s pages monthly, and the review turns on one question: what did the person who was woken actually do? The question is deliberately about behavior rather than intention. It does not ask whether the alert might matter someday, which every alert can claim, but what happened the last time it fired. A page whose honest answer is “acknowledged it and went back to sleep” goes at the next review, because it has shown, in the most direct way available, that it does not need a human.

Recap. A dashboard should make an argument: it states the question it answers, names its owner, and is pruned when that question no longer recurs. A page must be urgent, affect users and need human judgment; predictive causes qualify when they warn more reliably and earlier than the symptom could, and one page watches the telemetry itself, since silence can look like health. Alert fatigue is treated by deletion, guided by a regular review of what the people who were woken actually did.

Exercise 4.4. Export a month of pages from a rotation you know. For each alert rule, record what the responder did when it fired. Propose deleting or demoting every rule whose honest answer was “nothing,” and estimate how many hours of sleep the change would return to the rotation.

4.5 Asking Questions — High Cardinality, Exploration, and AI-Assisted Queries

Wide events

Section 4.1 argued that observability rests on telemetry that is wide, high in cardinality and explorable. Before we use such telemetry, it is worth asking what it looks like when it lands on disk, because the shape of a record decides which questions can be put to it later. In its simplest form it is the wide event: one structured record per request per service, carrying every field that might matter to someone investigating that request, often a hundred fields or more. Stripe calls the habit canonical log lines, and the name fits. Instead of scattering a request’s story across dozens of log lines that someone must later stitch together by timestamp and guesswork, the service writes one line when the request finishes, and that line tells the whole story.

At Encore, a checkout produces such a record. It carries the fan’s country, the encore.* identifiers from the registry of Section 4.2, the payment method, the PSP route, the build, the region, the feature flags in force and the duration of each step. None of these fields is remarkable on its own. The record is valuable because they all sit side by side in one place, so that later we can ask how any one of them relates to any other without having decided in advance which relationships we would care about.

Charity Majors, Liz Fong-Jones and George Miranda, in Observability Engineering, carry the idea further. They argue that such events are the fundamental unit of observability, and that metrics are best understood as cheap aggregates derived from them: a count or a percentile computed over events, with the detail summed away. That is one school of thought, and a persuasive one, though it is not the only workable arrangement. Encore takes a middle path. It keeps metrics for the questions it asks every minute, such as whether checkout is meeting its SLO, because metrics answer those questions quickly and at little cost. It keeps wide events for the questions it has not thought of yet, which are exactly the questions a gray failure poses. The trade-off is the one Snow’s register showed us in Section 4.1. A metric is economical because it has already decided which questions will never be asked of it, and that economy is a fine bargain right up to the afternoon when one of those questions is the one we need.

The core analysis loop

Having the right data is half the matter; the other half is a disciplined way of questioning it. The same authors describe a method they call the core analysis loop. We start from the signal that something is wrong. We isolate the requests that show the problem. We compare them with the requests that do not, across every dimension at once. Then we let the dimensions that differ most suggest the next filter, narrow the selection accordingly, and repeat until the difference has been explained.

It is Snow’s method made mechanical. Snow marked each death against its address and let the cluster around the pump speak for itself; the loop marks each slow request against every field it carries and lets the fields that cluster speak. The resemblance extends to the method’s limits, and two of them deserve to be stated plainly. The first is that, like Snow’s map, the loop finds correlates. It can tell us that slow checkouts come overwhelmingly from one country or use one payment method, but someone must still work out the mechanism that connects the field to the slowness, just as Snow still had to argue how water could carry cholera. The second is that the comparison is honest only if the data counts every request fairly. Encore’s tail sampler, for instance, keeps every checkout slower than 2 s but only a tenth of the ordinary ones. In that store, the “healthy” group is crowded with checkouts that took between 2 and 3 s, which were all kept, and thin on fast ones, which mostly were not, so its makeup no longer resembles real traffic, and a comparison against it would show differences that are artifacts of the sampling policy. So the loop needs data that is either unsampled or weighted by its sample rate, which is why Encore runs it on wide events, kept unsampled for 14 days, rather than on the sampled traces of Section 4.2.

A worked case shows the loop better than any description. One March afternoon, Encore’s support team relayed that payments from the Netherlands were “hanging” after the bank page. The checkout SLO barely moved, because Dutch fans are a small share of Encore’s traffic: even if many of their checkouts were slow, the share of all checkouts finishing within 3 s hardly changed. This was a gray failure in the sense of Section 4.1, plain to the fans who suffered it and invisible to the averages that were supposed to watch over them. The incident channel’s first instinct was to blame the PSP, which had been the usual suspect since Encore’s first outage. But the psp.authorize spans looked normal, at about 180 ms, so the PSP was answering authorization requests as quickly as ever. Rather than keep arguing about the suspect, the on-call engineer ran the loop: select every checkout slower than 3 s since 13:00, and compare those with all the rest. The first thing the selection showed was a time: the slow checkouts began abruptly at 14:02. The comparison itself is Figure 4.5.

Figure 4.5 — The core analysis loop on “Dutch fans’ payments slow since 14:02”. Country was the symptom; the payment method was the mechanism; the gateway rule was the cause.

Read from top to bottom, the figure retraces the loop’s narrowing. The first row confirms what support had reported: 81% of the slow checkouts came from the Netherlands, against 6% of the others. Had the investigation stopped there, it would have ended with a statement about Dutch fans, which is a description of the victims and says nothing yet about the culprit. The second row is sharper. Almost every slow checkout, 97% of them, used bank redirect, a method only 14% of other checkouts used, and that explains the first row, since bank redirect is how most Dutch fans pay. The third row says where the time went. Slow checkouts spent a median of 38 s in the saga step that waits for the payment to be confirmed, against 2.1 s for the rest, so the delay lay in waiting for news from the PSP rather than in asking the PSP anything. The fourth row names the reason the news was late: 95% of the slow checkouts had passed through a gateway rule called psp-callback-ratelimit, which almost no healthy checkout had touched.

The mechanism, once seen, is easy to follow. With bank redirect the fan leaves Encore for their bank’s page, approves the payment there, and the PSP later confirms it by calling back to Encore. At 14:02 the Bot Screening team had shipped a reasonable change to the API gateway: a per-source rate limit on unauthenticated POSTs, the kind of request a bot sends in bulk. The PSP’s callbacks, however, arrive from a handful of addresses, so to the gateway they looked like a few very busy sources sending unauthenticated POSTs. They were throttled, the PSP retried them with backoff, and they were delivered tens of seconds late while the fans stared at a hanging page. The fix, an exemption for the PSP’s signed callbacks, took four minutes. Finding the cause took 25, and nearly all of those were spent on the theory everyone already held. That ratio is the real lesson of the afternoon: the slow part of an investigation is seldom the repair, and it is usually the time spent defending the first explanation. A rule meant to stop bots had begun to punish ordinary retry behavior, an interaction this book will meet again.

It would be tempting to file this case under “high cardinality saves the day”, and that would be a misreading. High cardinality alone did not solve it. A few dozen countries, a handful of payment methods and a few dozen gateway rules would all fit comfortably as metric labels. What no metric had was the combination. To see this failure on a dashboard, someone would have had to build, in advance, a chart of checkout latency broken down by country, payment method and gateway rule together, and the number of such combinations grows by multiplication with every field added, so nobody builds them all. The wide event made the combination available after the fact, at the moment it was needed. Notice too that the decisive field existed at all only because someone had decided earlier that every policy decision the gateway makes deserved an attribute on the event. Instrumentation is a bet placed in advance on what the future will want to know, and this one paid. Finally, the truly high-cardinality fields, tenant, shard and version, showed no difference between slow and healthy checkouts, and that absence was evidence too. It ruled out a bad deploy and a hot shard at a glance, much as the spared workhouse and brewery strengthened Snow’s case by being exceptions that fit it.

Asking Cue

If the loop is mechanical, a natural question follows: can a machine run it for us? Assistants that query telemetry in plain language are young practice, and the evidence so far favors a narrow use. The best published production account is Honeycomb’s, of the Query Assistant it shipped in 2023. The assistant turns a question asked in English into a query, and the user sees that query, can edit it, and runs it. Honeycomb reported wide adoption and more complex querying among its users, a correlation it was careful not to oversell, and it wrote just as frankly about the failures, from schemas too large to fit in the model’s context to prompt injection. What makes this use tractable is that its output can be checked: the query is on the screen, and a person who knows the system can see whether it asks the right thing.

Diagnosis is harder, because its output is a conclusion, and a conclusion is much harder to check than a query. Meta reported in 2024 that its model for ranking the code change most likely behind an incident put the right change in its top five about 42% of the time. Put the other way, in more than half of incidents the culprit was not among the five suggestions at all. Meta’s response was telling: it withholds low-confidence suggestions and requires each suggestion to be explainable, so that an engineer can judge the reasoning and not merely the answer.

Encore’s own experience points the same way. Cue, the operations agent introduced in Section 3.5, is read-only by default and queries telemetry through the company’s Model Context Protocol (MCP) server, which exposes Encore’s systems to it as a set of tools. After the March incident, the team replayed the investigation with Cue and asked it the same question twice, in two different forms. Asked “why are Dutch payments slow since 14:02?”, Cue answered in seconds, fluently, and wrongly. It found two past postmortems that blamed the PSP and concluded that the PSP was degraded for Dutch banks, which was the prevailing theory, now with citations. Asked instead to “compare checkouts slower than 3 s since 13:00 with the rest, across all attributes,” it ran the core analysis loop faster than any human could and returned Figure 4.5 almost exactly. The difference between the two answers lies in the questions. The first asked for a conclusion, and the nearest conclusions in Cue’s reach were the team’s own earlier ones. The second asked for a procedure, and the procedure did not care what anyone already believed.

A confident wrong answer is more expensive than a slow right one, because it ends the search. A model that reasons from past incidents tends to reproduce the team’s prevailing theory at exactly the moment someone needs to question it, as Snow questioned miasma; Encore has seen this tendency in its replays, and Meta’s caution anticipates it. Encore’s rule is therefore to show the query and not to trust the diagnosis. Every answer from Cue must carry the query that produced it, so that a human can rerun it and see the data for themselves, and an answer without a query is treated as a rumor: possibly true, worth checking, never acted upon as it stands.

The other limit is the one Honeycomb met, and it is about vocabulary. In the replay, Cue at first missed the gateway rule entirely, because the gateway emitted the field as policy while the registry named it encore.gateway.rule. Cue asked for the registry’s name, found nothing under it, and moved on. A human at 3 a.m. works around such a slip, glancing at the raw event and recognizing the field by its contents whatever it is called; an assistant that trusts the schema cannot. The remedy was to remove the slip at its source rather than to teach Cue to guess: the registry check of Section 4.2 now fails the build when a service emits a field under a name the registry does not know. Consistent naming, which looked like housekeeping when we met it in Section 4.2, turns out to be what makes telemetry legible to machines as well as to tired people.

Used this way, an assistant is a tireless research clerk, of the kind Whitehead might have welcomed when he was knocking on doors around Broad Street. The analogy holds for the legwork: the clerk gathers and arranges the evidence faster than we can. It breaks at the point that matters most, because Whitehead was more than a clerk; he went in to disprove Snow and came out having found the source, which means he was still doing his own thinking all along. Choosing which doors to knock on, and being willing to be surprised by what is behind them, remains the engineer’s work. Chapter 5 shows Cue in a live incident.

Recap. Wide events let us compare failing requests with healthy ones on every dimension at once, which is the core analysis loop and Snow’s method in one. The usual obstacle is the prevailing theory, and AI assistants tend to repeat it fluently, because they reason from the team’s own past conclusions. The practice is young, and the evidence supports assistants that write and show queries over consistently named telemetry far more than assistants that offer diagnoses.

Exercise 4.5. Take a recent investigation and rerun it as a comparison: the failing requests against all the rest, across every attribute you have. Which attribute would have found the cause soonest, and was it present, consistently named and kept long enough for you to query it?

Kata: The Charging Network

Your brief: “Lumen.” Lumen runs 18,000 electric-vehicle chargers in six countries. Drivers start and pay for sessions through Lumen’s app, drivers from partner networks charge on Lumen’s chargers through roaming agreements, and payments go through two PSPs. A single charging session passes through four stages (authorization, the charger itself, which talks to Lumen over a cellular link, metering, and billing) and can last for hours, so the gap between a driver’s first tap and the final invoice is measured in hours rather than seconds. Lumen has 35 engineers. It spends €70,000 a month on telemetry, nearly twice Encore’s bill, and two-thirds of that money goes on logs, most of which nobody ever reads. It has 600 alert rules, which page the on-call rotation about 90 times a week: roughly 13 pages a day, or one every couple of hours around the clock. Even so, more than half of failed sessions are first reported by drivers on social media rather than by any of those alerts. And one roaming partner has threatened to leave over authorization failures that nobody at Lumen can explain, because Lumen’s dashboards show the authorization service as healthy throughout.

Read the brief twice before you begin, because each of its symptoms points at a different section of this chapter. The money spent on unread logs is a question of what to keep, which Section 4.2 answered with deliberate sampling and retention. The sessions that last for hours are a tracing problem, the asynchronous kind of Section 4.3, and they are also a sampling problem, since a tail sampler that decides too early judges a session before it has finished, as Encore discovered in March. The 90 weekly pages, set beside the drivers who find failures first, are the alert fatigue of Section 4.4 in its familiar form: a great deal of noise and too little signal. And the roaming partner’s complaint, with dashboards showing health while a customer reports failure, is exactly the unknown question that Section 4.5’s wide events and core analysis loop were built to answer. Lumen’s trouble is not a shortage of telemetry. It has plenty; what it lacks is telemetry shaped so that the questions it actually faces can be asked of it.

Deliverables:

1. Attribute registry — the 15 attributes every span and event must carry, among them the charger, the site, the charger’s firmware version, the roaming partner and the country, aligned to the OpenTelemetry semantic conventions wherever a convention exists. Say who owns the registry and how a new attribute is added to it, because a vocabulary that nobody owns drifts back into three names for the same thing.
2. Trace design — one charging session traced from the driver’s tap in the app to the invoice. The hops that do not happen within one request (the charger reporting meter readings over cellular, billing running long after the session ends) should be joined by span links, so that the trace stays followable without pretending the session was one long request. Mark the propagation points most likely to break, such as the cellular link to the charger and the hand-off to a roaming partner’s systems, and say how you would test each.
3. Pipeline and sampling policy — the Collector topology, meaning which tiers exist and what each one does; tail-sampling policies that combine as intended, remembering that the sampler keeps a trace if any policy matches; a decision wait that respects sessions lasting hours, or an explicit argument for why some parts of a session should be judged separately; redaction of driver data before it reaches storage; and a retention plan that cuts telemetry spend by a third, from €70,000 to about €47,000 a month. For each cut, name the questions it makes unanswerable, so that the saving is a choice Lumen makes knowingly.
4. Paging set — at most ten pages, each tied to an SLO burn or to a predictive cause that warns earlier than the symptom could, plus a telemetry heartbeat that pages when the observability pipeline itself goes quiet. List the rules you deleted or demoted from the original 600 and the reason for each group.
5. Investigation script — the core analysis loop applied to the roaming partner’s complaint. Say which failing sessions you would isolate, which healthy sessions you would compare them with, which dimensions you would compare across, and why you expect those dimensions to separate the two groups.
6. Assistant policy — what an AI query assistant may do with Lumen’s telemetry and what it may not, and what evidence each of its answers must show before an engineer acts on it.

Rubric:

A word on the first row, because it is the test the other five serve. The question it poses is the one the roaming partner is, in effect, already asking, and Lumen today cannot answer it at all: the authorization service reports success in aggregate, while the failures hide in a particular combination of firmware and partner that no dashboard was built to show. If your registry, your traces and your sampling policy together make that question a single query, you have given Lumen the ability to see. If the question still needs three teams and an afternoon, the remaining deliverables are decoration, however tidy they look.

Where you now stand

You can now design a system to be observable. That means wide events described in an owned vocabulary, so that the same thing has the same name in every service; traces that stay whole across queues and sagas, joined by span links where work leaves the request that started it; a pipeline that samples with intent, keeping every failure, and that watches its own health; dashboards that make an argument and pages that respect the sleep of the people they wake. You can run the core analysis loop yourself, comparing what failed with what did not until the differences point to a cause, and you can judge an AI assistant that runs the loop on your behalf by whether it shows its work. All of that was learned on ordinary afternoons, with time to think. The real test of observability comes on the morning when a million people are watching, the dashboards disagree with the support queue, and a team of people who were drinking coffee a minute ago must agree on what is happening before they can do anything about it. Chapter 5 takes us into that morning: the Vesper Hale on-sale, as it happened, and the craft of incident response.

References

• Charity Majors, Liz Fong-Jones, George Miranda — Observability Engineering, 2nd ed.. O’Reilly, 2026.
• Peng Huang et al. — Gray Failure: The Achilles’ Heel of Cloud-Scale Systems. HotOS, 2017 — free online.
• OpenTelemetry Authors — Tail Sampling Processor. OpenTelemetry Collector Contrib — free online.
• OpenTelemetry Authors — OpenTelemetry Semantic Conventions. OpenTelemetry project — free online.
• Yuri Shkuro — Mastering Distributed Tracing. Packt, 2019.
• W3C — Trace Context. W3C Recommendation — free online.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering, chapter 6. O’Reilly/Google, 2016 — free online.
• Phillip Carter — So We Shipped an AI Product. Did It Work?. Honeycomb, 2023 — free online.
• Engineering at Meta — Leveraging AI for Efficient Incident Response. Meta, 2024 — free online.
• Cindy Sridharan — Distributed Systems Observability. O’Reilly, 2018.
• Benjamin H. Sigelman et al. — Dapper, a Large-Scale Distributed Systems Tracing Infrastructure. Google Technical Report, 2010 — free online.
• The Joint Commission — Sentinel Event Alert 50: Medical Device Alarm Safety in Hospitals. 2013 — free online.
• John Snow — On the Mode of Communication of Cholera, 2nd ed.. John Churchill, 1855 — free online.