Operating Production Systems in the AI & Cloud Era

← the whole book on one page

Chapter 6: Learning from Incidents — Postmortems, Just Culture, and Drift

“For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.” — Richard Feynman

On the evening of 27 January 1986, engineers at Morton Thiokol joined a teleconference with NASA managers to argue that Challenger should not launch the next morning. Thiokol was the contractor that built the space shuttle’s solid rocket boosters, and the engineers’ worry was a small part with a large job. Each booster’s field joints were sealed by rubber O-rings. A rubber ring seals by springing back into the gap when pressure reaches it, and cold rubber springs back slowly. The forecast for the morning was far colder than any launch so far. The coldest earlier launch, in January 1985 at about 53°F (12°C), had shown more blow-by than any flight before it, which is to say that hot gas had slipped past a primary ring before the seal closed. Reasoning from that flight, the engineers recommended against launching below that temperature. NASA’s managers challenged the data, and the data was in fact thin. Thiokol’s managers conferred off the line and came back recommending launch. On 28 January, at about 36°F (2°C), Challenger lifted off; 73 seconds later it broke apart, and the seven people aboard were killed.

The first account of the disaster was a story of schedule pressure and rules bent, and it was not wrong as far as it went: the Rogers Commission did find that launch constraints had been waived without being reported upward. Diane Vaughan’s The Challenger Launch Decision (1996) told a more unsettling story, one that begins years before that cold night. O-ring erosion had been seen as early as the second shuttle flight, in 1981. Each time it appeared, engineers found a reason the margin still held and accepted the next flight as an “acceptable risk”, and they did so through exactly the procedures NASA had built for making such judgments. Each acceptance then became precedent for the next. A flight that showed some erosion and came home safely became evidence that some erosion was tolerable, and so the next case was judged against that new baseline, not against the expectation the rings had started with. By January 1986 the burden of proof had quietly turned over. Where a flight had once needed to be shown safe, the engineers were now asked to prove it unsafe, and with the data they had, they could not.

Vaughan called this the normalization of deviance: the gradual process by which a departure from an expected standard, repeated without harm, comes to be treated as the standard itself. What makes her account unsettling is where she places the failure. In her reading nobody broke the rules in any sense that mattered; the rules were how the deviance became normal. The reviews and sign-offs worked as they were designed to work, and what they produced, one reasonable decision at a time, was a launch that should not have happened. When Columbia was lost in 2003, the accident board drew explicitly on her work and found the same pattern a second time, seventeen years after the first. I have come to believe this is the most important lesson the history of accidents holds for people who run systems. An organization can drift toward danger while every person in it behaves sensibly, and the drift is hard to see from inside, because each step is small and each step was survived.

We already have the vocabulary for this. Chapter 2 introduced the ideas this chapter now puts to work, from Reason’s latent conditions to Leveson’s System-Theoretic Accident Model and Processes (STAMP). Chapter 5 followed Encore through the Vesper Hale on-sale as it happened: 47 minutes of degraded checkout, about 30,000 real fans blocked as bots, and not one seat sold twice. With the incident over, the slower work begins. That work is learning from what happened without punishing anyone and without excusing anyone, and then making the learning stick, so that it changes how Encore operates and not merely what Encore says about itself. It also means looking honestly at a slow change that set the stage for the incident. Encore’s change freeze, the rule that bars routine changes to the purchase path around a big on-sale, had been eroding for a year, one reasonable exception at a time, very much as the O-ring margins had. The rest of this chapter is about how to review an incident so that people tell the truth, how to see that kind of drift, and what to change once you have seen it.

6.1 Blameless, Not Accountability-Free

On the Monday after the on-sale, the first question in the leadership channel was “who approved the admission change?” It was a natural question to ask, and it was the wrong one to ask first. Its answer, a name, explains almost nothing about why the morning went as it did. The admission rate had been raised from 1,200 to 1,500 fans/s on the strength of a load test that ran against a warm seat-map cache, and the review process had accepted that warm-cache test as evidence of headroom. Any engineer on that team, holding that test result, would have approved the same change. The trouble lay in the process that let such a test count as proof, and a question that ends in a person’s name stops just short of the process.

That observation is the starting point for what the industry calls a blameless review. A blameless postmortem is a review in which the people involved give a full account of what they did and why, on the understanding that the account will be used to improve the system and not to punish them. The idea has a short lineage worth knowing. It grew from Sidney Dekker’s work on just culture; John Allspaw’s 2012 post on Etsy’s engineering blog carried it into web operations; and Google’s SRE book made it the default for a generation of engineers, adding that leaders must be seen to reward good postmortems, since a policy nobody visibly honors is soon not believed. The reasoning behind all of it is informational before it is moral. An organization learns only what its people tell it, and people tell only what is safe to tell. Punish one candid account and the next will be shorter, vaguer and carefully lawyered, and the organization will have bought a feeling of justice at the price of its own eyesight.

Blameless is often misheard as “nobody is accountable”, and the misreading does real damage, because it lets critics dismiss the practice as softness and lets careless defenders practice it as avoidance. Dekker’s Restorative Just Culture (2025, the fourth edition of his Just Culture) answers the confusion by contrasting two responses to harm. A retributive just culture asks which rule was broken, by whom, and what the consequence should be. A restorative just culture asks who was hurt, what they need, and whose obligation it is to meet that need. Both take accountability seriously; they differ in what they ask the accountable to do. The retributive form asks a person to bear a penalty. The restorative form asks for an account and for repair, and “account” is meant literally: to be accountable is to be able and willing to explain what happened, and then to help put it right. The framing comes from healthcare, where its best-documented adopter, the Mersey Care NHS trust in England, reported fewer suspensions and lower costs. In software the word “blameless” is common, but the restorative questions, asked by name, are still rare.

Encore asked them explicitly, one affected group at a time, and the answers turned a principle into a list of obligations.

The fans were hurt first. About 30,000 real fans were blocked outright as bots, and roughly 9% of the fans who were admitted hit errors at least once. What they needed was their place back, some recompense for a spoiled morning, and the truth about what had happened. Encore restored blocked fans’ queue positions during the incident itself, gave every blocked fan early access to further tour dates, and published a plain public account within the week. A public report of that kind is restorative work done in the open: it is the same act of giving an account, aimed outward at the people who were harmed. Cloud providers have made such reports routine. AWS published a post-event summary within days of its us-east-1 failure of October 2025, and Cloudflare explained its November 2025 outage in detail the same day. Encore’s other affected parties, the organizers and the artist’s management, received the same account, earlier and fuller, since they had their own audiences to answer to.

The engineers were hurt too, and they are the group a retributive culture forgets. The On-Sale engineer who proposed the admission change spent the weekend replaying it. Medicine has long recognized this kind of injury. In 2000 the physician Albert Wu called such people second victims: clinicians harmed by their own involvement in a patient’s harm. The term has been contested since; patient advocates have argued that it casts the harm as bad luck, and many hospitals now avoid it. But the practice Wu asked for, early support for the clinician, has become standard, and the practice is the part that matters. What the On-Sale engineer needed was to hear, early and from leadership, that the review would be about the system. Ask the third restorative question, whose obligation that was, and the answer is plain: it fell on the organization, and silence from leadership over that weekend would itself have been a failure to meet it.

None of this abolishes consequences. Sabotage and sustained recklessness are rare, and both Allspaw and Dekker send them to a separate process, so that the possibility of punishment does not hang over every review. Keeping the two apart is what lets the postmortem do its job, which is to collect the truest account it can. Everything else in this chapter, from the contributing factors of Section 6.2 to the drift of the change freeze and the changes Encore made afterward, can only be as good as what people were willing to say.

Recap. Blameless review exists so that people give full accounts, because an organization can learn only from what it is told. Accountability survives in restorative form, which asks who was hurt, what they need and whose obligation it is to meet that need. At Encore, fans were answered with restored queue positions, early access and a public report, and the engineers with early, explicit support from leadership.

Exercise 6.1. Take your organization’s last significant incident. Write the three restorative questions and answer them for every group affected, including the engineers. Which need went unmet, and whose obligation was it?

6.2 Beyond Root Cause — Contributing Factors and CAST

Why root cause fails

Ask “what was the root cause of Vesper Hale?” and four candidates step forward at once. The first is the admission change, which raised the Gate’s rate from 1,200 to 1,500 fans per second on the strength of a load test run against a warm cache. The second is the Catalog deploy at 09:52, which cold-started the seat-map cache eight minutes before the doors opened. The third is the node drain at 10:02, which forced a leader failover on Seat Inventory shard 3 and about 40 seconds of rejected writes. The fourth is the bot rule, which read the fans’ frantic retries as an attack and blocked about 30,000 of them. Each of the four was necessary, and none was sufficient. Remove any one of them and the morning is very likely an ordinary on-sale: a warm cache absorbs the higher admission rate, a failover on a lightly loaded shard passes unnoticed, and retries that never start trip no rule. Chapter 2 met Richard Cook’s observation that catastrophe needs several failures together, each insufficient alone. At Vesper Hale that observation is no longer a thesis to be argued; it is simply the timeline.

If no single cause is the cause, then the phrase “root cause” must be describing something other than the incident. I think it describes the investigation. Every chain of “why” can be followed further back. Why was the load test warm? Perhaps because nobody had asked for a cold case; and why had nobody asked? Each answer opens another question, and if we keep going we arrive at budgets, staffing and the history of the company. A root cause is wherever an investigation decides to stop asking why. That decision is sometimes sensible, but it is a decision about the inquiry, and the danger is that it gets mistaken for a discovery about the world.

Two biases push us to stop early, and both are strongest in exactly the moment after an incident, when everyone knows how the story ended. Hindsight bias, which Baruch Fischhoff demonstrated in 1975, is the tendency of people who know an outcome to judge it more predictable than it was. Once we know the cache was cold, the cold cache looks like the obvious thing to have checked; on the Wednesday of the readiness review it was one of hundreds of things that might have mattered and almost never did. Hindsight’s companion is the counterfactual, a statement of what someone should have done instead: “the team should have run a cold-cache test.” The sentence is true, and it is nearly useless, because it describes a world that did not exist and says nothing about why the world that did exist made sense to the people in it. Richard Cook, David Woods and John Miller drew the distinction that matters here in 1998. A first story explains an incident as a person’s error: someone approved the change, someone missed the risk. A second story explains the pressures, resources and trade-offs that made the same act reasonable at the time it was taken. The first story points at a person, and the only change it suggests is a different person or a sterner warning. The second story points at the conditions the person worked in, and those conditions are something an organization can actually change. Only the second story says what to change.

None of this requires banning a phrase, and Encore did not try. Mainstream operators still use the words, and mostly mean the plural by them. Google’s error-budget policy still asks for an action addressing “the root cause”, and CrowdStrike’s 2024 root cause analysis of its faulty sensor update described several conditions that all had to line up before the failure could happen. What matters is the shape of the analysis, not the label on the heading. Encore’s default review makes the plural explicit by putting it under a heading of contributing factors: the conditions that together made the incident possible, each explained from the inside, as the people and systems involved saw it at the time. For most incidents, a well-written list of contributing factors, each told as a second story, is enough.

When CAST earns its cost

For its largest incidents Encore goes further, because a list, however honest, has a limitation: it presents the factors side by side, and leaves the reader to guess how they relate. Nancy Leveson’s Causal Analysis based on STAMP (CAST) is a method for writing second stories systematically, and for showing how they fit together. It rests on the STAMP model from Chapter 2, which treats safety as something enforced by a structure of controllers. In that view an incident is a failure of control: some controller, human or automated, issued a control action that was unsafe in its context. Usually the controller did so because its process model, its beliefs about the state of the system it controls, was wrong, out of date or missing a signal that would have corrected it. The method proceeds in a fixed order. CAST names the loss, the harm that actually occurred, and the hazard, the system state that made the loss possible. It then draws the control structure: who and what could act on the system, and what feedback each of them received. Of each controller it asks three questions: what did it do, what did it believe, and why was that belief reasonable at the time? Finally it steps back to examine coordination between controllers, and how the structure itself had changed over time.

The discipline in those three questions is the point. “What did it believe, and why was that reasonable?” forces the second story for every controller, the reconciler and the bot rule as much as the engineers, and it forbids the analysis from stopping at the first person who could be blamed. A ship’s master after a collision is a fair comparison. Asking who was on the bridge yields a name; asking what the radar showed, what the charts said and what the other vessel was signaling yields a picture of why a competent officer steered as they did. The comparison breaks in one respect, and the break is instructive: a ship has one bridge, while Encore’s purchase path had six controllers acting on it that morning, none of them in charge of the others.

CAST is established in aviation, defense and healthcare, and young in software. Leveson’s free CAST Handbook appeared in 2019. The best public account of its use by a large operator is Google’s, whose SRE organization has described applying STAMP methods to its incidents in recent articles and talks, while its own page still calls the work exploratory. That is a serious precedent, and an honest reader should take it as exactly that much and no more: a respected organization finds the method worth pursuing, and has not yet published evidence of what it buys.

The method is also labor-intensive. Drawing a control structure with every team involved, and interviewing each one about what it believed, takes days of skilled people’s time. Encore therefore reserves CAST for two kinds of incident: SEV1 incidents, the top severity defined in Chapter 5, and anything touching seat or money integrity. Everything else gets the lighter contributing-factors review. The trade-off is deliberate. A heavier method on every incident would teach the organization to resent reviews, and a lighter method on the worst incidents would miss exactly the structural findings that only a drawn structure reveals. Vesper Hale qualified on both counts: it was a SEV1, and although no seat was sold twice, the purchase path that protects seat integrity was the thing under strain.

Encore’s Reliability team, the platform group’s ten site reliability engineers, ran the analysis with every team that had a controller in the story. They began by naming the two things CAST asks for first. The loss was fans unable to buy tickets during an on-sale. The hazard was the purchase path receiving more load than it could serve while its capacity was reduced. That hazard statement already does useful work: it says the incident needed both halves, rising load and falling capacity, and so it points the analysis at every controller that could raise one or lower the other. Figures 6.1 and 6.2 show the control structure that resulted and what each controller believed.

Figure 6.1 — The control structure of the Vesper Hale on-sale. Every controller could act on the purchase path, and two of the feedback paths that would have warned them did not exist.

Figure 6.2 — Encore’s control structure on the morning of the Vesper Hale on-sale, read through CAST. Every controller acted on a reasonable belief; the missing column is where the changes come from.

Each row, read on its own, is a small second story, and each could have been written in a contributing-factors review. The value of the table lies in reading it the other way. Read down the last column and a pattern appears that no single row reveals: nearly every gap is missing feedback between teams. The reconciler did not know about the on-sale calendar, which the readiness review owned. The Catalog team did not know its cache was on the On-Sale team’s critical path. The bot rule did not know that the web client’s retry behavior had changed, nor could it see the errors on the purchase path that were causing the bursts it was punishing. In every case the information existed somewhere at Encore; it simply did not flow to the controller that needed it.

Diane Vaughan, whose study of Challenger opened this chapter, had a name for this condition. She called it structural secrecy: the way an organization’s division of labor lets each group see its own piece clearly while nobody sees the whole. It involves no secrets in the ordinary sense, since nobody hid anything; the secrecy is built into the structure. And it cannot be removed by abolishing the structure, because the structure is how the work gets done at all. Decomposition is how 40 teams master a system that no one person could hold in their head, and the narrow, deep view each team has of its own service is the price of that mastery. A touring production works the same way: the lighting crew, the sound crew and the riggers each know their own rig thoroughly, and the stage manager’s calls are what tie them into one show. The analogy holds as far as it shows why the cure is coordination and not a merged department. It breaks on one point: a show has a stage manager watching every cue in real time, and a software organization of 40 teams has no such person, nor could one person keep up. So the answer at Encore cannot be a single all-seeing coordinator. It has to be the feedback itself, wired back in between the controllers that need it, so that the reconciler can see the calendar and the bot rule can see the purchase path’s errors.

That is the finding a lighter review would have missed. A contributing-factors list would have named the four factors, and each would have produced its own action item, owned by its own team. It took the drawn structure to show that the four were one finding: a control structure whose controllers could not see each other.

The freeze as normalization of deviance

One row of the table deserves a closer look, because it is where the chapter’s home touchstone returns. The second row, the change freeze, is Challenger in miniature.

Begin with the rule as it was written, because the rule itself was never the problem. The freeze said: no non-emergency changes to the purchase path from two hours before an on-sale until its end. Over the year before Vesper Hale, the freeze channel recorded 41 exceptions (Figure 6.3). Every one was granted in chat, and none was followed by an incident. Alongside those explicit exceptions sat a standing one. The node-upgrade reconciler had been classed as low-risk automation and exempted from the freeze altogether, and on three earlier on-sales it had drained nodes mid-sale without harm. It had done no harm only because none of those nodes happened to host a shard leader, but from the outside that luck looked exactly like safety. Finally, the freeze’s list of purchase-path services predated Catalog’s seat-map cache. When the cache moved onto the on-sale’s critical path, nothing prompted anyone to update the list, and nobody had reason to revisit it.

Figure 6.3 — Freeze exceptions granted in chat, quarter by quarter, in the year before Vesper Hale. Each quarter looked like the last one, a little busier; only the running total, which nobody kept, shows the drift.

The figure is worth dwelling on, because it shows why nobody noticed. Anyone looking at one quarter saw a handful of exceptions, each with a sensible reason, slightly more than the quarter before. The drift was visible only in the cumulative count, and no one was keeping that count. When we set the freeze’s year beside the history of the O-rings, the two drifts pass through the same stages in the same order.

Figure 6.4 — Two drifts, side by side. In both, each step was reasonable, made by careful people, and confirmed by an outcome that happened to be good.

The comparison has limits, and it is worth stating them. Challenger cost seven lives; Vesper Hale cost a bad morning for tens of thousands of fans and a hard weekend for the engineers involved. The mechanism, however, is the same, and the mechanism is what we are here to learn. In both stories a standard was set; a departure from it was allowed once, for a good reason; the departure was then reinterpreted as falling within the standard; each harmless repetition was taken as evidence that the departure was safe; and in the end the burden of proof had turned over, so that the person raising a concern had to prove danger instead of the person seeking the exception having to prove safety.

Nobody ignored the freeze. People followed it, asked for exceptions through the channel it provided, and received them for reasons that were sound each time. That is precisely how it drifted. When a rule’s exceptions are never counted, its meaning changes without anyone deciding to change it, and the rule on paper and the rule in practice slowly become two different rules.

Recap. Root cause is a stopping point, and hindsight pushes us to stop too soon. The everyday remedy is a plural list of contributing factors, each told as a second story that explains why the act made sense at the time. CAST, young in software and costly, earns its place on the largest incidents. At Encore it showed that the four factors were one finding, structural secrecy between teams, and it showed a freeze that had drifted through the same stages as the O-ring margins.

Exercise 6.2. Build a CAST table like Figure 6.2 for the most serious incident you know, with at least four controllers, one of them automated. For each controller, write why its belief was reasonable at the time. Then read down the last column: which missing feedback path, if it had existed, would have changed the most rows?

6.3 Writing and Running the Review

Deciding what gets reviewed

Before any review is written, someone has to decide that one will be written at all, and that decision is best made long before the incident it applies to. If the rule is settled in advance, the incident simply meets it or does not. If it is left open, the choice falls to whoever is in the room on Monday morning, and the quiet pressure in that room is to review the incidents that are safe to talk about and let the awkward ones pass. Left to the moment, embarrassment chooses. Google’s SRE book lists the triggers it uses: user-visible impact past an agreed threshold, any loss of data, on-call intervention such as a rollback, a long time to resolve, and a monitoring failure, by which it means a problem that a human found before the alerts did. That last trigger deserves a moment’s attention, because it is the one teams most often forget: an incident the alerts missed is also a review of the monitoring itself, even when the harm was small.

Encore’s rules follow the same spirit and come in two weights. Every SEV1 gets a full CAST review of the kind Section 6.2 described. Any incident that spends 20% or more of an error budget, loses data, or is reported by a customer before an alert fires gets the lighter contributing-factors review. And anyone at all may ask for a review of anything, which matters more than it seems, because it means that an engineer who sensed something wrong does not first have to win an argument about whether it was serious enough.

Preparing the account

The Vesper Hale review was facilitated by a member of the Reliability team who met two conditions: she had not been in the incident channel during the on-sale, and she had no authority over anyone she would interview. Etsy’s debriefing guide and Jeli’s Howie guide both insist on these conditions, and the reasons are not hard to see. A facilitator who was in the channel already carries her own version of events and will, without meaning to, steer the interviews toward it; a facilitator who manages any of the interviewees turns every answer into a performance for the boss. Within two days of the incident she held short one-to-one interviews with Marta, who had been incident commander, with the other responders, and with each team that appears in Figure 6.1. The interviews were deliberately one to one. Group meetings tend to converge on the account of whoever speaks most confidently, and the quiet engineer who noticed something odd at 10:03 may say so only in a room of two. Speed mattered too: memory of an incident fades and reshapes itself quickly, and within a week people recall the story the team has agreed on more readily than the confusion they actually felt.

The facilitator did not start from a blank page. Cue, Encore’s operations agent, had drafted most of the timeline within an hour of recovery. Drafts like this are a young feature of incident tools, and this one was accurate, but it told only half the story. A timeline built from logs records what the system did; a review also needs a second column for what people knew, what they believed and what they were trying to do at each moment. Only the interviews could fill that column. They showed, for example, that at 10:11 Marta already suspected the admission change and was weighing a rollback, and that when Cue posted the news of the node drain at that same minute, it changed which lever she reached for first. No log records a suspicion that was weighed and set aside, and yet that is exactly the kind of fact a review needs, because it explains why the response unfolded in the order it did.

The 2017 STELLA report, from the SNAFUcatchers workshop led by David Woods, gives a precise way to say why that second column matters. Its authors draw a line across the picture of any running system. Below the line sits the machinery itself: the code, the servers, the pipelines. Above the line sit the people, and they never touch the machinery directly; they act on it only through their mental models of it, by way of screens, dashboards and commands that are themselves representations. Each person’s model is partial, and different people’s models differ. Most days the differences do no harm. Incidents are where they come to light, because the system does something that at least one person’s model said it could not do. Encore’s mismatches surfaced quickly once the interviews began. The On-Sale team learned that the node-upgrade reconciler existed at all. Bot Screening learned that the web client had been retrying failed holds automatically since the previous spring, which meant that the bursts of identical requests their velocity rule watched for were now also what an ordinary frustrated fan looked like. And the velocity rule itself turned out to be older than the policy, adopted in the spring after an error-budget dispute between On-Sale and Bot Screening, that every block must offer a challenge. Nobody had reviewed the rule since that policy arrived, so it still blocked outright. None of these facts was secret; each was known to someone. The trouble was that each lived above the line in one team’s model and nowhere else.

Reviewing the machine’s part

One controller in Figure 6.2 was not a person, and the review had to decide how to treat it. At 10:11 Cue had suggested restarting the Seat Inventory pods, a suggestion that, as Chapter 5 showed, would have forced a leader election on every shard. The review examined that suggestion exactly as it examined every other controller’s action: what did Cue do, what did it believe, and why was that belief reasonable given what it could see? No industry convention yet says that an AI agent’s part in an incident should be reviewed this way. Encore chose the approach for two reasons. The first is that CAST already treats automated controllers this way; a reconciler and an agent are both controllers with process models, and the method does not care whether the model lives in a person’s head or in software. The second is that two easy first stories were on offer, and both were tempting: “the AI was wrong” and “the human caught it”. Each is true, and each ends the inquiry before it has learned anything.

Asked from the inside, the restart suggestion made sense. Cue had drawn on what it could read: two old postmortems in which restarts had cleared errors, both of them about stateless services, and nothing at all about leader-elected services under load. Given that evidence, a restart looked like the remedy that had worked before. Jonas, Seat Inventory’s on-call, declined it because he had run a shard failover by hand at a game day the quarter before and knew at once what a restart would do to the shard leaders. Chapter 3 described the irony that skill decays exactly where automation succeeds, because people stop practicing what the machine usually does for them. Here, for once, the irony did not apply: the game day had kept the skill alive in at least one person. The review’s finding concerned neither Cue nor Jonas; it concerned the structure around them. A suggestion had arrived in the channel without its blast radius, that is, without any statement of how much of the system it could disturb, and the safety of the morning had rested on one person’s memory of a drill. A system that is safe only when the right person happens to be on call has been lucky, which is not the same thing. So Encore changed the structure: runbook actions now carry a blast-radius tag, and Cue must show that tag with any suggestion it makes. The rule is Encore’s own devising, not an established practice, and it is offered here as one reasonable answer among several possible ones.

The meeting and the document

With the interviews done and both columns of the timeline filled, the review met as a group. The meeting ran for 90 minutes with about 25 people, more than the guides advise. A smaller group talks more freely, which is why the guides advise it, but this incident had touched so many teams that a smaller room was hard to justify, and the one-to-one interviews had already gathered what people might not have said in front of so many colleagues. The meeting’s work was less to discover facts than to let the teams see the same picture together and agree on what to change.

For the document that comes out of such a meeting, the right model comes from the theatre. After every performance a stage manager writes a show report: when the curtain went up and came down, which cues ran late, what broke, who was out, and what the crew did about it. It is plain and unembarrassed, because it is written every night, after the good shows as well as the bad, and so nobody reads it as an accusation. A review document should have that same tone. The analogy breaks in one important place. A show report is written by one person who watched the whole performance from the prompt desk, whereas an incident review has no single witness who saw everything; it must reconcile many partial accounts, some of them honestly disagreeing.

The skeleton below is Encore’s lighter review, the default for anything below SEV1. It follows the example postmortem in Google’s SRE book and Atlassian’s template, borrows the went well and surprises sections from the learning-from-incidents community, and adds a last line that only a CAST review fills in. Each line exists to make one of this chapter’s ideas routine. The two-column timeline turns the STELLA report’s line into a table. Contributing insists on the plural and on the second story of Section 6.2. Surprises is where the mismatched mental models get written down. And the strength field on each action looks ahead to the ranking of remedies in Section 6.5.

Summary        two paragraphs a fan could follow
Impact         who was affected, for how long, how we know
Trigger        why this got a review (severity, budget spent, found by a customer)
Timeline       what the system did | what people knew and were trying to do
Contributing   the conditions that together made it possible, each as a second story
Went well      what limited the harm, and why it worked
Surprises      where someone's picture of the system turned out to be wrong
Actions        owner, strength (stronger/intermediate/weaker), how we will know it worked
Language       reread for "should have", "failed to", "simply", "careless"
CAST (SEV1)    controllers, actions, beliefs, missing feedback

The language line needs a word of explanation, because it is easy to misuse. It is a prompt to reread the draft, not a list of banned words. A phrase such as “failed to” quietly brings back the counterfactual of Section 6.2, judging what people did against a world they could not see. But a writer could delete every phrase on the list and still tell the first story in politer words: “the reviewer approved the change without a cold-cache test” blames as surely as “the reviewer failed to require one”. So the check asks for something harder than search and replace. It asks the author to look again at each sentence that passes judgment and ask whether it explains why the act made sense at the time, or only says that it was wrong.

Recap. Triggers settled in advance decide which incidents are reviewed, and how heavily, so that the choice is not left to embarrassment. An uninvolved facilitator, prompt one-to-one interviews and a timeline with a second column for what people knew and believed produce an account that no meeting could, and the same questions apply to an AI agent’s suggestion as to any other controller’s action. A short skeleton makes the lighter review cheap enough to do every time.

Exercise 6.3. Rewrite a postmortem from your archive into the skeleton above, adding the missing column to its timeline: what people knew, believed and were trying to do. Count the counterfactual phrases in the original. What does the rewritten version show that the original did not?

6.4 Near Misses and Learning from Success

Everything we have done so far in this chapter began with harm: fans who could not buy tickets, an engineer who spent a weekend replaying a decision, a review that followed. But harm is a poor teacher in one respect, because it arrives rarely and late. For every morning like Vesper Hale’s there are many mornings on which the same conditions gathered and then, for one reason or another, dispersed. A near miss is an event in which the conditions for harm came together but harm did not follow, whether through luck or because someone adapted in time. Near misses are far more common than accidents, they carry much the same information about how the system can fail, and they cost nothing to learn from except the effort of noticing them.

Aviation understood this first and built an institution around it. In 1976 the United States set up the Aviation Safety Reporting System, run by NASA on behalf of the Federal Aviation Administration. Through it, pilots and air traffic controllers report close calls confidentially, and those who report promptly receive limited protection from penalty. The arrangement is worth pausing over, because its design is the whole idea. A pilot who has just misread a clearance and nearly crossed an active runway has every reason to say nothing: nobody was hurt, and speaking up invites scrutiny. The system is built to reverse that calculation, so that telling becomes cheaper than staying quiet, and the regulator, by placing the reports with NASA instead of keeping them itself, gives up some power to punish in exchange for knowledge it could not otherwise get. Hospitals later adopted the same idea. Few software organizations run a formal near-miss channel, so in our field the practice is borrowed and still young.

At Encore the CAST analysis of Section 6.2 turned up near misses that nobody had reported, for the simple reason that nothing had happened. These were the three earlier on-sales, met in Section 6.2, during which the node-upgrade reconciler had drained a node mid-sale. Each drain had passed without harm only because no shard leader happened to be on the node, and each was read, silently and by no one in particular, as evidence that the reconciler was safe to leave running. Here the near miss and the normalization of deviance turn out to be the same event seen from two sides: an escape that is reported becomes a warning, and an escape that goes unremarked becomes a precedent. Encore now asks the on-call engineers for each on-sale to note anything that surprised them, whether or not it caused harm, and the Reliability team reads these notes every week and answers each one. The answering matters as much as the reading. A channel into which notes disappear without reply teaches people to stop writing, and the whole value of the channel lies in people continuing to write.

The review then asked a stranger question. If the reconciler had been exempt from the freeze all year, why had the exemption seemed to work for so long? The answer had nothing to do with the reconciler’s design. For most of the previous year a platform engineer had paused the reconciler by hand before each big on-sale. The step appeared in no runbook, and no one had asked her to take it; drains during sales simply made her nervous. In the spring she moved to another team, and the practice left with her. Nobody removed a safeguard, because nobody knew there was one to remove. A large part of the safety that Encore believed lived in its process had in fact lived in one person’s unease.

This is Erik Hollnagel’s Safety-II, introduced in Chapter 2, made concrete. Traditional safety management, which Hollnagel calls Safety-I, studies the rare cases where things go wrong and tries to prevent them. Safety-II starts from the observation that things usually go right, and asks why. Its answer is that they go right because people continually adjust their work to the conditions in front of them, filling the gaps that procedures leave. Hollnagel names the two versions of the work that result. Work-as-imagined is the work as designers, managers and runbooks describe it; work-as-done is the work as people actually perform it, adjustments included. The two always differ, and the difference is usually where the resilience lives. The touring production of Section 6.2 shows the same thing from the wings. The prompt book records every cue as designed, yet a good crew carries a hundred small habits the book never mentions: the extra check on a flown piece in an old theater, the quiet word to a nervous understudy. When a crew member leaves the tour, the cue sheet is unchanged and the show is a little less safe. The analogy stops short in one respect. On a stage the crew can usually see each other’s habits, whereas in software an adaptation such as pausing a reconciler can be invisible even to the engineer’s own teammates, which is exactly why it has to be looked for deliberately.

Looking for it deliberately meant a second kind of review, one triggered by success instead of failure. Encore borrowed its shape from the US Army’s after-action review, which follows training exercises and operations whether they went well or badly and asks three plain questions: what was supposed to happen, what actually happened, and why the two differed. We should be honest about the evidence here. Safety-II gives this habit a strong theory, but the outcome evidence behind it is thin, and in software it remains a movement among practitioners more than an established discipline. Encore adopted it anyway, because its own experience with the reconciler was hard to argue with. After each major on-sale that goes well, the teams involved spend 30 minutes on three questions of their own: what they adjusted, what nearly happened, and what they did that no document describes. The first few of these reviews surfaced a dozen informal practices. Three of them became automation, so that the system no longer depends on someone remembering; one was retired as superstition, a ritual that protected nothing. I have come to believe that this sorting is the real product of studying success. Some adaptations deserve to be built into the machine, some deserve to be taught, and a few deserve to be let go, and you cannot tell which is which until you have found them.

The same lens applies to an incident, because even a bad morning contains things that went right. Vesper Hale’s postmortem had a “went well” section worth studying as carefully as the timeline. When the node drain triggered a leader failover on Seat Inventory shard 3, the shard rejected writes for about 40 s. Because leader terms are fenced, a deposed leader could not keep accepting writes, and because writes carry idempotency keys, a retried write could not be applied twice. Together they turned those 40 s into errors that fans could retry, where they could otherwise have become seats sold twice. That outcome was no accident of the day. It was the result of design decisions made long before and tested at every game day since. A postmortem that recorded only what failed would have left this out, and with it the reason the worst outcome, a broken promise that every seat is sold exactly once, did not occur.

Once near misses are being reported, there is a natural temptation to count them and push the count down. Resist it. In patient safety a rising count of near-miss reports is read as a sign that staff trust the channel, and a falling count as a possible sign that they have stopped using it. A team told to reduce reported near misses will most easily do so by reporting fewer. The same caution applies to the incident metrics that most dashboards display. The VOID, a public collection of thousands of software incident reports, found incident durations so skewed that mean time to resolve says little about whether an organization is improving: a few very long incidents dominate the average, so it swings with chance more than with skill. It also found that incident counts mostly reflect how willing people are to declare an incident, which is to say they measure the culture of reporting more than the reliability of the system. Numbers like these are not useless, but they answer a different question from the one they appear to answer, and a team that steers by them will be steering by its own willingness to speak.

Recap. Near misses are incidents the system escaped through luck or someone’s adaptation, and they go unreported unless reporting is cheap and safe; software is borrowing this practice from aviation and hospitals. Studying ordinary success reveals the adaptations that hold the system up, some of which live only in one person’s habits, and sorting them into automation, documentation and retirement is how an organization keeps them when that person moves on.

Exercise 6.4. Interview two colleagues about something they do before a risky change that no runbook asks them to do. Write each practice down. Which should become automation, which documentation, and which should be retired?

6.5 Turning Learning into Change

Why action items die

A review that ends in understanding has done half its work. The other half is change, and change is where most postmortems quietly fail. Nobody cancels an action item. It simply waits in a tracker, losing priority week after week to work that has a customer and a deadline, until everyone has forgotten why it mattered. When Encore’s Reliability team audited the previous year’s postmortems, they found that about 40% of the action items were still open after 90 days. More telling than the number was which items remained. The open ones were not a random sample. The small fixes, such as an alert threshold or a missing dashboard panel, had mostly closed, because one engineer could finish them in an afternoon. The items still open were the structural ones: those that crossed team boundaries, needed a design before anyone could start, or had no obvious owner. Those are exactly the items that address the conditions behind an incident rather than its symptoms. So an organization can close most of its action items and still learn very little, because the closed items are mostly the easy ones. Google engineers described the same pattern in 2017, and their remedy was a plain one: plan postmortem work like any other work, with owners, priorities and a place in the schedule, instead of leaving it to compete unaided with the roadmap.

Planning keeps an action item alive. A second question is whether the action, once done, will actually prevent anything, and here software has much to learn from older fields that have been asking it for decades. Occupational safety ranks its remedies in a hierarchy of controls: an ordering of ways to reduce a hazard, from the most reliable to the least. At the top is eliminating the hazard altogether. Next comes substituting something less dangerous, then engineering a physical barrier between people and the hazard, then procedures and training, with protective equipment last. The logic of the ordering is simple once seen. Each step down relies more on a person doing the right thing at the right moment, and people under pressure are exactly what an incident has already shown to be unreliable. A guard on a saw protects the operator whether or not they remember the safety course; the course protects them only on the days they remember it.

Healthcare carried this ranking from the factory floor into incident review itself. The National Patient Safety Foundation’s RCA² guidance of 2015 (the name stands for “root cause analysis and action”), building on the US Veterans Affairs’ action hierarchy, sorts the actions a review might propose into three tiers. Stronger actions are physical changes, forcing functions and simplification. Intermediate actions are checklists, redundancy and software enhancements. Weaker actions are training, new policy and warnings. The guidance asks every review to propose at least one stronger or intermediate action for each finding, and it accepts weaker actions only as stopgaps while the stronger work is done. The difference is easy to see in an Encore example. “Remind engineers to check cache state before changing admission rates” is a weaker action: it is a warning, and it works only if a busy engineer remembers it on the right Wednesday. “The load test cannot pass unless it includes a cold-cache run” is a stronger one: it is a forcing function, and it works whether or not anyone remembers. That example suggests a test I now apply to every proposed action. It should name the control it restores, meaning the specific check, barrier or feedback path whose absence the review found. An action that restores no named control is usually a wish. Figure 6.5 places Encore’s changes on the hierarchy.

Figure 6.5 — The hierarchy of controls, with the Vesper Hale changes placed on it. The higher a remedy sits, the less it depends on a tired person remembering it, which is also how RCA² ranks actions.

What Encore changed

With those two ideas in hand, that action items need a place in planned work and that some remedies are stronger than others, we can look at what Encore actually did. Figure 6.6 sets out the changes. Each row begins from a finding of the CAST analysis in Section 6.2, names the change made in response and its owner, and then says, before any work starts, what evidence will show that the change works. That last column is the discipline that keeps the table honest. An action whose success cannot be observed will be declared done when the ticket closes, which is not the same as working.

Figure 6.6 — From finding to change. Each row restores a control missing from Figure 6.2, and each says in advance what evidence will show it working.

Read against the hierarchy, the table leans toward the top. Most rows change what the machinery will allow, and few merely ask people to be more careful. Even the readiness review’s new question about cache state, a procedural step on its own, sits beside a load-test requirement that does not depend on anyone asking it.

The first row is the strongest of these, and it also has the firmest precedent. On the morning of Vesper Hale, the platform’s upgrade automation had no way to know that an on-sale was under way; the calendar lived in people’s heads and in a readiness review the automation never saw. The fix is to make the calendar a signal that machines can read. Argo CD, the GitOps agent that keeps Encore’s clusters matching what Git declares, offers sync windows (the Resilience book introduces them in its chapter on the runtime), which stop the agent from applying changes to chosen applications for a scheduled period, and managed Kubernetes services offer node-upgrade maintenance exclusions built on the same idea. Encore uses both kinds of hold. A small calendar controller, added beside the curtain-up workflow, now writes each on-sale in two places. It writes a deny window on the purchase-path projects in Argo CD, so that routine deploys wait. It also writes an entry in a small calendar service that the node-upgrade reconciler checks before every drain, so that nodes stay put. Both are generated from one source, the on-sale calendar, so the two holds cannot disagree about when a sale begins or ends. The window for a Friday on-sale looks like this:

# AppProject purchase-path; written by the calendar controller from the on-sale calendar
spec:
  syncWindows:
    - kind: deny
      schedule: "0 8 20 11 *"        # T−2h before a Friday 10:00 on-sale
      duration: 6h                    # through the end of the sale
      timeZone: Europe/Paris
      applications: ["sale-gate-*", "bot-screening-*", "seat-inventory-*", "orders-*", "catalog-*"]
      manualSync: true                # break-glass stays possible, and is logged

Three details in that block repay a closer look. The schedule opens the window at 08:00 Paris time, two hours before the 10:00 on-sale, which is exactly the freeze rule written in a form a machine obeys. The application list includes catalog-*, the service whose deploy at 09:52 fell outside the old freeze because its seat-map cache had never been recognized as part of the purchase path. And manualSync: true keeps a deliberate escape hatch. A freeze that no one can override during an emergency would be its own hazard, so a person may still push a change by hand, but that act is logged and therefore visible afterward. The window removes the routine change while keeping the deliberate one, and it leaves a record either way.

That record leads to the change I consider the quietest and the most important: the register of exceptions. Such registers are old practice in change and risk management. I should be candid about their limits, though. None is proven to prevent incidents, and a register that nobody reads becomes a rubber stamp, a place where exceptions are written down so they can be granted with a clear conscience. Its value lies in what it lets people see. Recall what Section 6.2 drew from Vaughan, and what Figure 6.3 showed. She never taught that deviance can be forbidden, and an organization that tried would freeze itself solid, because some exceptions are right; a genuine emergency fix during an on-sale is one. What she taught is that deviance becomes normal when nobody can see it accumulate. At Encore, as we saw, 41 exceptions had been granted in chat in a year, each judged on its own merits, and no one had kept the running total. The register makes each exception a decision with a name, a reason, an approver and an outcome. Whoever weighs the 42nd exception can now see the previous 41, and can ask whether this one is truly different or simply the next step in a drift everyone has stopped noticing. The monthly review is what keeps the register from becoming that rubber stamp: someone has to look at the list and say what it shows.

Tracking, sharing and reading across incidents

The changes in Figure 6.6 will only matter if they get done, which brings us back to where this section began. Encore’s answer follows Google’s: postmortem actions are tracked alongside feature work in the same planning system, instead of in a separate list that nobody opens, and they are reviewed every week. When a service has spent its error budget, Encore’s error-budget policy, signed by the heads of engineering and product (the Resilience book sets it out in its chapter on service level objectives), moves these actions to the front of the queue, so reliability work gains priority exactly when the evidence says it is needed. There is also a limit on how long an item may simply wait. An item older than 90 days must be closed, rescheduled with a written reason, or accepted as a risk by a named person. That third option matters more than it seems. Sometimes the honest answer is that a fix is not worth its cost, and saying so explicitly, with someone’s name attached, is far healthier than letting the item sit open as a polite fiction. Encore’s 90-day rule is generous by some standards: Atlassian sets deadlines of four and eight weeks for its priority actions.

Tracking keeps each review’s actions alive, but some lessons cannot be found in any single review at all. Drift shows across incidents more than within one. Each postmortem is written by different people, about a different failure, and each reasonably focuses on its own story, so a condition that contributes a little to many incidents never looks large in any of them. For this reason the Reliability team reads each quarter’s postmortems together, as a set, and tags the contributing factors that recur. The first such reading found stale cache assumptions in three reviews that never cited each other. Each review had noted its cache problem as a local detail, and only reading them side by side showed a pattern that deserved a structural fix.

Learning also has to travel to the people and systems who were not in the room. The Vesper Hale review went to all of engineering and to the monthly all-hands. Its timeline will be replayed at a game day, much as Google’s SRE teams replay old incidents for new on-call engineers in an exercise they call the Wheel of Misfortune, so that people who never saw the incident can practice the decisions it demanded. The review was also indexed for Cue, and here the loop closes in a way worth pausing over. Cue’s unhelpful suggestion during the incident, to restart Seat Inventory pods, came, as Section 6.3 found, from reading old postmortems that described restarts clearing errors without saying that those services were stateless. The postmortems had recorded the action and left out the conditions that made it safe. Postmortems teach future responders, human and machine alike, and a responder who learns an action without its conditions will one day apply it where it does harm. So every postmortem must say not only what was done, but under what conditions it was safe to do.

Recap. Action items die when they compete unaided with feature work, and the structural ones, which matter most, die first. Planning them as real work keeps them alive. Ranking remedies by how little they depend on someone remembering, as the hierarchy of controls and RCA² both do, and stating in advance the evidence that each one works, keeps them honest. A register of exceptions and a regular reading across postmortems are how an organization watches its own drift, and a postmortem that records the conditions of each action teaches its future readers, people and agents alike, the right lesson.

Exercise 6.5. Audit the action items from your last ten postmortems. What share closed within 90 days, and were the ones still open mostly small or mostly structural? Classify each open item as stronger, intermediate or weaker, using the RCA² tiers. For the three items that matter most, write down the evidence that would show they worked, as specifically as the last column of Figure 6.6.

Kata: The Second Story

This kata asks you to do for another organization what this chapter did for Encore, and to do it starting from a postmortem that has already been written. That is the harder situation, and the more common one. Most of us do not arrive at an incident with a blank page. We arrive with a first story already told, signed off and filed, and our work is to find the second story underneath it without humiliating the people who wrote the first.

Your brief: “Scriptline.” Scriptline runs an electronic prescription network that links 2,000 clinics to 6,000 pharmacies in two countries. It routes 1.2 million prescriptions a day, and 60 engineers run it. Last month an automated certificate-renewal job failed during the clinics’ morning peak, and for 3 hours prescriptions could not be submitted at all; pharmacists fell back to phone calls and paper. The head of engineering’s postmortem concludes “human error: the on-call engineer restarted the wrong cluster while mitigating” and lists 11 action items, most of them training. The renewal job had failed silently twice before, and each time the same platform engineer had fixed it by hand without reporting it. Scriptline has no written review triggers, it still carries 70 open action items from last year, and two senior engineers have asked to leave the on-call rotation.

Read the brief twice before you begin, because nearly every detail in it points back to a section of this chapter. The verdict of “human error” is a first story in the sense of Section 6.2: it names a person who erred and stops there. The 11 training items sit at the weakest level of the action hierarchy from Section 6.5. The two silent failures, each repaired quietly by one engineer, are near misses of the kind Section 6.4 describes, and they reveal an adaptation that held the system up while living only in one person’s habits. The 70 stale action items say that Scriptline’s learning is not turning into change. And the two senior engineers asking to leave on-call are a sign of harm the postmortem never asked about, which is where the restorative questions of Section 6.1 begin. Notice the setting, too. As in the hospitals from which so much of this chapter’s practice was borrowed, the people ultimately affected are patients waiting for medicine, and that raises the stakes of getting the account right.

Deliverables:

1. The second story — a one-page summary that explains why each key action made sense to the person who took it, given what they knew, the pressures they were under and the tools they had at the time. Cover at least the on-call engineer’s restart of the wrong cluster, the platform engineer’s repeated hand fixes, and the head of engineering’s conclusion. Your test is whether each action reads as reasonable from the inside, rather than merely forgivable from the outside.
2. Review policy — written triggers, settled in advance as Section 6.3 recommends, for when an incident gets the lighter contributing-factors review and when it earns a full CAST review (Causal Analysis based on STAMP, from Section 6.2). Then say which of the two this outage gets, and why.
3. CAST table — for at least five controllers, the control actions each one took, what each one believed about the system when it acted, and the feedback it was missing. Two of your controllers must be the certificate-renewal job and the head of engineering’s review process itself, because the process that produced the first story is one of the controllers whose beliefs need examining.
4. Restorative response — the three restorative questions answered for every group that was hurt, including at least the on-call engineer, the pharmacists and the patients: who was hurt, what each needs, and whose obligation it is to meet that need. Add an outline of the public report Scriptline should publish.
5. Near-miss channel — a design showing how the two earlier silent failures would have been reported had such a channel existed: who notices, where they write it down, who reads it and how quickly they answer. State plainly what the reporter is promised in return, since a near-miss channel works only if telling is cheaper than staying quiet.
6. Action-item triage — the 70 open items and the 11 new ones, each classed as stronger, intermediate or weaker on the action hierarchy. For each, decide whether to keep it, merge it with another or close it, and for every item you keep, write down in advance the evidence that would show it worked.

Rubric:

Where you now stand

You can now decide in advance which incidents deserve which kind of review, so that embarrassment does not make the choice for you. You can run a review in which people feel safe enough to tell the truth, write the second story that explains why their actions made sense at the time, and reach for CAST when an incident is large enough to repay its cost. And you can turn what a review finds into ranked changes, each with its evidence named in advance, that survive the pressure of the next quarter’s feature work. Above all, you have seen drift for what it is. It is rarely negligence. It is careful people following the rules as they understand them, confirmed again and again by good outcomes, until the margin that protected them has quietly gone. What remains is the organization that has to carry all of this, day after day: who owns which service and which seam, how people keep skills that automation lets fade, and whether the person who notices the next drift will feel able to say so. Chapter 7 turns to that organization, and then sends Encore back to the same artist and the same kind of morning to see whether it has learned.

References

• Diane Vaughan — The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA, enlarged ed.. University of Chicago Press, 2016.
• Betsy Beyer, Chris Jones, Jennifer Petoff, Niall Richard Murphy (eds.) — Site Reliability Engineering, ch. 15, “Postmortem Culture: Learning from Failure”. O’Reilly, 2016 — free online.
• John Allspaw — Blameless PostMortems and a Just Culture. Code as Craft (Etsy), 2012 — free online.
• Sidney Dekker — Restorative Just Culture: From Disciplinary Action to Meaningful Accountability, 4th ed.. CRC Press, 2025.
• Nancy G. Leveson — CAST Handbook: How to Learn More from Incidents and Accidents. MIT, 2019 — free online.
• Tim Falzone, Ben Treynor Sloss — The Evolution of SRE at Google. ;login: online, 2024 — free online.
• National Patient Safety Foundation — RCA²: Improving Root Cause Analyses and Actions to Prevent Harm. NPSF, 2015 — free online.
• John Lunney, Sue Lueder, Betsy Beyer — Postmortem Action Items: Plan the Work and Work the Plan. ;login:, 2017 — free online.
• Etsy — Debriefing Facilitation Guide. Etsy, 2016 — free online.
• Jeli (now PagerDuty) — Howie: The Post-Incident Guide. Jeli, 2021 — free online.
• David D. Woods (ed.) — STELLA: Report from the SNAFUcatchers Workshop on Coping with Complexity. The Ohio State University, 2017 — free online.
• Erik Hollnagel — Safety-I and Safety-II: The Past and Future of Safety Management. Ashgate, 2014.
• Albert W. Wu — Medical Error: The Second Victim. BMJ, 2000.
• The VOID — The VOID Report. Verica, 2021 — free online.