Building Software at Scale in the AI & Cloud Era

← the whole book on one page

Chapter 7: Building the SaaS Platform — Tenants, Identity, and Administration

Every customer wants a system built for them alone. The craft is to give them one by changing nothing but the data.

A software company that sells its system to other businesses soon finds that it has built two products. The first is the one in the brochure: for Encore, ticketing that a venue can put its own name on. The second is everything needed to have many such customers at once. Someone has to sign them up, give their staff accounts and take them away, decide what each has paid for, help them when something goes wrong, and one day let them leave. The Architecture book drew this second product as the control plane, the part of a software-as-a-service (SaaS) system that manages the tenants, as distinct from the application plane that serves them, and it argued for the shape of each piece (the Architecture book, Chapter 14). This chapter is about building those pieces and keeping them working, at 400 engineers, while the application plane changes under them every day. The question that runs through it is how a company can let hundreds of customers differ from one another without letting its code differ for each of them.

The best early answer came from a small company in a rented apartment on Telegraph Hill in San Francisco. On 8 March 1999, Marc Benioff, Parker Harris, Frank Dominguez and Dave Moellenhoff began work on the first version of Salesforce’s customer relationship management (CRM) service, and on 7 February 2000 they launched it at an event they called “The End of Software”. The slogan mocked the installed enterprise software of the day, in which every customer ran its own copy with its own customizations compiled in. Salesforce ran one shared copy for everyone, and that decision forced a second one. If thousands of businesses were to share one running system, and each wanted its own fields, its own forms and its own rules, then those differences could not live in the code, because there was only one code. They had to live in data. Over the next decade the company built this idea into a general platform, released as Force.com in 2008, and in 2009 its chief architect, Craig Weissman, with Steve Bobrowski, described the design at the ACM’s SIGMOD conference. Weissman had joined in 2002, when the engineering team was still small, so the paper describes what the founders’ design had grown into, and he did much to extend it.

The companion whitepaper Salesforce published in 2008 states the principle with unusual clarity. “Inherently, a multitenant application must be dynamic in nature, or polymorphic, to fulfill the individual expectations of various tenants and their users.” When a customer defines a custom table or lays out a form, the platform “does not create an ‘actual’ table in a database or compile any code. Instead, Force.com simply stores metadata that the platform’s engine can use to generate the ‘virtual’ application components at runtime.” A universal data dictionary holds every object, field, form and access rule as metadata. Tenant data sits in one very large table with hundreds of generic, variable-length columns, onto which each tenant’s custom fields are mapped. Every row carries an organization identifier, the OrgID, which leads every index and partitions every table. Governors cap the processor time, memory and queries any one tenant’s code may use in a single execution, so that no tenant can monopolize the shared engine, and deletions are soft, landing first in a recycle bin. Salesforce even reran every customer’s unit tests against each new release of the platform, to learn before its customers did whether it had broken one of them. Nearly every section of this chapter descends from one of those mechanisms: the organization key on every row becomes Encore’s tenant directory, the metadata becomes its configuration and entitlements, the governors become per-tenant quotas, and the recycle bin becomes the soft delete that Atlassian made universal across its own products after its outage of 2022. Figure 7.1 draws the design as the whitepaper describes it.

Figure 7.1 — Force.com’s multitenancy in schematic. The customizations live in the metadata, every tenant’s rows share one table, and the OrgID that leads every index is the only thing that keeps them apart.

Encore has run a control plane since the venue chain’s email in the Architecture book. About 500 self-service venues share pooled serverless cells of about 50, Professional venues run in Kubernetes cells with a database each, and one Enterprise tenant, venue-chain-a, has a silo of its own. Seven teams build Encore SaaS for venues, among them Tenant Onboarding, Tenant Identity and Tenant Configuration. Verified Resale now asks more of that control plane than venues ever did. It brings in resale partners, Rowmate and later Tessera, organizations with people, machines and a lifecycle of their own that are nevertheless not tenants. It gives every organizer a set of resale rules that must be edited, checked, rolled out and audited like code. And the people who run the marketplace behind the scenes need tools of their own. This chapter follows each of those demands into the control plane that has to meet it.

7.1 Onboarding as a Product

Time to first sale

Touring crews have a word for the work done before a show reaches a town: the advance. Weeks ahead, the production manager calls the venue and works through a long list. How wide is the loading door, how much power is on the stage, when may the trucks arrive, who holds the keys? A good advance is invisible on the night; a bad one shows up as a truck that cannot reach the dock at four in the afternoon. Onboarding a tenant is the software company’s advance, judged by what happens on the night. The comparison breaks on scale: a crew advances a few dozen venues a year by telephone, while a SaaS company must advance hundreds of customers it will never speak to, so its advance has to be software. The Architecture book set Encore’s measure in its Section 14.2: time from signature to first sale.

The first step is to define the event that stops the clock, and to define it so that it cannot be gamed. Activation is the moment a new customer first gets the value the product promised; the term comes from the startup “pirate metrics” of the late 2000s, and Alistair Croll and Benjamin Yoskovitz’s Lean Analytics (2013) treats it at length for SaaS. For a venue on Encore, the value is a ticket sold to a stranger. Encore therefore counts activation as the first paid order placed by someone who is not on the venue’s own staff, so that the box-office manager’s test purchase on the first evening does not count. Every step before that is instrumented as a tracking event, named by the convention the Products book’s Section 4.2 lays down: onboarding_storefront_provisioned, onboarding_payout_account_verified, onboarding_seat_map_completed, onboarding_first_event_published and onboarding_first_sale_recorded. With those events in place, the funnel is a query.

The funnel has to be cut by tier before anyone reads it. An Enterprise onboarding contains weeks of contract negotiation and identity setup, and those waits belong to the deal and say nothing about friction in the product, so mixing them into the self-service numbers would hide everything interesting. Figure 7.2 shows Encore’s self-service venues over the four quarters before Verified Resale.

Figure 7.2 — Encore’s self-service onboarding funnel over four quarters, with each step’s venues, share of sign-ups and median time from sign-up. The provisioning workflow takes minutes; the venue’s own work takes days, and most of the loss happens at the seat map.

Two readings of the figure are worth separating. The first concerns the workflow. Provisioning, the part the Architecture book asked to happen in minutes, does: 99% of venues have a storefront within four minutes of signing up. Engineers looking only at the workflow would conclude that onboarding works. The second reading concerns the customer. Fewer than half of the venues that signed up sold a ticket, and the largest single loss, 64 venues, is at the seat map. When Tenant Onboarding cut the funnel by venue type, the cause became plain. Standing-room clubs, which need only a capacity, passed through the step in minutes; seated halls had to draw every row in the seat-map builder by hand, and many gave up. The fix was a seat-map importer that reads the floor plans and seat lists venues already hold from their previous ticketing system, and Event Setup, which owns the builder, shipped it. The general lesson is that the activation metric must belong to the customer’s journey as a whole, because a workflow measured only on its own steps will report success while customers stall somewhere it cannot see.

The tenant directory

Before onboarding can be built, something has to hold the answer to the most basic question a control plane is asked: who is this customer, and where do they live? In Salesforce’s design that answer is the OrgID on every row. At Encore it is the tenant directory, the single system of record for each tenant’s identity, tier, placement, lifecycle state and contacts. The directory has one writer, the Tenant Onboarding team’s service. Every other service reads it, through an API or from the events it publishes when a record changes, and none keeps its own private copy of who a tenant is. This is ordinary single-writer discipline from the Architecture book’s Chapter 6, applied to the record that every other record depends on.

Verified Resale forced Encore to be precise about what the directory holds. Encore has four kinds of customer: fans, organizers, partners and SaaS tenants. Only venues are tenants. A tenant has a placement, a cell in a region, an isolation tier and a storefront of its own. An organizer is a customer with an organization account in the organizer console; it sells events through Encore and through venues, but it has no cell and no storefront. A partner is an organization that calls Encore’s public application programming interface (API). The three share a great deal, since all three have members, roles, a lifecycle and an audit trail, and so the directory records all of them as organizations, distinguished by a kind. Only tenants carry placement. Keeping them in one directory, while keeping the word “tenant” for venues alone, gives the identity and audit machinery one model to serve without pretending that an organizer lives in a cell. A record for a Professional venue looks like this:

org_id: org_7f3k2q            # opaque; never derived from a name or domain
kind: venue_tenant           # venue_tenant | organizer | partner | encore_staff
display_name: "Riverside Hall"
tier: professional           # placement follows tier; checks use entitlements
placement: { region: EU-2, cell: pro-eu2-04, data_residency: EU }
state: live                  # trial | live | suspended | offboarding | closed
contacts: { owner: user_19ab, billing: user_2c4d, security: user_19ab }
identity: { sso: saml, scim: enabled, recovery_owner: user_19ab }
resale_eligible: true        # market supports Verified Resale
version: 41                  # every change increments; history is kept

Two absences in that record matter as much as its fields. It holds no fan’s personal data, because a record read by almost every service is the worst place for data that must be erased on request. And it holds no entitlements: the directory says what tier a tenant is in, while what that tier includes belongs to the entitlement service of Section 7.3, so that packaging can change without touching the record of who a customer is.

Onboarding as a durable workflow

The Architecture book argued that onboarding should be a workflow with four steps, provision, configure, seed and verify, each with a compensation that undoes it, run on the workflow engine of its Chapter 6. Compensating sagas go back to Hector Garcia-Molina and Kenneth Salem in 1987, and durable engines such as Cadence, built at Uber from 2015, and its successor Temporal have made running them routine. What the argument leaves open is what building such a workflow demands of the people who write its steps, and four properties have mattered at Encore.

The first is that each step is idempotent, keyed by the tenant and the step. Workflow engines retry, and a step that creates a database must recognize, on its second attempt, that the database it is about to create already exists because the first attempt succeeded just before the network failed. The second is that each step states its precondition and checks its result. “Create the storefront domain” first asks whether the tenant’s record is in the trial state and the domain is free, and afterwards it fetches the domain and confirms that it resolves; a step that trusts its own success is a step that will one day report success for something that did not happen. The third is telemetry per step, so that the ops console shows each tenant’s onboarding as a timeline, with the step it is on, how long each step took and what each retry saw. The fourth is resumption. A workflow interrupted by a deploy or a regional failover resumes from its last completed step; it does not start again, and it never leaves a tenant half built.

The Operations book’s curtain-up workflow (its Section 3.2) follows the same pattern for the same reasons: long-running steps across many systems, partial failures, and operators who need to see where they stand. When a self-service venue’s onboarding stalls at two in the morning, nobody is woken; the step retries on its schedule, and if it fails for good the compensation runs and the venue is told plainly what to do next.

Signup and its abuse

Open signup is the reason self-service is cheap, and it is also an invitation. The fraud that follows a ticketing platform’s open door is the fake organizer, who publishes an event that will never happen, sells tickets to it and disappears with the money, leaving fans with the loss and the platform with chargebacks. The obvious defense, verifying every new business before it may sell anything, would destroy the activation curve of Figure 7.2, since know-your-business (KYB) checks, which verify that a company exists and who controls it, can take days.

The published practice of marketplace payment platforms resolves the tension by separating the right to sell from the right to be paid. Stripe’s Connect platform, for instance, tracks whether an account may accept charges separately from whether it may receive payouts, collects the required information “upfront or incrementally”, and disables payouts when a verification deadline passes. Eventbrite tells new organizers that their first payout will be sent after their event ends. The principle is to let activation proceed while the money waits, but it comes with conditions that are easy to forget. The payment service provider (PSP) decides what must be known before the first charge, and that minimum varies by country and business type, so “no checks before the first sale” is never quite true. And holding the payout is not enough to protect fans: the funds themselves must stay where a refund can reach them, or a fake event leaves fans with nothing to be refunded from.

Encore applies the rule to organizers, whose signup, verification and contracts belong to the Organizer Onboarding team. A new organizer can publish and sell as soon as the PSP’s hosted onboarding has collected what it needs for charges. Full KYB runs through the same hosted flow, and until it completes and the organizer’s first event has taken place, the organizer’s revenue stays with the PSP. Encore’s counsel also reads the Digital Services Act (DSA), the EU regulation that has governed online platforms since February 2024, as applying to organizers who sell to fans through Encore’s storefront: Article 30 requires a platform to collect a trader’s identity, contact and payment details, with a self-certification, before the trader may use the service. The signup form collects them, and the PSP’s checks later verify them. Partners are vetted on the same rails with stricter checks, which Chapter 8 describes.

The same rails for the largest customers

Sales-assisted onboarding tempts every company into a second process. The customer is large and the contract is valuable, so senior engineers set up its environment by hand as a mark of respect, and the result, as the Architecture book’s Section 14.2 warned, is a one-off that the next disaster-recovery drill rebuilds wrongly. The engineering answer is to model the human work as steps in the same workflow. A human task is a workflow step that waits for a person, with a named owner, a deadline and an escalation path; workflow engines have long supported such steps, as signals, callbacks or the user tasks of business-process notations.

When a group of 12 municipal venues in one Dutch city joined Encore as Professional tenants, its onboarding ran on the self-service workflow with six human steps added: the signed contract, identity-provider federation, a data migration from the group’s old box-office system, configuration of its door-scanning hardware with Venue Entry, a security review of its integration with the city’s finance system, and a joint test on-sale. Each had an owner, a due date and an escalation to the account lead after two working days’ delay, on the same timeline as the machine steps. When the city’s identity team took three weeks over federation, everyone could see where the onboarding was waiting, and when Encore later rebuilt the cell hosting the group’s venues during a drill, the workflow knew exactly how each venue had been made.

Recap. Onboarding has a customer’s measure, time to a first real sale, and only a funnel cut by tier shows where customers are lost. The tenant directory is the single, versioned record of every organization, holding neither fan data nor entitlements, and durable workflows carry every tier, human steps included, while payouts wait for verification and selling does not.

Exercise 7.1. Write down the tracking events of your own onboarding funnel, from sign-up to the first moment a customer gets real value from a stranger. Then name the step at which you expect to lose the most customers, the query that would prove or disprove it, and the cut (by tier, segment or customer type) without which the answer would mislead you.

7.2 Organizations and Identity

Users, organizations and memberships

Backstage at a large concert, nobody’s identity is a single fact. A lighting technician carries a laminate that names her, the tour she works for and the zones she may enter; on another tour next month she will carry another laminate with other zones, and she is the same person throughout. The laminate fails as a model only in its permanence, since it is printed once for a tour, while a membership must be revocable within minutes. Business software has to model the same thing, and the standard model for it has four parts. A user is a person who can sign in. An organization is a customer, partner or internal body that users act for. A membership joins one user to one organization, and a role on that membership says what the user may do there. Because memberships are many to many, one user can belong to several organizations, with different roles in each. GitHub organizations, Slack’s enterprise workspaces and the “organizations” features of the commercial identity services all take this shape.

At Encore the model has to say something the Architecture book deliberately left out. Its Section 14.2 simplified matters by giving each member of staff exactly one tenant, so that the token issued at sign-in could carry the tenant directly. That simplification served a chapter about architecture, and it no longer fits the company. A box-office manager at a Professional venue may also do the ticketing for a promoter that books shows into other venues, so she is a member of a venue tenant and of an organizer, with different roles in each. Encore’s model replaces the simplification in one precise way. The user chooses an active organization at sign-in, or switches to another later, and the token carries that one organization. The gateway therefore still resolves exactly one tenant or organization per request, as the Architecture book required; the change is only that the person chooses which one it is. Fans stand outside this model altogether. A fan’s identity carries no organization, and the venue a fan buys from is still determined by the storefront, as before.

Encore’s directory has four kinds of organization: venue tenants, organizers, partners and Encore staff. Treating Encore’s own staff as an organization is the choice that pays most over time, because a support agent acting inside a venue’s console is then simply a member of one organization acting on another, which the model can represent and the audit log can record.

Single sign-on in practice

Single sign-on (SSO) lets a customer’s staff sign in to Encore with their own corporate accounts, checked by the customer’s own identity provider (IdP). The two protocols that matter are Security Assertion Markup Language (SAML) 2.0, which OASIS standardized in March 2005, and OpenID Connect (OIDC), whose core specification the OpenID Foundation published in February 2014. The protocols are mature. The engineering lies in letting a customer configure them without a support case.

A Professional venue’s administrator sets up SSO herself in the console, and the console asks three things of her in order. First she proves that the venue owns its email domain, by publishing a token Encore gives her in a TXT record in the domain’s Domain Name System (DNS) zone; without that proof, anyone could configure an identity provider that claims every user at somebody else’s domain. Then she enters the identity provider’s metadata, Encore runs a test sign-in, and she chooses whether accounts are created just in time, on a user’s first sign-in, or only through provisioning. Throughout, the venue keeps a recovery owner: one account that can still sign in with a password and a hardware key when the venue’s identity provider is down or misconfigured. It is the customer’s own break-glass account, and it exists because the day an identity provider fails is exactly the day an administrator most needs to get in.

The one rule Encore adds from hard public experience concerns the parsing. In March 2025 GitHub’s Security Lab disclosed two vulnerabilities in ruby-saml, a widely used library for consuming SAML responses (CVE-2025-25291 and CVE-2025-25292, fixed in version 1.18.0). The library used two XML parsers that could be made to disagree about the same document, and an attacker holding one validly signed SAML document could exploit the disagreement to sign in as any user. What the episode teaches is that signature handling in SAML is subtle enough that nobody should write their own, and that the library one uses is security-critical code to be patched on the security-fix clock of the Resilience book’s Section 7.3. Encore uses one maintained SAML library across the console, and Tenant Identity owns its upgrades.

SCIM and the departing employee

SSO solves the arrival of a user, and the harder problem is the departure. When a box-office manager leaves the venue chain, her access to Encore should end on the same day without anyone filing a ticket. The System for Cross-domain Identity Management (SCIM), published by the Internet Engineering Task Force (IETF) in September 2015 as RFC 7643 for the schema and RFC 7644 for the protocol, lets the customer’s identity provider or human resources (HR) system create, update and deactivate users in Encore automatically. The protocol exchanges JavaScript Object Notation (JSON) documents over HTTP, and most of its value lies in deprovisioning.

The cost of slow deprovisioning has a public record. In a filing with the US Securities and Exchange Commission on 4 April 2022, Block disclosed that on 10 December 2021 a former employee had downloaded reports containing information on about 8.2 million current and former customers of Cash App Investing, “after the employee’s employment ended”. The filing did not say why the access remained. For our purposes the lesson is the interval itself: between the day a person leaves and the day their access ends, anything they can still reach is exposed, and every organization has such an interval, measured or not.

Encore measures the gap as time from deprovision to lost access, with a target of under five minutes, and “lost access” is meant literally. A SCIM request that sets a user’s active attribute to false, which identity providers send far more often than a delete, disables the account, but a session already open in a browser and a refresh token held by an integration both outlive the account unless something cuts them. So deactivation at Encore revokes the user’s sessions and tokens in the same transaction, the console’s sessions are short, and the metric is measured by a probe that tries to use a deprovisioned user’s session after the fact.

Testing a SCIM endpoint is mostly testing against other people’s implementations, since each major identity provider reads the RFCs slightly differently. Tenant Identity built an identity-provider simulator in the manner of Section 3.3, with the behaviors of the main providers on demand, and checks it against public validators such as Microsoft’s SCIM Validator and open-source conformance testers. The details that break integrations are small and specific: a request for an unknown user must return 404 with SCIM’s error schema, never a 500, and a duplicate create must return 409 with the uniqueness error type.

Roles, permissions and delegated administration

The long-standing answer to what users may do is role-based access control (RBAC): fine-grained permissions, such as refund.issue or event.publish, grouped into roles such as box-office manager or finance viewer, and roles assigned to memberships. Enterprise customers soon ask for custom roles, which RBAC handles with little strain. What strains it is hierarchy. The venue chain’s central administrator must manage the staff of every venue in the chain, a regional manager must manage only her region’s venues, and a venue manager only her own. Expressing that with flat roles means creating a role per venue, or writing code that walks the hierarchy, and Encore had done both, separately, in each tier’s console.

Relationship-based access control answers the hierarchy directly, by storing permissions as relations between objects, such as “user U is an admin of region R” and “venue V is in region R”, and computing what follows. Google’s Zanzibar, described by Ruoming Pang and colleagues at USENIX ATC in 2019, is the proof that this works at scale: the paper reports “trillions of access control lists and millions of authorization requests per second”, serving Calendar, Drive, Maps, Photos and YouTube with a 95th-percentile latency under 10 milliseconds and better than 99.999% availability over three years. Its “zookie” consistency token keeps a permission change and a content change from being applied out of order. Zanzibar proves the model, not that a company of Encore’s size should build one. Airbnb’s Himeji, a centralized authorization service inspired by Zanzibar and described by its engineers in 2021, is the closer precedent, and open-source implementations of the model now exist.

Encore replaced its tier-specific role code with one authorization service, run by Tenant Identity on an open-source Zanzibar-style implementation, which every console and API consults. Delegated administration fell out of the model. The chain’s administrator can grant roles anywhere below her in the hierarchy and nowhere above it, and the authorization service enforces the limit for every console.

Partners as organizations

Verified Resale brought in the first organizations that are neither tenants nor organizers. Rowmate, the launch partner, and Tessera, the second, are organizations with human members, who use a partner console to watch their listings, and machine clients, which call the public API. Each partner gets a sandbox of its own at once and, after it passes certification, a production grant of its own: its own scopes, keys and client certificates. The scopes keep the original boundary that held even when Rowmate’s key leaked (the Resilience book, Section 7.2): listings:read and resale:write, never primary inventory, fan data or payouts. Machine clients authenticate with the 15-minute certificate-bound tokens Encore adopted after that leak.

The organizer’s side of the relationship is recorded as configuration. An organizer’s resale rule for an event, described in Section 7.3, decides whether that event’s tickets may be resold at all and when; if they may, a listing enters Encore’s resale pool and is visible to every authorized partner. The gateway checks two records on every partner call: the partner’s grant, to establish what this partner may do, and the event’s effective resale rule, to establish whether it may be done for this event now. The grant records the Encore person who approved it and the partner administrator who accepted it, and revoking it takes effect within one token lifetime, 15 minutes, because no new tokens are minted and existing ones expire. A change to an organizer’s rule reaches the gateway through the ordinary configuration path, so the console and the gateway never disagree, and each partner receives a rules.changed webhook so that its own pages can follow.

Recap. The business identity model of users, organizations, memberships and roles replaces the one-tenant-per-person simplification with an active organization carried in the token. SSO needs a maintained parser, SCIM is measured to the last live session, relationship-based authorization gives hierarchy one home, and partners hold per-partner grants checked with each event’s rule on every call.

Exercise 7.2. Draw your product’s identity model as entities and relations. Mark where a person who works for two of your customers would break it, and where your own staff act inside a customer’s account. Then measure, with a real test account, how long a deprovisioned user’s open session keeps working.

7.3 Entitlements and Configuration

Entitlements as a service

The Products book’s Section 5.3 decides what each plan includes and argues that a tier should be held as data, a set of entitlements, the features, limits and quotas a customer has paid for, so that product code checks entitlements and never tiers. This section builds the service that makes the argument true. A mapping from each plan to its entitlements is held as versioned data; each organization’s effective entitlements are computed from its plan plus any contractual additions; and code asks a single question, “does this organization have this entitlement, and what is its limit?”, through one client library backed by one service. Stripe’s billing platform now sells the same shape as a product, a fair sign that the pattern has settled. Each entitlement also carries a contract test, of the kind Section 3.3 describes, which checks that an organization without it cannot reach the feature by any path, since an entitlement enforced on one path and forgotten on another is not enforced.

Two design choices decide whether such a service helps or harms. The first is the failure mode of each check, chosen per entitlement and written down. A check that only decides what to display, such as whether to show the resale badge in a venue’s storefront, should fail open, because showing a feature briefly to someone who has not bought it costs little. A check that moves money or exposes data should fail closed, because the cost of being wrong runs the other way. The second is caching: checks sit on hot paths, so the library caches answers, with a stated bound on staleness, 60 seconds at Encore. A third question waits for Section 7.5: what happens to work already in flight when an entitlement disappears.

Verified Resale is an entitlement, and it shows why the service is more than a commercial device. Every plan on every tier includes it, as the Products book’s Section 5.3 decided, so the entitlement does not distinguish one plan from another. It exists because it carries limits, such as the markets in which resale is offered, and because it has a lifecycle: an organizer whose verification has lapsed or whose account is suspended loses it, and the loss must reach every service that would otherwise list, sell or pay. If resale had been gated by a check on the plan, each of those services would have had to learn the suspension rules separately.

Configuration as versioned data

Entitlements say what a customer may use; configuration says how a customer has chosen to use it. A venue’s branding, its box-office hours, its refund-policy text and an organizer’s resale rules are all configuration. They change often, people outside engineering set them, and each change alters a running system, so Encore treats per-tenant and per-organizer configuration as versioned data: typed by a schema, validated before it is accepted, kept with its full history, reviewed when it is risky, and attributed to the person who made it.

The best published case for this discipline is Facebook’s. In “Holistic Configuration Management at Facebook” (SOSP 2015), Chunqiang Tang and his colleagues described Configerator and its companion tools, which handled thousands of configuration changes a day and trillions of configuration checks. In 2014, they reported, more engineers changed live configuration than changed the frontend product code. The paper observes that “configuration errors are a major source of site outages”, citing a 2003 study by David Oppenheimer and colleagues, which found operator error, much of it in configuration, among the leading causes of failure in the services it examined. Facebook’s response was to treat a configuration change “the same as a code change”: it is written as source, checked by a compiler and validators, reviewed, tested and rolled out in stages. The Resilience book’s Chapter 4 has already followed two recent outages that began as configuration pushed everywhere at once.

Salesforce’s whitepaper made the same point from the other direction. Because its tenants’ customizations were metadata, a change to one tenant required only “a simple non-blocking update to the corresponding metadata”, with no deployment and no downtime. That is the power of configuration as data, and it is also its danger, since an update that requires no deployment also passes through none of a deployment’s checks unless someone builds them in. Encore’s configuration service, owned by Tenant Configuration, builds them in: each configuration type has a schema owned by the service that interprets it; the configuration service stores, versions, validates and rolls out every type the same way. The Resale service owns the schema and meaning of resale rules, and the configuration service carries them.

Rolling out configuration like code

A configuration change that affects one organizer is reviewed and applied. A change to a platform-wide default, which affects every tenant or organizer at once, is a deployment and is rolled out like one. A change to a plan file, which alters the entitlements of every customer on that plan, is a deployment of this kind. Encore’s progressive delivery already runs canaries at 1%, 5%, 25% and 100%, with none during on-sales (the Resilience book, Section 4.3), and configuration uses the same mechanics. What is specific to configuration is the choice of who goes first. Code canaries are slices of traffic; configuration canaries are tenants. Encore’s first wave is internal test tenants, its second a few friendly customers who have agreed to receive changes early, and its later waves proceed cell by cell, with health signals computed per tenant and a rollback that restores the previous version of the configuration, which the history makes trivial.

The case for this discipline comes, fittingly, from the touchstone company itself. On 17 May 2019 Salesforce deployed a database script that, by its own statement, gave users in organizations that used or had used its Pardot marketing product “broader data access than intended”. To contain the exposure, Salesforce cut off access to every organization on the affected instances, of which press reports counted more than a hundred, including many customers that had never used Pardot. Most access returned after about 15 hours, and some customers’ administrators had to repair permissions by hand. A change to tenants’ permissions is a change to their configuration, and it needs a wave rollout and a tested rollback like any other.

When the Resale team migrated every stored resale rule to the second version of its schema, the change went first to internal test organizers, then to two design-partner organizers who had agreed to it, then to the rest by market, with the fairness invariant tests run against each wave.

Organizer resale rules

The resale rules are the most consequential configuration Encore holds, because they decide what fans may do with tickets they own and what the law permits in each market. The Products book’s Section 5.5 explains why price caps are both a product rule and a compliance capability.

An organizer may change four things, and only in one direction. It may disable resale for an event. It may shorten the resale window, by opening it later or closing it earlier; by default the window runs from purchase until two hours before doors open. It may lower the per-fan listing and buying limits, which default to the event’s own per-fan limit. And it may set a price cap below face value, where face value in these books means the original price the seller paid, primary fees included. It may never raise the cap above face value or loosen any default, and the fee is not an organizer setting at all: the buyer pays 10%, split as four points to the partner through which the buyer purchased, three to the organizer and three to Encore, and when the buyer purchases in Encore’s own app Encore keeps the partner’s four points. The third of Encore’s fairness invariants states the rule: “Organizer caps and rules can only tighten.” The schema makes that invariant structural, so that a loosening rule cannot even be expressed:

resale_rule:                       # owned by the Resale service; stored by config
  scope: { organizer: org_m2p8, event: evt_4471 }
  effective_from: on_publish        # or a future date; never retroactive
  enabled: true                    # false not offered where law forbids blocking
  price_cap: { platform_max: face_value, organizer_pct_of_face: 90 }   # at most 100
  window: { opens_after_onsale: P7D, closes_before_doors: PT48H }  # defaults: 0 and PT2H; may only grow
  per_fan: { list_max: 2, buy_max: 2 }    # bounded above by the event's per-fan limit
  # fee split is not configurable: 4/3/3 is platform policy
  set_by: user_7c1a                # person, never a shared account
  reason: "Tour-wide policy agreed with the artist's management"

The law enters as bounds that differ by market. Ireland’s Sale of Tickets (Cultural, Entertainment, Recreational and Sporting Events) Act 2021, in force since 31 July 2021, bans the resale of tickets above their original sale price for events at designated venues, those with a capacity of at least 1,000 whose operators have applied, and at designated events. The Act also voids terms that would block resale at or below face value, so the console offers Irish organizers only a lower cap, a window and per-fan limits, and no switch to disable resale. Where the law caps what the buyer pays in total, as it does for designated Irish events, the 10% fee comes out of the seller’s proceeds. The Resale service enforces these limits whatever the console, a partner or the rule assistant sends it. The assistant, which drafts rules from an organizer’s plain-language request (the Products book, Section 6.5), can express only what this schema allows, which is why a model’s mistakes cannot reach a fan’s price.

Every rule carries an effective date, because an organizer who tightens the window on Tuesday must not strand a fan whose listing was valid on Monday. Listings already reserved by a buyer complete under the rule in force when they were reserved, and listings that the new rule forbids are withdrawn with a message to the seller explaining why. The history of every rule is kept, so that a dispute about what the rule was on a given day is settled by a query.

Flags, entitlements and configuration

Three mechanisms can each switch behavior on for one customer and off for another, and that resemblance is the source of a persistent confusion. A release flag can be targeted at one organizer; so can an entitlement; so can a configuration value. Used interchangeably, they produce systems in which nobody can say why a customer sees a feature, who decided it, or whether removing the switch will break a contract. Figure 7.3 separates them by purpose, owner and lifetime.

Figure 7.3 — Three switches, three owners. The confusion begins when a switch outlives its purpose and is quietly used as one of the others.

Pete Hodgson’s taxonomy of feature toggles (2017), which Section 4.5 uses, includes permission toggles as a fourth kind of flag: toggles that turn features on for particular users, such as premium customers. Hodgson notes that permission toggles can be very long-lived. Encore’s rule is a refinement for a SaaS company: a permission toggle that will live longer than a release is an entitlement, and it moves into the entitlement service, where it acquires an owner, a plan and a place in billing. Knight Capital, which the Resilience book’s Section 4.1 recalls, showed how dangerous a repurposed switch can be.

Encore found a quieter example of its own during the build. In the Partner Listings years, some organizers objected to “resale available at Rowmate” links on their sold-out pages, and support turned the links off for them with a release flag left over from Partner Listings’ launch. By year 9 the flag was a contractual setting in all but name, with no owner and no record of who had been switched off or why. Verified Resale’s migration turned each organizer’s flag state into an explicit resale rule, disabling resale where the law allowed it; in Ireland, where it does not, the organizer kept the platform’s rule and was told which limits it could still tighten. Each organizer received a letter explaining the change. Then the flag was deleted.

Recap. Entitlements are data checked through one service with a stated failure mode, and configuration is versioned, typed, attributable and rolled out in waves of tenants. Resale rules are configuration whose schema can only tighten within each market’s law, and flags, entitlements and configuration keep separate owners and lifetimes.

Exercise 7.3. List every per-customer setting in your product and classify each as release flag, entitlement or configuration, with its owner and expected lifetime. Mark every flag older than six months that is targeted at named customers, and decide which of them is an entitlement in disguise.

7.4 Internal Tools

Internal tools have users

Every SaaS company runs a second set of products that its customers never see: the consoles in which support agents answer cases, finance staff reconcile payouts, partner managers watch partners, and operators intervene in a tenant’s affairs. These tools tend to be built last, from whatever screens already existed, and the people who use them all day learn to work around them. It shows up as slower cases, as errors made under time pressure, and as engineers who are asked to run a query in production because no tool can answer the question.

Encore treats these tools as products with users: research with the people who use them, design on the Setlist design system, and measures of task time and error. There is little published evidence specific to internal tools, and vendor surveys on the subject are marketing; the evidence for the methods comes from usability practice in general (the Products book, Section 6.3), which Section 6.2 applies to developers. When Support Tools observed agents at work, it found that they spent about a third of each case gathering context from six separate screens: the order, the payment, the ticket and its barcode history, the fan’s messages, the organizer’s refund policy and, after Verified Resale, the listing and its reissue. One case view, assembled from the same services and laid out in the order agents actually asked their questions, replaced the six.

Typed operator actions

The most dangerous internal tool is the one that is not a tool at all: an engineer with a database console and good intentions. An operator action is an operation on a customer’s state that staff perform outside the normal product flow, such as extending a hold, suspending a listing or restoring a deleted event. Encore’s rule is that every operator action is a typed API call in a catalog, with a reason code, stated preconditions, a dry run and an audit entry, and that nobody runs SQL against production data.

Atlassian’s outage of April 2022 is the clearest public argument for the rule, and its post-incident review is unusually candid. On 5 April, between 07:38 and 08:01 UTC, a maintenance script meant to delete a retired application, “Insight – Asset Management”, from customers’ sites was given a list of site identifiers instead of application identifiers. The API it called accepted either kind and gave no “warning signal to confirm the type of deletion (site or app) being requested”, and it deleted 883 sites belonging to 775 customers. Restoration took up to 14 days, because Atlassian “did not have the ability to select a large set of customer sites and restore all of their inter-connected products”, and data in shared stores had to be extracted by hand. Atlassian “missed our RTO but met our RPO”, its recovery-time and recovery-point objectives, losing at most five minutes of data. Among its remediations were soft deletion everywhere, staged rollout of deletions, automation for restoring many sites at once, and an immediate block on bulk site deletion.

Each remediation corresponds to a property of a typed action. A typed action knows what kind of identifier it takes, so an application identifier and a site identifier cannot be confused. It checks its preconditions, so “is this an application?” is asked before anything is deleted. Its dry run shows what would change, and for destructive actions the dry run is mandatory and its result must be confirmed. And its destructive effects are soft for a retention period, as Salesforce’s recycle bin was in 2008, so that a mistake can be undone by the same catalog that made it. Encore’s catalog holds 40 actions, among them “suspend partner listings”:

action: partner.listings.suspend
target: { type: partner_org, id_format: "org_*", kind_must_be: partner }
scope: [all_events, event_ids]        # an empty scope is rejected, never "all"
preconditions:
  - partner.state == live
  - no_open_suspension(partner, scope)
reason_codes: [policy_breach, security_incident, partner_request]
dry_run: required                     # shows listings affected and reserved sales spared
effect: listings -> paused            # reversible; reserved listings complete
undo: partner.listings.reinstate
approval: single                      # not on the dual-control list
audit: { statement_of_reasons: true }
available_to_cue: propose_only

The catalog is the same one that the Operations book’s Section 3.5 describes from the operator’s side, the platform-enforced list from which Cue, Encore’s operations agent, may propose actions. People in the ops console and Cue draw on one catalog, and each entry states whether Cue may propose it; actions that touch a tenant’s data or money are not available to Cue at all.

Two people, short access and the customer’s consent

Some actions are too consequential for one person, and some access is too sensitive to be standing. Three patterns govern them, all long established. Dual control requires a second person to approve an action before it runs, as in the two-person rule of nuclear weapons custody or the four-eyes principle of banking. Anyone who has worked on platforms inside a bank, as I have, learns that the principle is respected only where its scope is narrow enough to be honored every time. Encore therefore applies dual control to a named list of staff actions on tenant data: impersonation of a customer’s user, bulk refunds over €5,000 and changes to payout details. The refund agent’s limit, which auto-approves refunds under €80 when more than 48 hours remain before the show, is a separate rule about a separate actor. A vague rule that “money actions need two people” would have collided with the refund agent on its first day.

Time-boxed access means that elevated access expires on its own. The Resilience book’s Section 7.2 gives engineers break-glass access for 60 minutes with a second approver, and the same expiry and second person now apply to staff acting inside a tenant. The third pattern is access with the customer’s consent. Salesforce lets its customers’ users grant administrators or support login access for a period they choose. Microsoft’s Customer Lockbox goes further: an engineer requests access with a service request number and a duration, a Microsoft manager approves, and then the customer’s own approver must approve within 12 hours or the request expires; access lasts at most four hours, and every action is logged in the customer’s audit log as “Microsoft Operator”. Google Cloud’s Access Approval follows the same shape, and so does Encore’s. A support agent may view a venue’s or organizer’s console only with access that the customer has granted, by default for 24 hours and never more than 72, and impersonation additionally needs a second Encore approver.

The scope of consent must include what customers send to support as well as what support may view. In November 2023 Okta reported that, between 28 September and 17 October, an attacker had reached its support case system through a service account whose credentials an employee had saved to a personal Google account, most likely compromised from there, and had downloaded files associated with 134 customers. Some were HTTP Archive (HAR) files, browser recordings that contained live session tokens, and the attacker used them to hijack the sessions of five customers. The support system was a privileged system, and the files in it were credentials in all but name. Encore’s case view therefore scrubs session tokens from uploaded files on arrival, and every uploaded file expires with the case.

Audit as a product surface

The Architecture book’s Section 14.5 named the requirement: audit logs become a product surface, visible to the tenant, exportable and immutable. What remains is to design the record and the view. The catalog of security controls published by the US National Institute of Standards and Technology (NIST), SP 800-53, lists what an audit record must establish (control AU-3): what happened, when, where, from what source, with what outcome and by whom. Encore’s record adds the distinction that a multi-organization system cannot do without, between the actor, who performed the action, and the principal, the organization on whose behalf it was performed.

Each of Encore’s audit records therefore carries the actor (a user, a machine client or an Encore staff member), the organization the actor belongs to, the organization acted for, the action from the catalog, its target, the reason code and free-text reason, the outcome, the request identifier that joins it to traces, and the consent or approval under which it ran, if any. Records are append-only and chained to the event ledger, whose tamper evidence the Architecture book’s Chapter 6 describes. They are retained for the period each customer’s contract and the law require, and customers can query and export them.

The product consequence is that an organizer can see every action taken on its events, including those taken by partners and by Encore’s own staff. When a support agent extended a resale listing’s window for a fan at the organizer’s request, the organizer’s console showed it as an action by an Encore staff member, on the organizer’s behalf, with the reason and the consent that allowed it. Support teams used to working out of sight find this uncomfortable at first, and enterprise customers’ security reviewers ask about it before anything else.

Tools for running a marketplace

Verified Resale made Encore the operator of a marketplace, and a marketplace needs a console of its own. The partner manager’s console shows each partner’s health (call volumes, error rates, certification status and the age of its keys), listings under review, suspensions and reinstatements, goodwill credits and refunds subject to dual control, and the dispute queue. Each control in it is a catalog action, so the console adds no power that the catalog does not already define and audit.

For resale, Encore is an online marketplace under the DSA, and three of the regulation’s duties bear directly on this console: a mechanism for notices about illegal content (Article 16), a statement of reasons to anyone whose content or account is restricted (Article 17), and an internal complaint-handling system (Article 20). So a suspension records its reason in a form that can be sent to the affected party, and the dispute queue doubles as the complaint system. Fans who resell their own tickets within per-fan limits are consumers, not traders, and the regulation’s trader-traceability rules in Article 30 would apply to anyone reselling as a business. That is one more reason, alongside fairness, why Verified Resale admits no professional resellers. The policy on partners’ conduct belongs to Chapter 8’s Section 8.3.

The first dispute in the French launch tested the console. A buyer in Lyon wrote that his reissued ticket had not appeared in his phone’s wallet. From one screen the agent could see that the reissue had completed, that the seller’s old barcode was on the gates’ revocation lists, and that the buyer’s ticket was live and rotating in the Encore app; only the convenience copy in the wallet had failed to update. The agent re-sent the pass and closed the case in four minutes.

Recap. Internal tools have users whose time and errors can be measured. Operator actions are typed, checked, dry-run, reversible and audited, from one catalog shared by people and Cue; dual control covers a short named list, access to a customer’s data needs consent and expires, and audit records separate who acted from on whose behalf.

Exercise 7.4. Pick one action your support or operations team performs by hand in production. Write it as a typed catalog entry with its target type, preconditions, reason codes, dry run, undo and audit record. Then ask the person who performs it today which precondition you have missed.

7.5 The Tenant Lifecycle

A lifecycle as a state machine

The Architecture book named the stages, onboarding, tier changes, suspension and offboarding, in its Section 14.2. Building them well means making the lifecycle explicit as a state machine, a fixed set of states with defined transitions between them, owned by one service, with each transition published as an event that every affected service must handle. Figure 7.4 shows Encore’s.

Figure 7.4 — Encore’s tenant lifecycle, with what each state means for fans and for every service. Each transition is an event, and each consumer’s handling of it is tested by contract.

The directory owns the state, and the transitions are published as TenantSuspended, TenantReinstated, TenantOffboardingStarted and so on. A transition that a service ignores is a bug, and the platform finds it with a contract test: every service that holds tenant data or acts on a tenant’s behalf registers, in the service registry, how it handles each transition, and a consumer-driven contract test of the kind Section 3.3 describes checks that it does so. When Verified Resale’s services joined, the test failed on the first build: the Resale service did nothing on TenantSuspended, so a suspended venue’s events would have kept accepting new listings. The fix took a day; in production, the same gap would have surfaced only when a fan complained.

Verified Resale also added a state that belongs to an event. Resale paused is a sub-state of an event in which existing listings are hidden and no new ones are accepted, while primary sales and valid tickets are untouched. It is entered when an organizer’s rule disables resale with immediate effect, when Trust & Safety suspects a fraud pattern on one event, or when the venue or organizer selling the event is suspended. Keeping it separate from the tenant’s state allows a single event to be paused without touching anything else the venue sells.

Upgrade and downgrade

Moving between tiers has two parts that run at very different speeds. Entitlements change at once: when a self-service venue upgrades to Professional, the entitlement service reflects the new plan within its 60-second cache bound, and features appear. Placement changes slowly, because moving from a pooled self-service cell to a Professional cell with a database of its own, or back, is a data migration. The Architecture book argued that such a move must be planned and verified; here it is a workflow on the same engine as onboarding, with steps to copy the tenant’s data to its new home, verify counts and checksums, switch the tenant’s routing at the gateway, and keep the old copy for a reversal period. A self-service venue that upgrades sees Professional features the same afternoon and moves to its own database over the following week, scheduled around its on-sales.

A downgrade raises a harder question, because the tenant may be over the limits of its new plan. A Professional venue moving back to self-service may have more staff accounts than self-service allows, or a custom domain that self-service does not include. Encore’s rule is a grace period, 60 days, during which nothing is removed and the console shows what will change and when, followed by changes that never destroy data: surplus staff accounts are disabled and kept, and the custom domain redirects to the pooled storefront for a year. It is the Products book’s grandfathering with a sunset date (Section 5.5), applied to the shape of an account.

Suspension without collateral damage

Suspension is the lifecycle’s most delicate transition, because those who suffer from a careless one are rarely the ones being suspended: a venue may be suspended for an unpaid invoice or a breach of terms, and its fans did neither. The Products book’s Section 5.4 established that suspension is a state only a person can enter, that no on-sale is stopped without a person’s decision and that a suspended organizer’s fans are never harmed; the Architecture book’s Section 14.2 established that suspension blocks access without deleting anything. Encore adds the fans’ side, and states it as a rule: a fan’s valid ticket is never revoked because of a dispute between Encore and an organizer or a venue.

The rest follows from that rule. When a venue is suspended, tickets already sold stay valid and continue to rotate, so that the gates admit their holders. New sales stop. Resale listings for its events move to resale paused, and listings already reserved complete. Payouts to the venue are held, while payouts to fans who resold tickets proceed on their normal schedule, since the sellers are not party to the dispute. Each of those behaviors is a service’s response to TenantSuspended, checked by the contract test described above. An organizer, though not a tenant, has a lifecycle state in the same directory, and its suspension, the case the Products book’s dunning policy has in mind, triggers the same guarantees for its events.

The Salesforce incident of May 2019 shows the other side. Cutting off every organization on the affected instances was defensible as containment, but it was suspension at the scale of an instance, falling for many hours on customers who had done nothing. Encore’s lifecycle is designed so that the narrowest scope is always available: one event, one tenant, one partner, or one partner on one event.

Export and switching

Every tenant will one day leave, and the law now says a good deal about how. For people, the General Data Protection Regulation (GDPR) gave a right to data portability in Article 20, covering personal data they provided where processing rests on consent or contract. For business customers of cloud and SaaS services, the EU Data Act (Regulation 2023/2854) applies from 12 September 2025, and its chapter on switching between data processing services sets clocks that a switching runbook must meet. Figure 7.5 gathers them.

Figure 7.5 — The Data Act’s switching clocks, as a runbook must meet them. The charges row agrees with the Architecture book’s Section 13.4.

The practice that satisfies those clocks is a documented export format, published in advance with the exhaustive list of data categories; a self-service export through the console and an open API; and a switching runbook run as a workflow whose timers are the clocks of Figure 7.5. Encore’s export for a venue tenant includes its events, seat maps, orders, tickets with their current state, resale history and audit log. Organizers, which are not tenants, get the same export by Encore’s own policy.

A venue’s resale history and orders contain fans’ personal data, for which the venue and Encore play different roles under the GDPR, depending on the field and the purpose. Encore’s export format therefore declares, field by field, what personal data it carries and the basis on which it is transferred, and its design was reviewed with Encore’s data protection officer before the first tenant used it.

Offboarding and erasure with evidence

The last transition, to closed, cannot be undone, and the quality of an erasure is invisible unless it is proved. The Architecture book’s Section 14.5 explained crypto-shredding, deleting data by destroying the keys that encrypt it, and its limit: it reaches only copies encrypted under the tenant’s own keys. Encore’s offboarding workflow therefore begins with an inventory of every store that holds the tenant’s data, drawn from the service registry, and ends only when each store has reported its erasure or its key destruction. The result is a deletion record, a signed document that lists each store, the method used and the time, which the departing customer’s auditors can keep.

Erasure has limits set by law, and the workflow must know them. The GDPR’s right to erasure does not apply where a legal obligation requires data to be kept (Article 17(3)(b)), and financial records are the usual case. Retention periods are national, and some have changed recently: in Germany, accounting vouchers fell from ten to eight years from 1 January 2025. Encore keeps such records for the statutory period of the relevant country, in a separate store under keys of their own, and erases the rest after the retrieval period the Data Act requires.

Two further practices complete the design. The first is soft deletion with a retention window before anything is destroyed, which Atlassian made universal across its products after 2022. The second is a rehearsed restore of one tenant. Restoring a whole database is a drill the Resilience book’s Chapter 6 recommends; restoring one tenant from pooled storage without disturbing its neighbors is a different operation, and the one Atlassian lacked. Encore rehearses it each quarter on a self-service test tenant in a pooled cell.

When a Professional arena in the Netherlands left Encore for another provider, it had sold tickets, some of them resold and reissued, for events after its departure, and those tickets had to work. So the export included every valid ticket for future events with its current identity, and the revocation list of every barcode that reissue had killed, in the documented format that the new provider’s gates could load. Encore kept serving those tickets in its app until each event had passed, and only then erased the arena’s personal data on schedule and issued the deletion record.

Recap. The lifecycle is a state machine whose every transition is checked by contract test, and suspension never revokes a fan’s valid ticket. Tier changes migrate placement with verification and downgrade with grace, export runs to the Data Act’s clocks, and offboarding ends in erasure with evidence and a rehearsed single-tenant restore.

Exercise 7.5. Write the offboarding of one customer as workflow steps, from notice to deletion record. Mark the data you are legally required to keep and for how long, the stores that crypto-shredding would not reach, and the step at which your customer’s own users, as distinct from the customer, could be harmed.

Kata: The Inspector’s Copy

Your brief: “Countwise.” Countwise sells stock-control and controlled-drug-register software as a service to about 4,000 independent community pharmacies in three EU countries. It has 60 engineers in eight teams. Every pharmacy is a tenant in one pooled database, onboarded by hand by a support engineer in nine days on average. A buying group of 20 member companies, which together own 140 pharmacies, wants to move all of them to Countwise this year, on four conditions: single sign-on with its own identity provider; automatic provisioning from its HR system, so that departing staff lose access the same day; one administrator who can manage all 140 pharmacies, with regional managers below her; and an audit log of every entry in each pharmacy’s controlled-drug register that the national pharmacy inspectorate can read directly. Many of the group’s pharmacists are locums, who work in three or four pharmacies a week. Last year two departing pharmacies asked for their data; each export took three weeks of hand-written SQL, and one lacked its register history. The Data Act now sets the clocks for any pharmacy that leaves. Leadership can spare eight engineers for two quarters.

Deliverables:

1. Tenant directory — The record that makes each pharmacy, member company and the buying group an organization, the events it publishes, and the services that must stop keeping their own copy.
2. Onboarding workflow and activation — The steps from contract to first register entry, with preconditions, checked results and compensations, the human steps with owners, and the activation event that stops the clock.
3. Identity model — Memberships and roles for locums, an active-organization rule, SSO with domain verification and a recovery owner, a deprovisioning target that includes open sessions, and delegated administration over the group’s hierarchy.
4. Audit and inspector access — The audit record, distinguishing actor from principal, and a design for the inspectorate’s access that requires the pharmacy’s consent, expires, and appears in the pharmacy’s own log.
5. Operator actions — Three actions that support performs by hand today, rewritten as typed catalog entries with preconditions, dry runs and undo, and which of them need a second person.
6. Offboarding and switching runbook — The export format and its data categories, the Data Act clocks as workflow timers, how controlled-drug records are kept for the statutory period, and the deletion record.

Rubric:

Where you now stand

You can now build the control plane that a SaaS architecture only sketches: a directory of every organization, onboarding measured by the customer’s success, an identity model in which people belong to several customers and leave cleanly, entitlements and configuration rolled out like code, internal tools that are typed and audited, and a lifecycle that suspends without harming the innocent and lets customers leave with proof of erasure. Verified Resale tested each of these with organizations that are neither tenants nor organizers, the partners whose machines call Encore’s API every minute. Those partners are the beginning of something larger. Once other companies build their businesses on Encore’s resale pool, Encore is running an ecosystem as well as selling a product, with a marketplace to design, a partner program to govern and economics to keep honest, and inside the company the same question arises about code that many teams share. Chapter 8 takes up business platforms, partners and InnerSource.

References

• Craig D. Weissman, Steve Bobrowski — The Design of the Force.com Multitenant Internet Application Development Platform. Proceedings of ACM SIGMOD, 2009.
• salesforce.com — The Force.com Multitenant Architecture: Understanding the Design of Salesforce.com’s Internet Application Development Platform. Whitepaper, 2008 — free online.
• Tod Golding — Building Multi-Tenant SaaS Architectures. O’Reilly, 2024.
• Atlassian — Post-Incident Review on the Atlassian April 2022 Outage. Atlassian Engineering, 2022 — free online.
• European Parliament and Council — Regulation (EU) 2023/2854 (Data Act). Official Journal of the EU, 2023 — free online.
• Chunqiang Tang, Thawan Kooburat, Pradeep Venkat, Akshay Chander, Zhe Wen, Aravind Narayanan, Patrick Dowell, Robert Karl — Holistic Configuration Management at Facebook. Proceedings of SOSP, 2015.
• Ruoming Pang et al. — Zanzibar: Google’s Consistent, Global Authorization System. USENIX ATC, 2019 — free online.
• P. Hunt (ed.), K. Grizzle, M. Ansari, E. Wahlstroem, C. Mortimore — RFC 7644: System for Cross-domain Identity Management: Protocol. IETF, 2015 — free online.
• P. Hunt (ed.), K. Grizzle, E. Wahlstroem, C. Mortimore — RFC 7643: System for Cross-domain Identity Management: Core Schema. IETF, 2015 — free online.
• Microsoft — Customer Lockbox requests. Microsoft Learn — free online.
• Okta Security — Unauthorized Access to Okta’s Support Case Management System: Root Cause and Remediation. Okta, 2023 — free online.
• Hector Garcia-Molina, Kenneth Salem — Sagas. Proceedings of ACM SIGMOD, 1987.
• Oireachtas — Sale of Tickets (Cultural, Entertainment, Recreational and Sporting Events) Act 2021. Irish Statute Book, 2021 — free online.
• European Parliament and Council — Regulation (EU) 2022/2065 (Digital Services Act). Official Journal of the EU, 2022 — free online.
• David Oppenheimer, Archana Ganapathi, David A. Patterson — Why Do Internet Services Fail, and What Can Be Done About It?. USENIX USITS, 2003 — free online.
• Pete Hodgson — Feature Toggles (aka Feature Flags). martinfowler.com, 2017 — free online.