Software Architecture in the AI & Cloud Era

← the whole book on one page

Chapter 14: Multi-Tenant SaaS Architecture

One system, many customers, and every customer convinced it was built for them alone.

It began with an inbound email. A national venue chain wants “Encore, but ours”: its own brand on every page a fan sees, its own data kept apart from everyone else’s, its own compliance regime honored, and all of it running on Encore’s machinery. Until now, Encore’s system has served exactly one business: Encore. Saying yes means becoming a different kind of company, one whose product is the ticketing system itself, rented to many businesses at once. This is the turn to software as a service (SaaS), and it reaches much further than deployment.

It is tempting to treat SaaS as a hosting arrangement: run the software for the new customer and send an invoice each month. The deeper change is multi-tenancy, in which one running system serves many customers, called tenants, and that is a defining architectural constraint, one that shapes the data model, the request path, operations and the business itself. Three properties matter most. Isolation means one tenant can never see, or tamper with, another tenant’s data. Fairness means one tenant’s busy on-sale cannot starve the others of capacity. Per-tenant economics means Encore knows what each tenant costs to serve, and so whether each one is profitable. All three are cheap to design in and expensive to retrofit. By the time we retrofit them, every table, queue, cache key and log line has already been written without a tenant in mind, and each must be found and changed while the system is live.

None of this is new, though the scale is. Chapter 11 told the story of the computer utility, and multi-tenant SaaS is that utility in commercial form. The time-sharing pioneers wrestled with three problems: protecting one user from another, dividing a scarce machine fairly among its users, and billing each user for what they consumed. Those are this chapter’s problems under new names. Protection has become isolation, fair division has become fairness, and billing has become per-tenant economics. What has changed is where the problems live. The pioneers could enforce their answers in one machine’s operating system. Encore must enforce them across fleets of machines, in many services, databases and regions, none of which knows anything about tenants unless we teach it.

This chapter also leads directly to the next. A SaaS product is a platform with billing attached: it offers shared capabilities to many customers who serve themselves, which is exactly what an internal platform offers a company’s own teams, minus the invoice. So the control plane, self-service onboarding and tenant-aware operations we build here are what Chapter 15 generalizes to internal platforms, with Encore’s engineering teams as the tenants. The tenant also reaches back into earlier chapters: into identity (Chapter 10), because every request must say whose it is; into partitioning (Chapter 4), because the tenant becomes a natural key for dividing data and load; into cost (Chapter 11), because each tenant’s share of the bill must be measured; and into AI spend (Chapter 12), because model calls must be budgeted per tenant too. That is why tenancy is less a component than a dimension. We cannot draw it as one box on the architecture diagram, any more than we can draw security as one box; it is an axis that passes through every box, and each one must be designed to respect it.

14.1 The SaaS Mindset

Control plane, application plane, and deployment models

Tod Golding organizes SaaS around one split, and the rest of his thinking follows from it. On one side is the application plane, the ticketing machinery that tenants actually use: catalogs, queues, seat maps, orders. When a fan joins the queue for a venue’s on-sale and buys two seats, every step of that runs in the application plane. On the other side is the control plane, the machinery that manages the tenants themselves: onboarding, identity, configuration, billing and per-tenant operations. The control plane sells no tickets and shows no seat map, so no fan will ever notice it. Yet every new tenant is onboarded through it, every invoice is produced by it, and every support case starts with it, because answering “what is wrong for this venue?” means first knowing who the venue is, what tier it bought and where it runs. The application plane is what tenants pay for; the control plane is what makes it possible to serve many of them at once.

A city is a good way to picture the split. The application plane is the streets, shops and homes, where daily life happens. The control plane is city hall, with its land registry, building permits and utility meters. Nobody moves to a city because of its registry. But a city without one cannot say who lives where, who owes what, or whose water to shut off when the bill goes unpaid, and the same is true of a SaaS product without a control plane. The analogy breaks in a useful place, though. A registry changes slowly, through paperwork, and nobody consults it while crossing the street. A control plane is consulted constantly: every request a tenant’s staff or fans make must be traced to its tenant, and the answer to “whose request is this?” has to arrive in milliseconds, on every request, not in the weeks a land deed takes. City hall can afford to be slow; the control plane cannot.

Figure 14.1 — The two planes: the control plane’s five services (onboarding, tenant identity, configuration and tiering, billing and metering, tenant-aware operations) beside the application plane they manage. Companies that skip building a control plane still have one, made of runbooks, spreadsheets, and a heroic ops engineer named something like Dana.

The control plane decides who a tenant is; the next question is where that tenant’s part of the application plane runs. Tenants are deployed on one of three models. Chapter 10 introduced them as isolation choices, using the silo/pool/bridge vocabulary of Golding and AWS’s SaaS guidance, and they are worth restating in Encore’s terms. Silo gives each tenant a whole stack of its own: separate compute, separate databases, nothing shared, so the venue chain’s on-sale cannot slow anyone else’s and its data never sits beside another tenant’s. Pool runs one shared deployment for many tenants, so the same services and the same tables serve every venue, and every request is scoped by its tenant so that one venue’s queries see only that venue’s seats and orders. Bridge (close to what Golding calls mixed mode) mixes the two layer by layer. Its most common form is pooled compute over siloed data: the services are shared, but each tenant’s records live in a database of its own.

The choice among these models is made per tenant and per tier, not once for the whole company, and Encore will run all three at the same time. Small self-service venues will share a pool, professional tenants will sit on a bridge, and the venue chain will get its own silo; Section 14.5 puts a price on each. This is why I distrust a company that calls itself “a pool shop” or “a silo shop.” It has taken a trade-off that should be weighed again for each new kind of customer and turned it into part of its identity. A decision can be revisited at the next design review, but an identity is defended, so when the first customer arrives whose needs the house model cannot meet, that company will argue with the customer before it argues with itself.

Recap. The control plane manages tenants and the application plane serves them. No tenant ever sees the control plane, yet it is as much a part of the product as the application plane, because without it the company cannot onboard, bill, configure or support anyone. Silo, pool and bridge are a menu, chosen per tier rather than once for the company, and one company can run all three.

Exercise 14.1. Inventory what your company would need in a control plane today. Take Figure 14.1’s five boxes one at a time and, for each, write down whether it exists as software or as Dana: a person, a runbook or a spreadsheet doing the job by hand. Count how many boxes fall on each side.

14.2 Tenant Lifecycle

Onboarding and tenant identity

A tenant’s life begins at onboarding, so that is where we begin too. We measure onboarding the way Chapter 7 measured developer experience (DX): Chapter 7 counted the minutes from a developer’s arrival to a first successful API call. Here the measure is time from signature to first sale: the clock starts when a venue signs up and stops when the first fan buys a ticket from it. For Encore’s self-service tier, that time must be minutes. A small venue that signs up on a Tuesday evening, hoping to put its Friday gig on sale, will not wait until someone in operations reads a ticket on Wednesday morning.

Minutes are only possible if no person stands in the path. Onboarding therefore has to be an automated, observable workflow running on Chapter 6’s engine, with four steps: provision the tenant’s resources, configure its settings, seed its starting data (the venue, its seat map, a first event) and verify that a test sale works end to end. Each step can fail partway through, and over hundreds of onboardings some certainly will: the database is created but the storefront domain is not, or the seat map is seeded but payment configuration times out. So each step carries a compensation, the action that undoes it, and the workflow either finishes or cleanly rolls itself back, leaving no half-built tenant behind. Because the workflow is observable, anyone can see which step a stuck tenant is on. A checklist in Confluence cannot do any of this. It depends on a person who is awake, undistracted and following every line in order, and a self-service tier that onboards venues at midnight cannot depend on such a person.

The enterprise tier adds steps, among them single sign-on (SSO) federation with the customer’s own identity provider, custom domains and data residency, but the same pipeline runs them. The temptation is to treat this customer differently. The deal is large, the customer is important, and it feels respectful to have senior engineers build their environment by hand. The trouble is that a hand-built tenant is a one-off, and one-offs are what disaster-recovery drills rebuild wrongly. When the environment must be recreated, during a drill or a real outage, nobody has a pipeline that knows how it was made, only someone’s half-remembered notes. A tenant the pipeline built can be rebuilt by the pipeline, exactly. Treating the big customer’s environment as special is precisely what makes it fragile.

Data residency shows why the record matters. Residency is a placement the control plane writes down at onboarding: the tenant’s region, and the cell within that region that will serve it (Section 14.3 develops cells fully). From that one record follows where the tenant’s logs, backups and AI calls are allowed to go. Suppose the venue chain’s contract requires its data to stay in the EU. If the control plane records that fact, every service that writes a backup or calls an AI model can look it up and obey it. If instead the fact lives in the heads of the engineers who signed the deal, one of them will eventually forget it, perhaps while adding a new log pipeline to a US region. Recorded once, a rule can be enforced everywhere; remembered by people, it will eventually be forgotten by one of them.

Onboarding is only the first event in a tenant’s life, and the same pipeline owns the rest. Tier changes move a tenant between deployment models: from pool to silo when a busy venue upgrades and needs its own stack, and back again when it downgrades. Either way it is a planned, verified data migration, run by the workflow engine with the same steps and compensations, rather than a weekend of manual copying. Suspension blocks access without deleting anything, for an unpaid invoice or a security hold, so that when the invoice is paid or the hold lifts, the tenant returns exactly as it was. Offboarding exports the tenant’s data in a documented format, deletes every copy (including the ones in backups, caches and analytics stores), and produces the deletion evidence that the customer’s auditors will ask for. A system designed only for a tenant’s birth will improvise everything that follows, and improvisation is exactly what we cannot afford when the task is moving, freezing or destroying a customer’s data.

Once a tenant is live, identity does the daily work, and it does so under one rule that prevents a whole category of catastrophe: tenancy is resolved once, at the edge, and carried as trusted context from there on. “At the edge” means at the API gateway, the first component a request meets. That is where Encore decides, a single time, which tenant the request belongs to. How the gateway decides depends on who is asking.

Venue staff are the simple case. Each member of staff belongs to exactly one tenant, so the identity token issued at login can carry the tenant directly. When the box-office manager at a venue chain logs in, the token says tenant: venue-chain-a, and every request she makes is stamped with it. Fans are different, because one fan buys from many venues: tonight from the venue chain, next week from a small club in the self-service tier. A fan’s token therefore proves only who the fan is, and the tenant comes from where they are shopping. A purchase on the storefront tickets.venuechain.com belongs to the venue chain’s tenant because of the storefront’s domain, whoever the fan is. In both cases, the gateway checks identity and tenant together, and only then lets anything else run.

This rule is an old principle, finally made practical. Complete mediation, Saltzer and Schroeder’s principle from Chapter 10, requires that every access to every object be checked for authority. In a single program that is hard enough; across many services it is impossible if each service has to work out for itself who the caller is and which tenant they act for. Resolving tenancy at the edge solves this by giving the check a single home. The gateway does it once, carefully, and every service downstream relies on its result instead of repeating it, slightly differently, in every service.

The failure mode is downstream code deciding tenancy for itself. One service parses a venue ID out of the URL. Another looks up the tenant in a table on every request. A third simply assumes that “the service knows” because it has only ever been called for one tenant. Each choice is reasonable where it is made. Taken together, they give the system as many definitions of tenancy as it has services, and those definitions will drift apart. Sooner or later, one of them will disagree with the others, perhaps after a URL format changes or a lookup table goes stale, and that service will serve one tenant another tenant’s data. No single line of code will look wrong; the error lives in the disagreement between services.

So, from the gateway inward, tenant context travels the way Chapter 5’s correlation ID does: in every call, every event and every log line, set once and never recomputed. A service that needs the tenant reads it from the context it was handed; it never derives the tenant on its own. The payoff goes beyond safety. Once the tenant is present everywhere, questions that would otherwise be projects become simple queries over data Encore already collects. How many errors did the venue chain see last night? How many API calls did each self-service venue make this month, and therefore what should each one be billed? Is one tenant consuming more than its share of the queue? Per-tenant metrics, limits and bills stop being engineering projects and become queries.

Recap. Onboarding is a compensating workflow on Chapter 6’s engine. It is the same pipeline for every tier, measured in minutes from signature to first sale, and it also carries each tenant through tier changes, suspension and offboarding, so nothing in a tenant’s life is hand-built or improvised. Tenancy is resolved once, at the edge, from the staff token or, for fans, from the storefront, and it then propagates like a correlation ID through every call, event and log line. That propagated context keeps tenants’ data apart and makes the rest of this chapter measurable.

Exercise 14.2. Sketch Encore’s self-service onboarding as a Chapter 6 process model. List the steps in order, give each step its compensation (the action that undoes it if a later step fails), and describe the verification that flips a tenant to “live.”

14.3 Building Multi-Tenant Services

Tenant context in shared services

Section 14.2 resolved the tenant at the edge. The next question is what the services behind the edge should do with that knowledge, and here the danger is gradual, which is exactly what makes it dangerous. It begins with one reasonable exception: an enterprise customer needs longer seat holds, so someone wraps the hold logic in if (tenant.tier == "enterprise"). Next quarter brings another exception, and the quarter after that another, each one sensible on the day it was written. A few years on, such conditionals are scattered through the business logic, and nobody can say with confidence what a given path does for a given tenant. Each branch is a small piece of accidental complexity, in Brooks’s sense: complexity that comes from how we built the system, not from the problem of selling tickets. And the branches compound instead of adding up, because each one multiplies the paths a test must cover. Ten independent tier checks along one checkout flow describe, in principle, 1,024 different checkouts, and no test suite visits them all.

The remedy is to give tenancy three homes, each with one job. Tenancy is resolved once, at Section 14.2’s edge, so the system has a single answer to “whose request is this?” It is enforced in shared infrastructure layers (the data-access library, the rate limiter, the queues), so no business function has to remember to check it. And it is expressed in business code only as configuration: values the code reads, never conditions the code tests. Encore’s seat hold, the five-minute claim of Chapter 7, has a time to live (TTL) that is a per-tenant configuration value. The hold code reads that value and applies it, with no tier branch around it, so the same lines run for a small club and for the venue chain’s arenas, and one test of the hold logic covers every tenant. This is Parnas’s criterion from Chapter 1 applied to tenancy. How the tiers differ is a design decision likely to change, since sales and product will keep inventing tiers and exceptions, so we hide that decision behind one interface, the tenant’s configuration, instead of letting it leak into every function that touches a seat.

The same discipline reaches into the data. Partitioning by tenant reuses Chapter 4’s machinery, with the tenant as the partition key, and there are three standard ways to draw the lines. Each one trades the blast radius of a mistake, meaning how many tenants a single error can expose, against the bill for running it:

Figure 14.2 — The three partitioning schemes, compared by mechanics, blast radius and bill. The pooled row’s blast radius is why Chapter 10 insisted that the tenant predicate (the filter that limits a query to one tenant’s rows) be centralized in the object-relational mapping (ORM) layer, in row-level security, or in the data-access library. With the predicate in one place, 10,000 queries rely on one guarded implementation instead of on 10,000 developers’ memories.

Centralizing the predicate makes the mistake unlikely, but not impossible. A forgotten WHERE clause is a very small mistake for so large a consequence: one missing line, and a venue’s box-office report lists another venue’s customers. A guarantee that depends on every developer remembering something, every time, is only a hope with good manners. So Encore adds a second line of defense below the application, in the database itself. With tenant-scoped credentials, the database session serving a request is bound to that request’s tenant, and the database filters every query to that tenant’s rows on its own authority. If the code above forgets its predicate, the query still returns only the right tenant’s data. The application can still have bugs, but this particular bug can no longer leak one tenant’s data to another. The credentials put a floor under the application, and the floor holds whatever the code above it forgets.

In Postgres-style row-level security, two details decide whether that floor actually holds, and both are easy to miss. The first is ownership. Postgres does not apply row-level policies to a table’s owner by default, so an application that connects as the role owning its tables skips every policy, silently. The application’s role must therefore not own the tables, or the policy must be forced on owners as well. The second is connection pooling. The policy usually reads the current tenant from a database setting, and a pooled connection is handed from one request to the next. If the tenant is set for the whole session, the next request to borrow that connection inherits the previous tenant and sees its rows. Setting it with SET LOCAL scopes it to the current transaction, so it vanishes when the transaction ends. Both details share a treacherous property. With one tenant in the test database everything passes, because there is no other tenant’s data to leak. They fail only once two tenants are present, and they fail quietly, by returning plausible rows that belong to someone else.

Isolation keeps tenants from seeing each other’s data; fairness keeps them from starving each other of capacity. The noisy-neighbor problem, one tenant consuming so much shared capacity that the others suffer, is existential for Encore and not merely annoying, because of what its tenants do. One tenant’s on-sale is a Chapter 4 spike aimed at shared infrastructure. When a pooled venue puts a popular tour on sale at ten o’clock, its fans arrive at the same workers, queues and databases that every other pooled venue relies on. A small club selling a Tuesday jazz night at the same moment finds its checkout crawling, for reasons it cannot see and did nothing to cause. Anyone who has lived in an apartment building knows this shape: one neighbor’s party becomes everyone’s evening. Buildings answer with two kinds of remedy, rules about behavior (quiet hours) and structure (thicker walls), and Encore uses both.

The rules come first, applied at each layer the spike passes through. At the edge, per-tenant rate limits cap how many requests per second one tenant’s traffic may send, and concurrency quotas cap how many of its requests may be in flight at once, so a surge is turned away early and cheaply instead of filling the system. In the shared workers behind the edge, fair queueing serves tenants in turn instead of strictly in arrival order, so the jazz club’s one checkout does not wait behind the tour’s ten thousand. And the Sale Gate’s admission control, which already meters fans into an on-sale at the rate Seat Inventory can sustain, is made tenant-aware, so one tenant’s on-sale cannot take admission capacity that belongs to the others. When the rules fall short, because a tenant is busy not occasionally but chronically, the bridge model provides the thicker walls: the tenant moves to dedicated capacity, at a price Section 14.5 names.

At scale these tools combine into a larger structure, the cell, which Chapter 2 introduced and which multi-tenancy puts to full use. A cell is a complete, self-contained instance of the application plane, with its own compute, stores and queues, serving a fixed subset of tenants. The control plane routes each tenant to exactly one cell, so every request from a given venue lands in the same cell every time. A cell is Chapter 4’s bulkhead at full size, and it earns its place by being three units at once. It is the unit of blast radius: a bad deploy, or a poison-pill tenant (one whose data or traffic reliably crashes whatever serves it), takes down one cell, and the product stays up for the tenants of every other cell. It is the unit of scale: Encore grows by adding cells of a known, tested size instead of enlarging one deployment into sizes nobody has tested. And it is the unit of operations: we canary a release, drain traffic or run a game day on one cell at a time, so every risky act has a small, bounded audience.

Here the bulkhead image falters, and where it falters is instructive. A ship’s bulkheads are meant to seal each compartment from the next, but cells share a good deal: a control plane, an identity provider and a deploy pipeline. A failure in any of those crosses every bulkhead at once. If the identity provider goes down, fans cannot sign in to any cell; if the pipeline ships a bad build to every cell together, the cells’ independence buys nothing. Cells contain the application plane’s failures and leave the control plane’s to us, which is why the shared pieces deserve the most conservative engineering in the system.

Cells divide tenants into fixed groups, which means a noisy tenant still shares everything with its cell-mates. Shuffle sharding, an AWS technique described by Colm MacCárthaigh, improves fairness further by making those groups overlap differently for each tenant. Instead of giving each tenant one fixed group of resources, it gives each tenant a small random combination of resources drawn from a larger pool. Suppose a cell has eight workers and assigns each tenant two of them. There are 28 different pairs, so two tenants rarely hold the same pair; most share one worker or none. Now let one tenant’s on-sale flood its two workers. Every tenant that shares one of those workers loses part of its capacity but keeps its other worker, and tenants that share neither notice nothing. A noisy neighbor therefore degrades a sliver of many tenants’ capacity instead of all of a few tenants’ capacity, and nobody but the noisy tenant loses everything. Spreading harm thinly sounds like a modest goal, but it is the difference between many fans seeing a slower checkout and a few venues seeing no checkout at all.

Cells are not a fourth deployment model beside silo, pool and bridge; they are a container for the shared ones. A cell can hold pooled or bridged tenants (Golding treats a similar unit as a pod), and what it changes is the worst case. Here is Section 14.1’s menu again, compared by blast radius, with cells as the unit that scales it:

Figure 14.3 — Deployment models compared by blast radius. Cells shrink the pooled row’s worst case from every tenant to one cell’s tenants, without paying silo prices.

Encore’s end state has three shapes, one for each kind of tenant. Its 500 self-service venues run pooled, in serverless cells of about 50 venues each, so a failure in one cell reaches roughly a tenth of them instead of all. Professional tenants run bridged, in Kubernetes cells where compute is shared and each tenant has its own database. The venue chain runs in its own siloed stack. Tying the three together is the control plane, which already records where every tenant lives and now also becomes the cell router, sending each request to the cell its tenant belongs to. To return to Section 14.1’s city: the registry that knew who lives where now directs the traffic too.

Recap. We resolve tenancy once at the edge, enforce it in shared layers, and express the differences between tiers as configuration that code reads, never as branches that code tests. We choose a partitioning scheme by the blast radius we can afford, centralize the tenant predicate, and back it with tenant-scoped credentials so that a forgotten WHERE clause cannot leak data. Quotas and fair queues keep one tenant’s on-sale from starving the others, shuffle sharding spreads the remaining harm thinly, and cells cap how many tenants one failure can reach. A tenant that outgrows these rules pays for dedicated capacity.

Exercise 14.3. Search a codebase you know for tier or tenant conditionals in business logic, such as checks on a tenant’s tier or ID. Pick the worst one (the most widely copied, or the hardest to test) and redesign it as configuration: name the configuration key, list its per-tenant values, and show how the code reads it without branching.

14.4 SaaS on Real Stacks

So far the deployment models of Section 14.1 have been ideas on a whiteboard: silo, pool and bridge, drawn as boxes. Ideas meet reality on a substrate, the platform the code actually runs on, and each substrate has its own walls, its own leaks and its own levers for fairness. Two substrates dominate today, Kubernetes and serverless functions, and the same model lands differently on each. Encore’s end state uses both: serverless cells for the 500 self-service venues and Kubernetes cells for the professional tier. So we take them one at a time, asking the same three questions of each: how strong is the wall between tenants, what leaks through it, and how do we keep one tenant from crowding out the rest?

Kubernetes SaaS. On Kubernetes the natural unit of tenancy is the namespace, a named partition of a cluster. Give each tenant its own namespace, attach a resource quota (a ceiling on how much CPU and memory it may claim) and network policies (rules saying which pods may talk to which), and tenants can no longer see or starve one another by accident. That is soft isolation, and the word “soft” matters. Every namespace’s containers still run on the same Linux kernel on each node, and every namespace is managed by the same cluster control plane (the API server and scheduler that run the cluster, not to be confused with Encore’s own SaaS control plane). A bug in the kernel, or a container that escapes its sandbox, crosses every namespace at once.

A namespace is a partition wall of plasterboard. It gives privacy and quiet to neighbors who mean no harm, and it is no defense against a neighbor with a hammer. A typical professional tenant is the first kind: a venue running its own on-sales, with no wish to break into anyone else’s. A tenant that is hostile, or merely untrusted (one running code Encore did not write), or regulated (one whose auditors require proof of separation and will not accept good intentions) needs a stronger wall. The options rise in strength and in cost. A sandboxed runtime puts an extra isolation layer, such as a lightweight virtual machine, between each container and the shared kernel. A virtual cluster gives the tenant its own API server and control plane while still sharing the nodes underneath. A dedicated cluster shares nothing at all. Siloed tiers, such as the venue chain, sit at the strong end, with dedicated node pools (machines reserved for one tenant) or clusters of their own.

Walls stop tenants from reaching one another; they do not stop one tenant from using more than its share. On Kubernetes, the noisy-neighbor fairness of Section 14.3 is enforced by the scheduler, through three settings. Requests tell it how much CPU and memory a workload is guaranteed, so a node is never promised to more tenants than it can serve. Limits cap how much a workload may take, so one venue’s on-sale cannot consume a whole node. Priority classes decide who yields when a node runs short, so that, for example, checkout for a paying tier is protected while a batch report waits. Together they turn “one neighbor’s party becomes everyone’s evening” into a rule the platform enforces rather than a hope.

Serverless SaaS. With serverless functions the picture changes, because we no longer manage the machines at all. The provider runs each function invocation inside an execution environment, a small isolated sandbox, so the wall between tenants is the wall between those environments. The catch lies in how providers keep functions fast. Starting an environment from scratch (a cold start) is slow, so once an invocation finishes, the provider keeps the environment warm and reuses it for the next one. In a pooled function, that next invocation may well belong to a different tenant. Anything the function stored in a global variable, or in a module-level cache, is still there when the next request arrives.

That is why pooled functions must keep no tenant state in globals. Suppose a function caches “the current venue’s pricing rules” in a global to save a database lookup. A request for venue A fills the cache; seconds later the same warm environment serves venue B, which sees A’s prices and perhaps A’s seat holds. A variable that outlives one request is a message left for whichever tenant happens to arrive next. The defense is a discipline: all tenant state lives in the request’s own scope, or in caches keyed explicitly by tenant ID, and nothing is ever shared by accident.

AWS Lambda’s tenant isolation mode (2025) moves part of that discipline from the programmer to the platform. The caller passes a tenant ID with each invocation, and Lambda dedicates execution environments to that tenant, so a warm environment that served venue A is reused only for venue A. Global state can then no longer leak across tenants, though it can still leak between two requests of the same tenant, so the discipline above stays good practice.

Fairness and cost work differently on serverless as well. There is no scheduler to tune; the lever is concurrency, the number of invocations allowed to run at once. A per-tenant concurrency limit ensures that one venue’s on-sale, which might want thousands of simultaneous invocations, cannot exhaust the account-wide pool and leave the other venues in its cell waiting. On cost, serverless looks as if it solves attribution for free, since the provider bills per invocation. Yet the bill arrives per function, not per tenant: a pooled function serves every venue in its cell, about 50 of them,, and the invoice cannot tell them apart. To know what each venue costs, every invocation must be metered and tagged with its tenant, and only then can Chapter 11’s cloud financial operations (FinOps) reports attribute cost per tenant.

Operations, too, gains a dimension. Until now, dashboards, alerts and service-level objectives (SLOs) have measured the system as a whole. In a multi-tenant system each of them needs a tenant axis: the same error rate, latency and budget, but broken down by tenant. Adding that axis is cheap at Encore, because Section 14.2 made tenant context propagate like a correlation ID, so it is already in every call, every event and every log line. Slicing a metric by tenant is then a query over data Encore already collects, not a new instrumentation project.

The tenant axis changes the question the on-call engineer asks. Without it, the question is “is the system up?” With it, the question becomes “is it up for whom?” The difference is not academic. Imagine a pooled cell answering 1,000,000 requests in a day at 99.99% success, which means only 100 failures, a number that looks perfectly healthy on any dashboard. Now suppose a small venue in that cell, perhaps after a bad configuration change, sends just 100 requests that day, and every one fails. The system-wide number looks healthy, the alert stays quiet, and for that venue Encore has been down all day. Nobody will notice until metrics are sliced by tenant, or until the venue’s manager phones support. Averages are kind to systems and cruel to the customers hidden inside them.

So Encore tracks health per tenant and gives each paying tier per-tenant error budgets, so that one venue exhausting its budget registers as an incident even when the cell’s overall number is fine. Its support tooling can likewise reconstruct one tenant’s experience: given a venue and a time window, it gathers that venue’s requests, errors and traces, so that when a box-office manager says “your system was broken for us on Friday night,” support can see exactly what that venue saw, rather than a cell-wide graph that says all was well.

Recap. Kubernetes namespaces give soft isolation, because tenants still share a kernel and a cluster control plane; hard isolation requires sandboxed runtimes, virtual clusters or separate clusters, and the scheduler’s requests, limits and priority classes keep neighbors fair. Serverless isolates per execution environment, but warm reuse means pooled functions must hold no tenant state in globals, and per-tenant cost still needs tenant-tagged metering. Encore runs both, and slices every dashboard, SLO and alert by tenant so that incidents cannot hide inside averages.

Exercise 14.4. Take one alert your team owns and rewrite it per tenant. State the new threshold, say which tenants (or tiers) it covers, and then replay last month’s data against it: which tenants would have breached it, and what did the averaged version hide?

14.5 The Business of Multi-Tenancy

Tiers and pricing

So far this chapter has asked how tenants are deployed, isolated and operated. Sooner or later the chief financial officer (CFO) asks a blunter question of all that machinery: what does each tenant pay, and what does each one cost us to serve? Tiering is where Section 14.1’s menu of silo, pool and bridge meets that question, and it is the reason architecture and packaging must be designed together. Every tier makes a promise about isolation and reliability, and every such promise is a deployment model with a running cost. If the product team prices a tier without knowing what it costs to run, or engineering builds an isolation model that nobody can sell, the mismatch eventually shows up on an invoice, as a tier that loses money or an isolation model nobody buys. Encore’s tier sheet puts both halves of the decision on one page:

Figure 14.4 — Tiering as co-design: each row pairs a deployment model, an isolation level, an SLO and an AI allowance with the logic of its price. Read it with Chapter 11 in mind: a row can be priced only because per-tenant cost is measurable. Metering (counting requests, storage and AI tokens per tenant) is the control-plane service that turns margins from folklore into numbers.

It is worth reading the sheet row by row, because each row is an argument. The self-service tier is pooled, so small venues, about fifty to a serverless cell, share the same compute and the same tables, kept apart by a tenant predicate on every row and by quotas at the edge. Sharing is what makes it cheap, and so the price can be low; the cost of that sharing is that its 99.9% objective is a single, shared promise, with one error budget spent by everyone together. The professional tier is a bridge: compute is still pooled, which is where Encore’s margin comes from, but each tenant has its own database, which is what lets a professional tenant sleep at night knowing its customer lists never sit in a table beside a rival’s. It also gets its own error budget, so one tenant’s bad week does not use up another’s. The enterprise tier is a silo, and here the logic inverts: the venue chain is not paying for ticketing features it could get on the cheaper tiers, it is paying for the separation itself, with an objective written into its contract and measured on its slice alone.

None of these prices would survive a hard look without the meter beneath them. Suppose Encore sells the professional tier “its own AI budget” but cannot say how many AI tokens a given professional tenant consumed last month. Then the price of that feature is a guess, and the margin on it is a story people tell in planning meetings, which is what I mean by folklore. Metering is what replaces the story with a number. Because Section 14.2’s tenant context already arrives in every call and every log line, the meter is mostly a matter of counting what the system already knows, per tenant, and handing the totals to billing.

Enterprise deals then bring a checklist of their own, and it reads like a list of features while really being a list of architectural commitments. Consider what each item asks of the system.

Single sign-on (SSO) through Security Assertion Markup Language (SAML) or OpenID Connect (OIDC) federation lets the venue chain’s staff sign in to Encore with their own corporate accounts, checked by the venue chain’s own identity provider. For Encore, that means the identity layer of Section 14.2 can no longer assume one identity provider for everyone; it must hold per-tenant identity providers, each with its own trust configuration, and route each sign-in to the right one based on which tenant the person belongs to. System for Cross-domain Identity Management (SCIM) provisioning lets the customer’s human resources (HR) system create and revoke accounts in Encore automatically, so that when a box-office manager leaves the venue chain, their access to Encore disappears the same day without anyone filing a ticket. That is an inbound API, called by software Encore did not write, reaching straight into Encore’s identity model, and so it needs all of Chapter 7’s contract discipline. Audit logs become a product surface: visible to the tenant, exportable to the tenant’s own tools, and immutable, so that when the customer’s auditors ask who changed a price or refunded an order, the answer comes from a record nobody (Encore included) could have edited afterward. It is the Chapter 6 ledger, given a customer-facing user interface.

Customer-managed keys come in two forms. With bring your own key (BYOK), the customer supplies the key that protects its data, and Encore uses it. With hold your own key (HYOK), the key never leaves the customer’s own key service, and Encore must ask that service each time it needs the key. Both push Chapter 10’s key hierarchy to its limit, because the tenant now controls its root. A promise comes attached: if the customer revokes that root, its data must become unreadable everywhere, and not only in the primary database but in every derived copy too, such as caches, search indexes, backups and AI embeddings. This is crypto-shredding, deleting data by destroying the key that unlocks it instead of hunting down every byte, and here it is offered as a contractual feature. The catch is that crypto-shredding only reaches copies encrypted under the tenant’s hierarchy. A search index built from the venue chain’s event descriptions but encrypted with Encore’s own key would stay perfectly readable after revocation, and so the feature obliges us to know where every copy of a tenant’s data lives.

Per-tenant rate limits and SLOs turn Section 14.3’s fairness machinery into line items in a contract. A rate limit that was once an internal safeguard becomes something the customer has bought, and an objective that was once an engineering target becomes a promise with consequences when it is missed. Notice that no pricing toggle delivers any item on this list; there is no switch in the billing system that turns a pooled tenant into an enterprise one. Each item is structure that must already exist in the identity layer, the key hierarchy, the data stores or the edge. The venue chain understood this. Its security questionnaire arrived before its signature did, and it read like a structural survey of a building they were about to move into: they were checking the foundations and the wiring before signing the lease.

Migration. Many SaaS products do not start multi-tenant; they start as software for one customer and are converted later. The conversion follows Chapter 3’s strangler approach, in which the new structure grows around the old one and replaces it piece by piece while the system keeps running. Here order matters more than speed. Tenant context goes in at the edge first, so that every request carries a tenant tag while there is still only one tenant. The tag costs almost nothing, and it makes every later step observable: by the time a second tenant arrives, metrics and logs are already sliced by tenant, and any code path that ignores the tag can be found by looking. Next the tenant predicate is centralized, so that one guarded layer filters every query by tenant instead of each query remembering to. Only then does data consolidate toward the pool, one tier at a time, with silo customers moving last or never. Each step is small enough to verify and to roll back before the next begins. It is rather like renovating a hotel floor by floor while the guests stay on; the analogy breaks only in that a hotel can close a floor for the work, whereas Encore’s venues keep selling tickets throughout. A big-bang rewrite would instead stake every customer on a single cutover night, and if that night went wrong during a major on-sale, every venue would fail together.

Generative AI (GenAI) multi-tenancy. Chapter 12 previewed this, and here it becomes doctrine: the tenant boundary does not stop at the database, but extends through prompts, corpora and spend. Each of the three needs the same structural treatment as the rest of the chapter.

Start with prompts. Prompt-building code must make cross-tenant context structurally impossible, because a prompt is just another query, and a model uses whatever context it is handed. It has no notion of tenants. If a rival tenant’s sales figures appear in the context window, the model will summarize them as readily as anyone else’s, and an instruction such as “use only this venue’s data” is a request, not a guarantee. The boundary therefore has to live in the code that assembles the prompt, which draws only on data scoped by the tenant context resolved at the edge.

Corpora come next. An assistant that answers questions from documents first retrieves the relevant ones from an index and then places them in the prompt, so the index is part of the boundary too. It needs per-tenant indexes or tenant-filtered retrieval, so that the venue chain’s sales data cannot reach a rival’s assistant by any path. A shared index with an optional filter is Section 14.3’s forgotten WHERE clause in a new setting, and it fails the same way.

Finally there is spend. AI calls are paid by the token, so they need per-tenant AI budgets, metered like any other resource. An unmetered AI feature demos well and leaks margin in production: in the demo, one person asks a few questions; in production, one tenant’s staff may run thousands of queries a day against a flat subscription, and the cost of those queries lands on Encore’s side of the ledger. Tenancy, as the opening of this chapter promised, is a dimension of everything, and that includes the newest things we build.

Recap. Each tier pairs a deployment model with a price, and per-tenant metering is what makes those prices defensible instead of guessed. Enterprise features such as SSO, SCIM, audit logs, customer-managed keys and contractual SLOs are architectural commitments, not switches in billing. We migrate a single-tenant system by introducing tenancy at the edge and working inward, step by reversible step. AI features inherit the tenant boundary across prompts, corpora and spend, and, like everything else in this chapter, that boundary is held by structure, not by good intentions.

Exercise 14.5. Draft your product’s version of Figure 14.4, real or hypothetical: one row per tier, with its deployment model, isolation, SLO, AI allowance and the logic of its price. Then find the cell you cannot fill because the metering that would justify it doesn’t exist. That cell belongs on the control-plane backlog; write it down as a backlog item naming what must be measured, per tenant.

Kata: The Studio Suite

This kata asks you to do for a smaller, messier company what this chapter did for Encore: take software built for one customer at a time and turn it into one system that serves many. When the venue chain’s email arrived, Encore had one running system to reshape. Metronome has no such luxury, because it already has 180 customers, each running its own copy. That makes it the more common case in practice, and the more instructive one.

Your brief: “Metronome.” Beloved studio-management software (bookings, invoicing, class schedules) for music schools, currently installed per customer: 180 on-prem and virtual private server (VPS) installations, each on its own version, backed up by email reminders. The founders want SaaS: keep the 180 (some contractually on-prem for two more years), add self-service signup for small studios, and land “Allegro,” a 90-school national chain demanding single sign-on (SSO), in-country data residency, and a contractual service-level agreement (SLA). The team is 12 engineers. The codebase assumes one tenant so thoroughly that the database is named after the customer.

Deliverables:

1. Two-plane design — Figure 14.1’s control plane, scoped to what 12 engineers can actually build and run. For each of its boxes, say whether Metronome builds it, buys it, or explicitly leaves it to Dana for now: that is, to a person following a runbook by hand, named openly as a manual step rather than hidden.
2. Tier sheet — Metronome’s version of Figure 14.4, with a row for each of four tiers: self-service, standard, Allegro, and the legacy on-prem tail. For each tier, give the deployment model (silo, pool, bridge, or the customer’s own servers) and what is metered per tenant.
3. Migration plan — the path from 180 one-off installations to a multi-tenant system, in the order this chapter recommends. First, tag every request with its tenant at the edge. Next, centralize the tenant predicate (the “which school owns this row?” filter) in one guarded layer. Then consolidate the data, and say in which order the schools move. At each step, state what “reversible” means: what you would do to undo that step, and what state you would return to.
4. Fairness & isolation — the September enrollment spike, when every school opens enrollment in the same week, which makes it Metronome’s equivalent of an on-sale. Show the per-school quotas, the fair queues in shared workers, and the protection Allegro’s contract guarantees.
5. One architecture decision record (ADR) — “Allegro: silo or bridge?” Write out the margin math (what each option costs Metronome against what Allegro pays) and the operations cost (what each option adds to the 12 engineers’ running workload).

Rubric:

A word on how to approach the brief. The hardest part is not Allegro, though Allegro is the part that will get the most attention in meetings. The hardest part is the sentence about the database being named after the customer. It tells you that the assumption of one tenant runs through the code, the schema, the deployment scripts and the backups, and the migration plan has to unpick it without stranding any of the 180 schools that are paying today. Start there, and the other deliverables will tend to fall into place around it.

Where you now stand

You can now design one system that serves many customers. You split it into a control plane that manages tenants and an application plane that serves them. You resolve each request’s tenant once, at the edge, and enforce it in shared layers. You engineer fairness with quotas and fair queues, contain failures in cells, and price tiers by the isolation each one buys. Now turn the lens around and look at what Encore has built for itself across 14 chapters: the services, pipelines, ledgers, observability and guardrails. Then ask who runs all of that for the teams that depend on it. The question has this chapter’s shape, except that the tenants are now Encore’s own engineering teams, and the product they consume is the machinery they build on. The final chapter answers it with platform engineering and hands you the capstone.

References

• Tod Golding — Building Multi-Tenant SaaS Architectures. O’Reilly, 2024.
• Sarah Wells — Enabling Microservice Success. O’Reilly, 2024.
• Martin L. Abbott, Michael T. Fisher — The Art of Scalability, 2nd ed.. Addison-Wesley, 2015.
• Amazon — The Amazon Builders’ Library — free online.
• Colm MacCárthaigh — Workload isolation using shuffle-sharding. The Amazon Builders’ Library — free online.
• AWS — Reducing the Scope of Impact with Cell-Based Architecture. AWS Well-Architected whitepaper — free online.
• AWS — Tenant isolation (AWS Lambda Developer Guide). AWS, 2025 — free online.