# Tenant isolation in practice: one LLM gateway

[Skip to content](#lm-inhoud)Network/[NL](/en/tenant-isolatie-in-de-praktijk-n-gateway-vele-klanten)EN[Hubhub.llmnet.nlCompare models on task, language, cost and licence.](https://hub.llmnet.nl/en/)[Communitycommunity.llmnet.nlPrompt techniques, patterns and system prompts.](https://community.llmnet.nl/en/)[APIapi.llmnet.nlLLMs in production: rate limits, routing, structured output.](https://api.llmnet.nl/en/)[Consultancyconsultancy.llmnet.nlRolling out AI in an organisation, pilot to production.](https://consultancy.llmnet.nl/en/)[Newsnieuws.llmnet.nlAI developments, explained for the Netherlands.](https://nieuws.llmnet.nl/en/)[Benchmarkbenchmark.llmnet.nlMeasure AI quality yourself, on your own tasks.](https://benchmark.llmnet.nl/en/)[Careersvacatures.llmnet.nlAI roles, salaries and career paths in the Netherlands.](https://vacatures.llmnet.nl/en/)[Learnleren.llmnet.nlAI concepts in plain language, beginner to builder.](https://leren.llmnet.nl/en/)[Guidegids.llmnet.nlRun AI privately on your own Mac, PC, NAS or home server.](https://gids.llmnet.nl/en/)[Directorydirectory.llmnet.nlMapping the AI ecosystem: tools, models, companies.](https://directory.llmnet.nl/en/)[Radarradar.llmnet.nlSignals from X, research and communities for indie developers.](https://radar.llmnet.nl/en/)[Appsapps.llmnet.nlReviews of AI apps and open-source repos, with tips for builders.](https://apps.llmnet.nl/en/)[llmnet.nl — main site](https://llmnet.nl/en/)[](https://x.com/intent/post?url=https%3A%2F%2Fapi.llmnet.nl%2Fen%2Ftenant-isolatie-in-de-praktijk-n-gateway-vele-klanten&text=Tenant%20isolation%20in%20practice%3A%20one%20LLM%20gateway)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fapi.llmnet.nl%2Fen%2Ftenant-isolatie-in-de-praktijk-n-gateway-vele-klanten)[](https://www.reddit.com/submit?url=https%3A%2F%2Fapi.llmnet.nl%2Fen%2Ftenant-isolatie-in-de-praktijk-n-gateway-vele-klanten&title=Tenant%20isolation%20in%20practice%3A%20one%20LLM%20gateway)[](#)[](https://x.com/intent/post?url=https%3A%2F%2Fapi.llmnet.nl%2Fen%2Ftenant-isolatie-in-de-praktijk-n-gateway-vele-klanten&text=Tenant%20isolation%20in%20practice%3A%20one%20LLM%20gateway)[](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fapi.llmnet.nl%2Fen%2Ftenant-isolatie-in-de-praktijk-n-gateway-vele-klanten)[](https://www.reddit.com/submit?url=https%3A%2F%2Fapi.llmnet.nl%2Fen%2Ftenant-isolatie-in-de-praktijk-n-gateway-vele-klanten&title=Tenant%20isolation%20in%20practice%3A%20one%20LLM%20gateway)[](#)

 
# Tenant Isolation in Practice: One Gateway, Many Customers

 By Ivo Donker — compiled with AI assistance (Claude & Gemini)

 In a modern software-as-a-service architecture where dozens or hundreds of business customers (tenants) use shared AI functionality, traditional relational database isolation is no longer sufficient. Once requests flow through a shared runtime proxy or application gateway to external foundation models, a new spectrum of operational, financial, and security risks emerges. A single overactive customer can exhaust the shared upstream token quota, sensitive document fragments can unintentionally end up in shared intermediate storage, and faulty prompts can block global rate limiters.

 A robust LLM gateway acts as the central control point where routing, authentication, cost monitoring, and separation come together. The goal of tenant isolation is to give every connected organization the illusion and guarantees of a fully private AI infrastructure of its own, while the underlying resources are managed efficiently and centrally. For those who want to study the broader theoretical framework and initial data modeling, the overview on [building multi-tenant LLM applications](https://api.llmnet.nl/en/multi-tenant-llm-apps) offers a solid foundation for data architectures.

 
## The four dimensions of isolation in model traffic

 Traditional API gateways focus mainly on URI paths, HTTP statuses, and simple compute quotas. With large language models, however, isolation reaches deeper into the dynamic runtime behavior and the non-deterministic nature of generative systems. A watertight architecture enforces isolation along four separate dimensions:

 First, there is the data and context isolation. Prompts, system instructions, RAG document fragments, tool calls, and generated responses from tenant A must never, under any circumstances, become mixed with the working memory, caches, or contexts of tenant B. This applies not only during network transmission, but especially in aggregated systems such as semantic caches and vector indexes.

 Second, the runtime requires capacity and throughput isolation, also known as preventing the noisy neighborsyndrome. When one customer starts an unplanned batch processing job on tens of thousands of documents, the resulting peak load must not cause increased wait times or HTTP 429 errors for interactive users of other organizations.

 Third, there is financial and budgetary isolation. LLM calls carry variable, real marginal costs per generated and processed token. The gateway must measure usage in real time, attribute it directly to the correct cost center, and automatically block access once a predefined budget ceiling is reached.

 Fourth, we must account for cryptographic and compliance separation. Regulated customers often require that requests are processed exclusively through special contractual endpoints with guaranteed zero retention, or through their own provider accounts (Bring Your Own Key). The gateway must enforce these policy rules dynamically and flawlessly.

 
## Data and cache isolation: the risk of semantic leaks

 Implementing a central response cache is one of the most effective methods for reducing latency and API costs. However, when a gateway uses a semantic cache (where questions with a similar vector representation get the same answer back), a significant risk of data leaks between organizations arises.

 Suppose an HR employee at Company A asks: "What is the departure arrangement for executives under our 2026 policy?". If the generated answer, containing specific trade secrets, is stored in a global vector cache without strict separation, a similar question from an employee at Company B could accidentally receive that same cached answer back. This type of cross-tenant data contamination bypasses all traditional authorization layers within the application logic.

 The only safe approach is to apply hard cryptographic or logical separation within the cache keys and vector spaces. A cache key must never consist solely of the hash of the prompt or the embedding vector, but must always be composed of multiple mandatory metadata elements:

 
 
 
 
 Component | 
 Key / Index Structure | 
 Isolation Mechanism | 
 Failure Mode if Missing | 
 

 
 
 
 Exact HTTP Cache | 
 sha256(tenant_id + ":" + env + ":" + prompt) | 
 Prefix-based Redis namespaces | 
 Full response with company data leaks to another customer | 
 

 
 Semantic Cache | 
 Vector index filtered on metadata tenant_id == X | 
 Isolated vector partitions or metadata pre-filtering | 
 Partial contextual data leaks via similarity match | 
 

 
 RAG Context Store | 
 Row-Level Security (RLS) in relational/vector database | 
 PostgreSQL RLS policies per database role | 
 Retrieval of unauthorized document sections | 
 

 
 Audit Logs | 
 Encrypted with tenant-specific KMS key | 
 Envelope encryption per customer organization | 
 Central administrator can view confidential payloads | 
 

 
 
 

 Besides storage, the validation of generated text must also be able to differ per tenant. Organizations apply varying standards for reliability and source attribution. Anyone who wants to set up automated checks to test for hallucinations and factual inaccuracies per tenant request will find detailed protocols in the guide on [fact-checking AI answers](https://gids.llmnet.nl/en/ai-antwoorden-factchecken) for setting up verification layers.

 
## Noisy neighbors: two-tier rate limiting and throughput management

 External LLM providers enforce hard operational limits on two parameters: requests per minute (RPM) and tokens per minute (TPM). When a central gateway bundles all outgoing connections over a single provider account, all tenants effectively share the same upstream capacity pool. Without active throttling, a single tenant can consume the entire TPM quota within seconds by analyzing a large document in parallel chunks.

 To effectively counter this, a two-tier rate-limiting architecture based on the token-bucket algorithm is used:

 
 
- Tier 1: Local Tenant Bucket (Ingress). Every incoming request from a specific tenant is first checked against that customer's contracted SLA (for example, a maximum of 100 requests and 50,000 tokens per minute). If the tenant exceeds this threshold, the gateway immediately returns a local HTTP 429 status with a clear Retry-After header, without burdening the upstream provider.
 
- Tier 2: Global Provider Bucket (Egress). Requests that pass the tenant check enter a central priority queue that monitors the total outgoing capacity to OpenAI, Anthropic, or Google. If the global provider limit is at risk of being exceeded, the gateway buffers low-priority background tasks in favor of interactive user sessions.
 

 This separation ensures that operational spikes from one individual customer are isolated and dampened locally, guaranteeing the overall stability of the platform for all other customers.

 
## Authentication models: Pooled Gateway Keys versus BYOK

 When designing a multi-tenant gateway, a fundamental architectural choice must be made regarding upstream API keys: do we manage central keys (Pooled Keys) or support Bring Your Own Key (BYOK)? Both scenarios impose different requirements on the gateway.

 In the Pooled Keys model, the SaaS organization owns the contracts with the LLM providers. The gateway stores these master keys in a central secrets vault (such as HashiCorp Vault or AWS Secrets Manager). Incoming requests are authenticated via internal tenant tokens, after which the gateway dynamically injects the required provider credentials. For operational guidelines on automated rotation, auditing, and secure handling of these central secrets, consult the guide on [managing LLM API keys securely](https://api.llmnet.nl/en/api-sleutels-veilig-beheren).

 In the Bring Your Own Key (BYOK) model, enterprise customers supply their own API key, for example from OpenAI or Microsoft Azure. This lets the customer take advantage of their own enterprise discounts, custom data processing agreements, or private network connections. The gateway must securely encrypt these customer keys with a unique Key Encryption Key (KEK) per tenant. During runtime, the gateway decrypts the key exclusively in the volatile memory of the processing process.

 
## Implementation example: Multi-tenant Gateway Middleware

 The TypeScript example below shows robust Express middleware for an LLM gateway. The code demonstrates tenant authentication, throughput-limit isolation via Redis, model-access validation, budget checks, and conditional key injection.

import { Request, Response, NextFunction } from 'express';
import Redis from 'ioredis';

interface TenantProfile {
 id: string;
 allowedModels: string[];
 rpmLimit: number;
 monthlyBudgetEur: number;
 currentSpendEur: number;
 customApiKey?: string;
 enforceZDR: boolean;
}

const redis = new Redis(process.env.REDIS_URL || 'redis://127.0.0.1:6379');

export async function tenantGatewayMiddleware(
 req: Request,
 res: Response,
 next: NextFunction
): Promise<void> {
 const tenantId = req.header('X-Tenant-ID');
 const apiKey = req.header('X-Gateway-Key');
 const requestedModel = req.body?.model;

 // 1. Verplichte header-validatie
 if (!tenantId || !apiKey || typeof tenantId !== 'string') {
 res.status(401).json({
 error: {
 code: 'unauthorized',
 message: 'Geldige X-Tenant-ID en X-Gateway-Key headers zijn verplicht.'
 }
 });
 return;
 }

 try {
 // 2. Ophalen en valideren van tenant-profiel uit beveiligde cache
 const rawProfile = await redis.get(`tenant:profile:${tenantId}`);
 if (!rawProfile) {
 res.status(403).json({
 error: {
 code: 'tenant_forbidden',
 message: 'Tenant is niet geactiveerd op deze gateway.'
 }
 });
 return;
 }

 const tenant: TenantProfile = JSON.parse(rawProfile);

 // 3. Toegangscontrole op modelniveau
 if (!tenant.allowedModels.includes(requestedModel)) {
 res.status(403).json({
 error: {
 code: 'model_not_allowed',
 message: `Model '${requestedModel}' is niet vrijgegeven voor uw organisatie.`
 }
 });
 return;
 }

 // 4. Strikte lokale RPM rate-limiting via Redis sliding window
 const currentMinute = Math.floor(Date.now() / 60000);
 const rateLimitKey = `tenant:ratelimit:${tenantId}:${currentMinute}`;
 const requestCount = await redis.incr(rateLimitKey);
 
 if (requestCount === 1) {
 await redis.expire(rateLimitKey, 65); // Houd sleutel iets langer vast dan 1 minuut
 }

 if (requestCount > tenant.rpmLimit) {
 res.setHeader('Retry-After', '60');
 res.status(429).json({
 error: {
 code: 'rate_limit_exceeded',
 message: 'Verzoeklimiet per minuut voor deze tenant is overschreden.'
 }
 });
 return;
 }

 // 5. Harde budgetcontrole
 if (tenant.currentSpendEur >= tenant.monthlyBudgetEur) {
 res.status(402).json({
 error: {
 code: 'budget_depleted',
 message: 'Het maandelijkse AI-budget voor deze organisatie is bereikt.'
 }
 });
 return;
 }

 // 6. Request verrijken met tenant-specifieke runtime-context
 req.tenantContext = {
 tenantId: tenant.id,
 upstreamApiKey: tenant.customApiKey || process.env.CENTRAL_PROVIDER_KEY!,
 enforceZDR: tenant.enforceZDR,
 spendTrackingKey: `tenant:spend:${tenant.id}`
 };

 next();
 } catch (error) {
 // Interne gatewayfout veilig afhandelen zonder stacktrace te lekken
 res.status(500).json({
 error: {
 code: 'gateway_internal_error',
 message: 'Er is een interne fout opgetreden bij het verwerken van het tenant-beleid.'
 }
 });
 }
}

 
## Precise cost allocation and budgetary kill switches

 A crucial operational component of multi-tenancy is financial administration. Because language models bill based on token counts — with often substantial price differences between prompt tokens, generated output tokens, and cached contexts — a simple request count is not enough.

 The gateway must extract the actual usage statistics from the upstream response for every completed call. For streamed responses (Server-Sent Events), modern providers include the final usagestatistics in the last data chunk. The gateway intercepts this stream, calculates the exact cost using an internal price matrix, and asynchronously updates the tenant's cumulative usage in a distributed database.

 For an in-depth look at how these micro-transactions should be logged, calculated, and processed in SaaS subscription models, the article on [Allocating API costs per end user in a SaaS product](https://api.llmnet.nl/en/kosten-per-gebruiker-toerekenen) offers practical formulas and database patterns.

 Besides measurement, the gateway must also provide an automated kill switch. When a customer sets a monthly budget of, say, 500 euros, a runaway loop in an AI agent must not be allowed to consume thousands of euros in tokens within an hour. As soon as the threshold is reached, the gateway immediately blocks all new non-essential calls with an HTTP 402 (Payment Required) error.

 
## Compliance, routing, and zero data retention

 Within a business customer base, there are often large differences in compliance requirements. A marketing agency might accept that data flows through regular cloud endpoints, while a law firm or financial institution demands strict contractual guarantees. The most common requirement here is Zero Data Retention (ZDR), where the LLM provider guarantees that prompts and responses are never stored to disk in any way, are not logged for debugging, and are not reused for model training.

 The central gateway acts as the policy enforcer here. When a tenant's profile has the enforceZDR property active, the gateway will route requests exclusively to approved endpoints that fall under these specific agreements. To see which provider headers, encryption requirements, and contractual checkboxes are needed for this, read the technical documentation on [configuring zero data retention with LLM APIs](https://api.llmnet.nl/en/zero-data-retention-api-configuraties).

 
## Architecture choice: Build your own gateway versus an aggregator

 When an organization starts centralizing AI traffic across multiple customers, the question quickly arises whether to build a proprietary gateway or use a commercial API aggregator. Both approaches have clear pros and cons that must be weighed against available engineering capacity and organizational complexity.

 
 
 
 
 Criterion | 
 Custom-built Gateway (Custom Proxy) | 
 Managed Aggregator Service | 
 

 
 
 
 Data sovereignty | 
 Full control: runs within your own VPC / cloud environment | 
 Third party may process metadata and logs | 
 

 
 Fine-grained tenant isolation | 
 Fully programmable down to the custom-header level | 
 Depends on the provider's features and RBAC | 
 

 
 Maintenance burden | 
 High: team must manage rate limiting, retries, and updates itself | 
 Low: ready-made dashboard, SLA, and integrations | 
 

 
 BYOK support | 
 Flexible integration with your own HSM / Key Vault | 
 Keys must be entrusted to the aggregator platform | 
 

 
 
 

 For an extensive analysis of the market, protocols, and features of ready-made routing services, we refer to the background article on [the power of an LLM API aggregator](https://api.llmnet.nl/en/aggregator-uitleg) to make the right build-versus-buy choice.

 
## Operational audit and separation checklist

 Securing a multi-tenant gateway is not a one-time configuration step, but a continuous operational process. Before a gateway architecture goes into production, the following checks should be structurally performed and automated in CI/CD pipelines:

 
 
- Memory lifecycle: Make sure buffers, request strings, and intermediate JSON objects are released by the runtime garbage collector immediately after the HTTP request completes, to prevent cross-request context contamination in application servers.
 
- Pseudonymization in central logs: Never store unencoded prompts or responses in application-wide logs (such as Datadog, Grafana, or CloudWatch). Use only hashed tenant IDs and anonymous correlation IDs.
 
- Automated isolation tests: Run continuous regression tests where two virtual tenants simultaneously send similar prompts, and verify that no data leaks into semantic caches or downstream error reports.
 
- Per-tenant circuit breakers: Configure error detection so that when the upstream provider fails on prompts from a specific tenant (for example, due to exceeding context length or content policy filters), the circuit breaker pauses only that specific tenant, not the entire gateway.
 

 By combining data isolation, dynamic rate limiting, cost monitoring, and strict authentication into one central gateway layer, a robust and scalable foundation emerges. This allows hundreds of customers to be served safely and predictably without sacrificing reliability, privacy, or financial control.