Distributing API keys under high concurrency

By Ivo Donker — compiled with AI support (Claude & Gemini) · Last updated: 6 August 2026

When an application that uses Large Language Models (LLMs) grows in the number of concurrent users or tasks, the infrastructure quickly runs into the limits set by API providers. These limits are typically expressed in Requests Per Minute (RPM), Tokens Per Minute (TPM), or the number of simultaneously open connections (concurrency). In many enterprise environments, an attempt is made to increase capacity by deploying multiple API keys within a central gateway.

Distributing network requests across multiple API keys under a high degree of concurrency, however, brings specific technical challenges. Unlike traditional HTTP web servers, where requests often have a predictable duration and processing load, the load of LLM calls varies enormously. A request can consist of a short ten-token classification or a complex summary of a document with thousands of tokens in the prompt and output. This article covers the architecture patterns, selection algorithms, error handling, and prerequisites for effectively distributing API keys under high concurrent load.

The problem of saturation at the key level

API providers tie their throughput capacity and rate limits to specific identification methods, such as an API key, a project ID, or an organization account. When an application routes all incoming requests through a single API key, that specific key quickly becomes a bottleneck. This often happens long before the provider's underlying infrastructural capacity itself has been reached.

With a sudden spike in concurrent requests, the key's assigned limit is exhausted within seconds. The API provider then responds with an HTTP 429 status code (Too Many Requests). If the infrastructure then immediately retries all failed requests through that same key, a so-called retry storm arises. As a result, the key remains blocked for the entire recovery window of the provider. Distributing the load across multiple keys prevents a single key from blocking the entire throughput of the gateway.

Why simple rotation (round-robin) falls short

The most obvious way to distribute requests across a pool of API keys is a static round-robin selection. Here, each incoming request is assigned the next key from the list in sequence. In an environment with high concurrency and variable LLM workload, however, this pattern proves insufficiently effective.

The cause of this inadequacy lies in the asymmetric nature of LLM requests:

• Variable processing time: A simple request might take 200 milliseconds, while a generated response with a long context occupies a connection for 15 seconds.
• Skewed token consumption: Ten consecutive requests distributed via round-robin across ten keys can randomly consume 50 tokens on one key and 80,000 tokens on another key.
• Invisibility of saturation: Round-robin doesn't take into account the current status or remaining capacity of a key. A key that just received a 429 error simply keeps getting assigned requests in the next cycle.

When a round-robin algorithm assigns heavy requests to a key that is already close to its TPM limit, this inevitably leads to errors, while other keys in the pool remain unused. To achieve an even load, a dynamic algorithm is necessary that takes into account the state of requests and the provider's response headers.

Advanced distribution strategies and their complexity

To guarantee stable throughput under high concurrency, more complex selection rules are applied in production environments. These methods require the gateway to actively track the status of each key.

Selecting based on fewest in-flight calls (least in-flight)

In the least in-flightpattern, the gateway keeps a counter for each API key of the number of requests currently actively being handled. When a new request comes in, the gateway selects the key with the lowest counter value at that specific moment. Once the request has finished (or failed), the counter of the relevant key is decreased again.

This pattern automatically responds to the latency of requests. If a specific API key is used for a slow, long generation, the counter for that key stays higher. Subsequent requests are automatically redirected to keys that process fast responses and therefore become available more quickly. This reduces the buildup of concurrent queues per key.

Selecting based on dynamic capacity (header-aware)

An even more precise method ties the selection process to the token bucket algorithm in the LLM gateway. Here, the gateway estimates or calculates the remaining capacity of each key based on historical data and the provider's headers. The gateway always chooses the key that at that specific moment has the highest margin to the set RPM and TPM limits.

The complexity of this approach lies in maintaining the memory state. Where a round-robin counter suffices with a simple integer in memory, a dynamic capacity distributor requires a data structure in which wait times, active connections, and token quotas per key are tracked synchronously.

Response headers as the source of truth

Many large LLM providers include specific response headers with every HTTP response that reflect the current status of the rate limits. Examples of standard headers include:

• x-ratelimit-remaining-requests: The number of remaining allowed requests in the current time window.
• x-ratelimit-remaining-tokens: The number of remaining allowed tokens in the current time window.
• x-ratelimit-reset-requests: The time (in seconds or milliseconds) until the request quota resets.
• x-ratelimit-reset-tokens: The time until the token quota resets.
• retry-after: The mandatory wait time if a 429 error has occurred.

It's a crucial design rule to use these response headers as the absolute source of truth for the distribution algorithm, rather than relying solely on the application's own counts. Local counts diverge in practice due to various factors: the exact way the provider counts tokens (for example different tokenizers), network timeouts, or requests executed by other systems using the same key.

Upon receiving each HTTP response, the gateway reads the headers and updates the status of the key used in the central memory. If the header x-ratelimit-remaining-tokens drops below a critical threshold, the gateway can temporarily exclude that key from selection until the reset time has passed, even before a 429 error actually occurs.

Handling limit errors and quarantine

Despite accurate distribution, unexpected spikes can still trigger HTTP 429 errors. How a gateway responds to such an error determines the stability of the entire chain.

When a key returns a limit error, the gateway should apply the quarantine pattern (or circuit breaker):

1. Change status: The key in question is given the status Cooling Down or In Quarantaine.
2. Determine wait time: The duration of the quarantine is preferably read from the retry-after header. If this is missing, a calculated declining wait time via an exponential backoff scheme applies. See the guidelines on retries and backoff strategies.
3. Retry request: The failed request is immediately re-offered by the gateway to a *different* key that has the `Active` status and sufficient capacity.
4. Recovery: Only once the timer has expired does the key return to the active pool, possibly first in a `Half-Open` status in which only a single test request is allowed.

Distinguishing between temporary and permanent errors

Not every error code requires the same response. The distribution mechanism must make a strict distinction between temporary limit errors and permanent key errors. Miscategorizing errors can lead to unnecessarily disabling healthy keys or endlessly retrying an invalid key.

An HTTP 429 error indicates that the time window's limit has been reached; this is a **temporary error**. The key will be usable again within seconds or minutes.

Error codes such as HTTP 401 (Unauthorized) or HTTP 403 (Forbidden) point to problems with the key itself: the key has been revoked, has expired, or no longer has permissions for the requested model. A notification of the type insufficient_quota (the organization's prepaid balance or credit has run out) also means the key will no longer function without human intervention.

When a **permanent error** occurs, the following protocol applies:

• The key is **immediately and permanently** removed from the active rotation pool.
• The status of the key is set in the administration to Revoked or Disabled.
• An automated alert is sent directly to the administrator team.
• Incoming requests are redirected to the remaining valid keys without interruption.

Distributed architecture: shared state across multiple gateway instances

In modern cloud infrastructure, an API gateway rarely runs on a single server. To provide scalability and high availability, multiple instances of the gateway are run alongside each other behind a network load balancer.

This introduces the problem of **split-brain status**. If each gateway instance tracks the counter for API keys locally in its own memory, they have no visibility into the requests being processed by other instances. Five gateway instances that each think a key still has 20% of its capacity left will together comfortably exceed the provider's limit.

To prevent this, the state of the key pool must be shared in a central, fast data store such as Redis or KeyDB. Important elements in a distributed architecture are:

• Atomic operations: Incrementing the counter for active requests and checking the thresholds must be performed atomically (for example using Redis Lua scripts or MULTI/EXEC transactions) to prevent race conditions between gateway nodes.
• Central quarantine status: If instance A receives an HTTP 429 on key X, instance A must set a quarantine flag in the shared memory. Instances B, C, and D read this flag and immediately stop sending traffic to key X.
• Lightweight synchronization: To keep network latency to the data store layer low, gateway instances store response headers locally for fast read operations, but validate critical limit breaches via the shared state.

Legal frameworks and providers' terms of use

When implementing key distribution, it's essential to maintain transparency with regard to the terms of service (ToS) of the relevant API provider.

There is a significant legal and operational difference between two situations:

1. Legitimate use of multiple keys: An organization creates multiple API keys within *one and the same* verified business organization account. This is facilitated by providers to separate different projects, teams, or internal microservices, and to manage aggregated quota within the organization.
2. Circumventing limits via multiple accounts (Sybil behavior): A developer creates dozens of individual free or trial accounts with a provider with the goal of circumventing the platform's throughput restrictions without paying for higher tiers.

Circumventing limits by creating fake accounts is an explicit violation of the terms of virtually all AI providers. When a provider detects this pattern (for example based on IP addresses, payment details, or request patterns), all related accounts and API keys are immediately blocked.

For organizations that need strict throughput guarantees, the designated path is entering into enterprise agreements with reserved capacity. See the legal and operational aspects of this on the page about AI contracts and SLA agreements for more information on securing processing capacity.

Security risks and key management in key pools

Managing a pool with multiple API keys increases the application's security risk. Where the leaking of a single key is already damaging, storing and rotating ten or twenty keys significantly increases the possible attack surface (blast radius).

Important security measures when deploying key pools are:

• No hardcoded keys: API keys must never be stored in the source code or environment variables of the gateway application itself.
• Central secrets manager: Use a specialized vault (such as HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault) to load the pool of keys dynamically when the gateway instance starts.
• Principle of least privilege: Ensure the keys in the pool only have access to the specific models and endpoints used by the application.

For a detailed treatment of securing, storing, and periodically rotating these credentials, see the article on managing API keys securely.

Observability: validating the distribution strategy

To verify whether a configured distribution strategy functions correctly under high concurrency, a well set-up observability stack is necessary. Without concrete measurements it's impossible to determine whether the load balancing is running efficiently or whether certain keys are becoming overloaded unnoticed.

The key metrics that must be monitored in a dashboard (such as Prometheus/Grafana) are:

• Request distribution per key: A bar chart showing what percentage of incoming calls is routed to each individual API key. In a healthy situation with identical key quotas, this distribution should be proportional.
• Error rate per key (HTTP 429 vs 5xx): The number of error messages expressed as a percentage of total traffic per key. A rise on one specific key points to an incorrectly estimated threshold.
• Wait time and queue length: The time an incoming request has to wait in the gateway before a suitable key with available capacity is assigned. A rise in this wait time is the first signal of overall saturation of the key pool.
• Quarantine frequency: How often a key switches to the status Cooling Down and how long it stays in quarantine on average.

By continuously analyzing this data and comparing it with the agreements in the SLA and uptime specifications of LLM providers, the infrastructure team can adjust the size dimension of the key pool in time to match the actual usage pattern.

Further reading

• Managing API keys securely
• Model routing
• Token bucket algorithm in an LLM gateway
• Retries and backoff strategies
• SLA and uptime of LLM providers
• AI contracts and SLA agreements