Distributed rate limiting across multiple API instances

When an LLM infrastructure scales from a single server to a distributed cluster with dozens of API instances, a local rate-limiting mechanism loses its effectiveness. If individual workers maintain their own counters in local memory, a burst of incoming requests can still breach upstream rate limits or lead to unfair allocation among tenants. Within architecture pillar A2, central coordination serves as the foundation; read the guide on self-hosting an LLM gateway to see how proxy layers and failover clusters are fundamentally configured.

In this article, we analyze how distributed rate limiting operates within a multi-instance gateway. We cover the specific dimensions of LLM limits (requests per minute, tokens per minute, and concurrency), the underlying data structures and Redis Lua scripts, local batch reservation techniques to minimize round trips, and the trade-offs between absolute synchronization and throughput.

The unique challenge of rate limiting in LLM gateways

Traditional HTTP API rate limiters primarily measure Requests Per Minute (RPM). With language models, this single dimension is insufficient. Upstream model providers (such as Anthropic, OpenAI, or self-hosted vLLM clusters) enforce three distinct, simultaneous constraints that can saturate independently:

• Requests Per Minute (RPM): The raw number of HTTP requests within a time window.
• Tokens Per Minute (TPM): The sum of prompt tokens and generated completion tokens across a sliding 60-second window.
• Concurrency (inflight requests): The number of parallel model calls actively being processed at any exact moment.

The asymmetry of token consumption makes distributed synchronization complex. At the start of an API call, the input size is known, but the output length remains uncertain until the response is fully generated or terminated. To manage the risk of overruns, we must tie this mechanism to the core principles of throughput management; consult the analysis on the token bucket algorithm in an LLM gateway for the mathematical properties of continuous replenishment and burst tolerance.

Architectural Patterns for Distributed Coordination

There are three main patterns for keeping usage state synchronized across independent API instances. Each pattern involves its own trade-offs regarding network latency, consistency, and complexity.

1. Central State Store (Redis / Dragonfly)

Every incoming request triggers a fast remote operation to a central in-memory data store. Because multiple instances attempt to update the same counter concurrently, updates are wrapped in atomic Lua scripts. This prevents race conditions (such as check-then-act bugs), but introduces an additional network round-trip (typically 0.5 to 2.0 milliseconds within the same datacenter) before the gateway forwards the request to the model provider.

2. Batch Allocation / Token Leasing

Instead of querying a central store for every individual request, each API instance periodically reserves a "lease" or "token block" from the central pool (for example, 50 requests and 50,000 tokens for 5 seconds). The instance handles incoming traffic locally without network latency as long as the lease budget has not been exhausted. Unused tokens are returned after the lease period expires.

3. Consistent Hashing on the Load Balancer

The load balancer routes incoming traffic to the same gateway instance based on the hash of the tenant ID or API key. This allows rate limiting to take place entirely locally in the instance's RAM. The downside is that adding or dropping instances redistributes the hashes, and "hot tenants" can disproportionately overload a single instance.

Implementation: Atomic Sliding Window in Redis via Lua

A sliding window (sliding window log or sliding window counter) avoids the well-known boundary violations of fixed windows, where a double burst can occur around the minute boundary. In Redis, we implement this using a sorted set (Sorted Set, ZSET), where both the score and member consist of the timestamp in milliseconds.

-- Lua script voor gedistribueerde sliding window rate limiting
-- KEYS[1]: Redis key (bijv. "ratelimit:tpm:tenant_123")
-- ARGV[1]: Huidige timestamp in milliseconden
-- ARGV[2]: Venstergrootte in milliseconden (bijv. 60000)
-- ARGV[3]: Maximale capaciteit (bijv. 100000 tokens)
-- ARGV[4]: Gevraagde eenheden voor dit verzoek (bijv. 1200 tokens)
-- ARGV[5]: Unieke request identifier

local key = KEYS[1]
local now = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local max_limit = tonumber(ARGV[3])
local cost = tonumber(ARGV[4])
local req_id = ARGV[5]

local clear_before = now - window

-- 1. Verwijder records die buiten het glijdende venster vallen
redis.call('ZREMRANGEBYSCORE', key, '-inf', clear_before)

-- 2. Bereken de huidige som van het verbruik
local entries = redis.call('ZRANGE', key, 0, -1, 'WITHSCORES')
local current_usage = 0
for i = 1, #entries, 2 do
  -- Het element bevat "cost:req_id"
  local raw_val = entries[i]
  local sep = string.find(raw_val, ":")
  if sep then
    local item_cost = tonumber(string.sub(raw_val, 1, sep - 1))
    current_usage = current_usage + (item_cost or 0)
  end
end

-- 3. Valideer of er voldoende ruimte is
if current_usage + cost <= max_limit then
  -- Voeg het huidige verbruik toe aan de ZSET
  local member = tostring(cost) .. ":" .. req_id
  redis.call('ZADD', key, now, member)
  -- Stel TTL in op iets meer dan het venster voor automatische opruiming
  redis.call('PEXPIRE', key, window + 1000)
  return {1, max_limit - (current_usage + cost), 0} -- Toegestaan
else
  -- Bereken wachttijd tot er voldoende capaciteit vrijkomt
  local oldest_entries = redis.call('ZRANGE', key, 0, 1, 'WITHSCORES')
  local retry_after = 0
  if #oldest_entries >= 2 then
    retry_after = math.max(0, (tonumber(oldest_entries[2]) + window) - now)
  end
  return {0, max_limit - current_usage, retry_after} -- Geweigerd
end

This script performs the cleanup, calculation, and allocation in a single atomic block on the Redis server. Because Redis operates in a single-threaded manner with respect to command execution, two API instances at the exact same millisecond interval can never observe or exceed an inconsistent state.

Two-Phase Token Reservation with Unknown Completion Length

When forwarding a prompt to an LLM, we know how many tokens the prompt contains ($T_{in}$), but not how many tokens the completion will generate ($T_{out}$). If measured only after the fact, a concurrent batch of long completions can breach the upstream TPM limit. If one conservatively registers the maximum model context (`max_tokens`) upfront, legitimate requests are incorrectly rejected (false 429 errors).

The solution to this is a two-phase token reservation protocol (Two-Phase Token Reservation):

Phase 1: Upfront reservation
The API instance calculates $T_{in}$ via a local tokenizer and makes an estimate of $T_{est} = T_{in} + \min(\text{max\_tokens}, \text{historisch\_gemiddelde} \times 1.5)$. The instance reserves $T_{est}$ tokens in the central Redis limiter via the atomic script.

Phase 2: Post-settlement and reconciliation
After the response is completed via Server-Sent Events (SSE) or standard JSON, the instance reads the actual reported token count ($T_{actueel} = T_{in} + T_{out}$) from the provider's payload. The script performs a reconciliation:

• If $T_{actueel} < T_{est}$: The difference ($T_{est} - T_{actueel}$) is immediately credited back to the sliding window counter as a negative entry or credit.
• If $T_{actueel} > T_{est}$: The excess usage is debited immediately, potentially briefly throttling subsequent calls.

For insight into how upstream quotas and financial limits correlate with these technical buffers, see the article on rate limits, tokens, and costs to prevent overages across the organization.

Concurrency Limiting with Distributed Semaphores

In addition to time-based limits (RPM/TPM), concurrency limiting is crucial to prevent streaming connections from consuming all available sockets or vLLM batching slots. A distributed semaphore via Redis monitors the number of concurrently active connections across all instances.

import time
import uuid
import redis

class DistributedConcurrencyLimiter:
    def __init__(self, redis_client: redis.Redis, resource_key: str, max_concurrent: int, timeout_sec: int = 120):
        self.r = redis_client
        self.key = f"semaphore:{resource_key}"
        self.max_concurrent = max_concurrent
        self.timeout_sec = timeout_sec

    def acquire(self) -> str | None:
        """Probeert een slot te reserveren. Retourneert slot_id bij succes, anders None."""
        slot_id = str(uuid.uuid4())
        now = time.time()
        
        # Ruim verouderde slots op (bijvoorbeeld door gecrashte instances)
        pipe = self.r.pipeline()
        pipe.zremrangebyscore(self.key, '-inf', now - self.timeout_sec)
        pipe.zcard(self.key)
        _, current_count = pipe.execute()

        if current_count < self.max_concurrent:
            # Voeg slot toe met huidige timestamp als score
            added = self.r.zadd(self.key, {slot_id: now}, nx=True)
            if added:
                return slot_id
        return None

    def release(self, slot_id: str) -> None:
        """Geeft het gereserveerde slot direct vrij."""
        self.r.zrem(self.key, slot_id)

A hard lesson learned from production environments is the risk of "ghost slots": when an API instance crashes or encounters a network disruption while a streaming response is in progress, the slot must not remain occupied indefinitely. Purging entries older than timeout_sec on every `acquire` operation prevents a dead connection from permanently degrading total cluster concurrency.

Pre-fetching and Local Token Buckets at High Throughput

At thousands of requests per second, a centralized store creates a latency bottleneck. If each gateway node handles 100 requests per second, ten nodes collectively generate 1,000 Redis calls per second per tenant. To mitigate this, we apply a hybrid strategy: Local Leased Token Buckets.

Here, an API instance periodically requests a block of tokens from the central Redis instance. Once the instance acquires a local budget—for instance, 20,000 tokens—it validates and processes incoming calls entirely in local memory (latency < 0.05ms). Only when the local buffer falls below 20% capacity does the background thread dispatch an asynchronous renewal request to the central Redis server.

Queueing and Backpressure on Depletion

When a rate limit is reached, immediately returning an HTTP 429 status code is often undesirable for the end user or the calling pipeline. A robust architecture combines rate limiting with a distributed queue. Instead of dropping requests, they are parked in a priority buffer until capacity becomes available again.

To prevent interactive user interactions from being blocked by heavy background tasks, we split the queue based on urgency; read the documentation on priority queues for LLM tasks for implementation patterns regarding latency budgets and starvation prevention.

For a broader overview of open-source and enterprise tooling capable of providing this logic modularly, the guide to cost monitoring and token management for LLM applications offers comparative insights into turnkey gateway solutions versus custom builds.

Fault Tolerance and Fail-Open vs. Fail-Closed Strategies

What happens when the central Redis cluster becomes unreachable or experiences a network partition? In the gateway architecture, an explicit choice must be made between two lines of behavior:

In practice, Degraded Local Fallback proves to be the most stable choice for production environments: when the central store fails to respond within a configured timeout of 15 milliseconds, the instance temporarily falls back to a strict local in-memory counter until the central connection is restored.

Summary and Operating Rules

Distributed rate limiting across multiple API instances requires a careful balance between accuracy, latency, and fault tolerance. To keep the system stable in production, we adhere to the following rules of thumb:

• Manage RPM, TPM, and Concurrency as distinct dimensions in the central state store.
• Use a sliding window with atomically executed Lua scripts to eliminate threshold spikes at window boundaries.
• Implement a two-phase reservation protocol for tokens to bridge the discrepancy between estimated and actual completed token usage.
• Always add an automatic TTL/timeout to concurrency semaphores to neutralize deadlocks caused by crashed workers.
• Opt for Degraded Local Fallback during outages in the central coordination layer to prevent total downtime without overloading upstream limits.