Rate limit simulator for LLM APIs

By Ivo Donker - 3 August 2026

When building applications on top of Large Language Models (LLMs), rate limits from API providers are one of the most significant infrastructure hurdles. This simulator helps you calculate whether your expected traffic pattern fits within the configured limits, which limit hits first, and what the expected percentage of HTTP 429 errors (Too Many Requests) will be.

How LLM provider rate limits work

Rate limiting is a fundamental mechanism API providers use to protect the stability and availability of their infrastructure. With traditional REST APIs, the focus is almost exclusively on the number of network requests per time unit. With generative AI and Large Language Models this is more complex, because the compute power and memory usage (VRAM) per request vary enormously based on the amount of text processed and generated. If you want to learn more about how input is converted into these compute units, read the guide on tokenization explained.

Providers typically use three to four different dimensions for rate limits:

• RPM (Requests Per Minute): The maximum number of individual HTTP requests you may send within a rolling 60-second window.
• TPM (Tokens Per Minute): The total number of tokens (both input and output tokens) the model may process within 60 seconds.
• TPD (Tokens Per Day): A daily cap that often applies to free tiers or specific trial accounts, to prevent abuse over longer periods.
• Concurrency (Concurrency Limit): The maximum number of requests that may be in flight at exactly the same time on the provider's infrastructure.

When an application exceeds one of these limits, the provider's API responds with an HTTP status code 429 Too Many Requests. Understanding the exact relationship between these limits is essential for a stable architecture, as also described on the page about rate limits and costs.

Why TPM usually becomes the bottleneck before RPM

A common pitfall for software architects is assuming that the RPM limit is the main bottleneck. In practice, however, the TPM limit is almost always reached first. This is due to the relatively high number of tokens per request in modern LLM applications.

Take, for example, a scenario where a provider sets a limit of 500 RPM and 90.000 TPM. If your application sends an average of 800 input tokens and receives 400 output tokens back, each request consumes exactly 1.200 tokens. To reach the TPM limit of 90.000, only 75 requests per minute are needed ($90.000 / 1.200 = 75$). At that point you're only using 15% of your allowed RPM ($75 / 500$). In this case, the TPM limit therefore becomes a bottleneck at just a fraction of the maximum request frequency.

Moreover, output tokens are much more expensive to generate server-side than input tokens (due to the auto-regressive nature of LLMs), although API providers count the sum of both toward the TPM limit. If a prompt grows to 4.000 tokens per request due to a large context or extensive RAG documents, a single parallel batch of requests can exhaust the TPM limit within seconds.

Correctly handling HTTP 429 errors

When a 429 error occurs, it's crucial that the client application handles it in a controlled way. Immediately and endlessly repeating the request (brute-force retries) leads to a so-called retry storm, where the network becomes completely clogged and the API provider may start rejecting requests at the IP level.

The industry standard for handling rate limits includes the following elements:

• Exponential Backoff: Increase the wait time between successive attempts exponentially (for example 1s, 2s, 4s, 8s).
• Jitter (noise): Add a random variation to the wait time to prevent multiple concurrent requests from being retried at exactly the same moment.
• Respecting response headers: Many providers include useful headers such as x-ratelimit-reset-requests or retry-after. The client should read these values and wait exactly that long before retrying. To see how to build this in practice, you can consult the guide on retries and backoff is worth consulting.

Strategies: batching, spreading, and routing

If the simulator shows that your expected peak load exceeds the available limits, several architectural measures can help prevent outages:

1. Batch processing for asynchronous tasks

Tasks that don't need to run in real time (such as processing documents, newsletters, or data analysis) are better off being batched together. Many providers offer dedicated batch processing endpoints at a lower rate and with a separate, higher rate limit. You can find more details on the page about batch processing for LLM APIs.

2. Dynamic rate limiting at the client level

By incorporating a local token-bucket or leaky-bucket algorithm into your own backend, you can smooth out outgoing traffic (traffic shaping). Instead of firing requests directly at the external API, they are captured in an internal queue that releases requests at a controlled pace based on measured token usage.

3. Multi-provider and model routing

When a single provider key doesn't offer enough capacity, a routing layer can help. This dynamically distributes requests across multiple API keys, multiple accounts, or even different LLM providers. See the article on model routing for strategies on smartly distributing requests when limits are looming.

Measuring actual limits instead of estimating them

While a simulator like this provides excellent theoretical insight, practice sometimes differs from the static calculations. Two important factors cause these deviations:

1. Variable response lengths: LLMs don't generate a fixed number of tokens. A question might be answered in 50 tokens one time and 800 tokens the next.
2. Sliding windows: Providers rarely reset the TPM limit at a fixed minute boundary. Instead, they often use a rolling window of, say, 60 seconds, or a leaky-bucket algorithm at the millisecond level.

To map out exact limits and actual usage, it's necessary to set up telemetry in your software pipeline. Both the number of tokens used per request and the latency and any 429 errors need to be monitored. For a deeper dive into setting up active monitoring, we refer you to the guide on observability and logging.

Assumptions and limitations

This simulator uses a simplified mathematical model to visualize traffic flows and rate limits. When interpreting the results, keep the following assumptions in mind:

• Static token lengths: The calculation assumes a fixed average for input and output tokens per request. In reality, token length varies per request.
• Even distribution: Requests within a minute are assumed to be evenly distributed over time. In practice, peak loads often occur in very short bursts (for example 20 requests in 1 second), which triggers a 429 error sooner than the model predicts.
• Simple queuing model: The queue calculation assumes a FIFO principle (First In, First Out) without priority rules or network overhead latency.
• No provider-specific exceptions: Some providers charge a minimum token count per request or apply separate limits based on the model chosen within the same account. Make sure to factor these specific rules into your own parameters.

Further reading

• Rate limits and costs in practice
• Handling retries and exponential backoff
• Optimization through batch processing for LLM APIs
• Strategies for model routing and fallback
• Measuring API speed and latency
• How exactly does tokenization work?