Robust LLM API Integrations: Retries, Timeouts, and Exponential Backoff without Double Costs
Integrating Large Language Models (LLMs) via an API brings specific technical challenges. Unlike traditional REST APIs, where a response is often generated within a few milliseconds, an LLM request can take tens of seconds. This long execution time, combined with the high server load on the provider's side, makes LLM APIs inherently more sensitive to network fluctuations, timeouts, and rate limits.
As a developer, you want a seamless experience for your users. This means that when an API request fails, you don't immediately show an error message, but automatically retry the request. However, implementing a "retry mechanism" with LLMs introduces a significant risk: accidentally paying twice for the same request. In this article, we discuss the architecture of safe retries, correctly configuring timeouts, the theory behind exponential backoff, and how to prevent financial leakage.
The Anatomy of a Failed LLM Request
To determine an effective retry strategy, we must first understand why a request to a model like GPT-4 or Claude fails. We can roughly divide these errors into three categories. For a complete overview of what each HTTP status code means within our architecture, you can visit our internal page on common API error codes.
- Client-side or Network Errors: The connection drops before the request reaches the server, or the connection is broken while the client is waiting for a response. This often results in a timeout exception in your code, without an HTTP status code.
- Rate Limits (HTTP 429): You are sending too many requests per minute (RPM) or too many tokens per minute (TPM). The server rejects the request immediately to prevent overload.
- Server Errors (HTTP 500, 502, 503, 504): The API provider is experiencing internal issues, insufficient GPU capacity, or network problems in their own infrastructure.
The Danger of "Ghost Requests" and Double Costs
The greatest financial risk when building applications on top of commercial LLM APIs is the "ghost request". This occurs when a request reaches the provider's server, the provider starts generating tokens (which costs computing power and therefore money), but the connection between the client and the server drops before the response is delivered.
From your application's perspective, the request failed (due to a timeout or a 502 Bad Gateway). A naive retry mechanism will immediately send this request again. The provider then processes the new request. The result? You pay for the tokens of the first, aborted request as well as for the tokens of the successful retry, while the user only sees a single response.
Strategies to Prevent Double Costs
Completely eliminating ghost requests is complex because you depend on the architecture of the API provider. However, there are effective patterns to minimize the financial impact.
1. Use Idempotency Keys (If Supported)
Idempotence means that performing an action multiple times yields the same result (and in this case: the same costs) as performing it once. Some modern APIs support sending an Idempotency-Key in the HTTP headers. If you retry a request after a network interruption and use the same key, the server knows this is a retry. If the original request has already been processed, the server returns the saved result instead of having the model recalculate. Always check the current documentation of your specific LLM provider to see if they actively honor this header for their inference endpoints.
2. Lower the Read Timeout and Use Streaming
A standard HTTP client often waits passively until the full response is received. With long LLM responses, this increases the chance of a connection timeout. By switching to streaming mode (such as Server-Sent Events), you receive the response token by token. As soon as the first tokens arrive, you know the request is being processed successfully. If the stream breaks halfway through, you still pay for the generated tokens, but at least you know exactly at what point it went wrong. You can find more information about implementing this on our streaming endpoints page.
Configuring Timeouts Correctly
In the network layer, you must make a strict distinction between two types of timeouts. An incorrect configuration here is the main cause of unnecessary retries.
- Connection Timeout: The time your application takes to establish a TCP connection with the API server. Since LLM APIs are often behind fast load balancers, this should be very short. A safe but generous setting is 3 to 5 seconds.
- Read Timeout: The time your application waits for the (first) data packet from the server after the connection is successfully established. Because the model must first process your prompt (Time To First Token), this timeout must be set generously.
Exponential Backoff and Jitter
When you receive a 429 (Rate Limit) or a 503 (Service Unavailable), retrying immediately (a so-called tight loop) is the worst possible response. It worsens the overload on the server side and leads to you being blocked even longer.
The industry standard for handling these temporary errors is "Exponential Backoff with Jitter". The basic principle of exponential backoff is that the wait time between consecutive attempts increases exponentially.
As a rule of thumb, not a strict measurement, you can assume an initial wait time of about 1 to 2 seconds. With each subsequent failed attempt, you double this wait time. This gives the network or the API a chance to recover.
The Thundering Herd Problem and Jitter
If your application experiences a spike in traffic and the API returns a 429 Rate Limit to a hundred concurrent threads, all these threads without Jitter will fire their retry at the exact same time (for example, after exactly 2.0 seconds). This creates a new spike, or a "thundering herd", which immediately results in new 429 errors.
Jitter is adding randomness to the calculated backoff time. Instead of waiting exactly 4 seconds, you have the application choose a random wait time between, for example, 2 and 4 seconds. This spreads the retries evenly across the timeline.
A Calculation Example
| Attempt | Base backoff (2^n) | With Jitter (random addition) | Cumulative wait time (approximate) |
|---|---|---|---|
| 1st retry | 2 seconds | 1.5 to 2.5 seconds | 2 seconds |
| 2nd retry | 4 seconds | 3.0 to 5.0 seconds | 6 seconds |
| 3rd retry | 8 seconds | 6.0 to 10.0 seconds | 14 seconds |
| 4th retry | 16 seconds | 12.0 to 20.0 seconds | 30 seconds |
It is crucial to set an absolute maximum for both the number of attempts (for example, a maximum of 4) and the maximum total wait time. If an LLM API still returns a 500 error after 30 seconds and multiple retries, it is better to fail gracefully and inform the user via the interface that the AI service is temporarily unavailable.
Broader Perspective
Implementing robust backoff strategies is just one aspect of building scalable AI applications. To further explore the fundamentals of prompt engineering, model selection, and system architecture, we recommend consulting our comprehensive main guide at https://gids.llmnet.nl/. Here we cover the broader context of AI integrations in production environments.
Conclusion
Building a reliable connection to an LLM API requires more than simply instantiating an HTTP client. Because LLMs are computationally heavy, the risk of timeouts and rate limits is always present. By making a sharp distinction between connection and read timeouts, you prevent your application from becoming impatient and aborting requests that are still generating costs in the background.
When errors inevitably do occur, a thoughtful implementation of exponential backoff with jitter ensures that your system behaves gracefully. It prevents you from bombarding the API provider with repeated requests, maximizes the chance of a successful next attempt, and keeps the operational costs of unintended ghost requests under control.