Retries, timeouts, and fallbacks: building robust LLM integrations

Large Language Models are powerful, but the APIs that expose them are subject to network latency, rate limits, and unexpected downtime. Especially when building complex RAG pipelines, a single failing API call can break the entire chain. To build production-ready, robust applications, your architecture must anticipate failure.

Why LLM APIs fail

There are several reasons why a request to a model provider might fail:

• Rate limits (429): You are sending too many requests or tokens per minute.
• Timeouts (504): The model takes too long to process the prompt (especially with large context windows).
• Server errors (500/503): The provider is experiencing capacity issues.

Setting timeouts

By default, HTTP clients often wait indefinitely. For LLM interactions, this is disastrous because threads or serverless functions will block. Always implement a hard timeout. For generation tasks, 30 to 60 seconds is often sufficient; for vector embeddings, 5 seconds at most.

Retries and Exponential Backoff (with Jitter)

If a request fails due to a transient error (such as a 429 or 503), you want to retry it. However, retrying immediately increases the load on the server. Exponential backoff ensures that the wait time between attempts increases (e.g., 1s, 2s, 4s, 8s). Add jitter (randomness) to this to prevent the thundering herd problem.

Fallbacks: From primary models to efficient alternatives

If, after multiple retries, your primary API (for example, a heavy Claude Pro instance) remains unreachable, your system must switch seamlessly. You can set up a fallback to a faster, more efficient model, such as DeepSeek via OpenClaw, or even a locally hosted model. This guarantees uptime, albeit sometimes with slightly lower reasoning quality, which is acceptable as an emergency solution for most web applications.

Putting it all together (Pseudocode)

Below you can see how these concepts come together in a robust wrapper function for LLM calls:

function callLLMWithResilience(prompt, idempotencyKey) {
const maxRetries = 3;
const baseDelayMs = 1000;
const primaryModel = "claude-3-opus";
const fallbackModel = "deepseek-chat";

for (int attempt = 0; attempt <= maxRetries; attempt++) {
try {
// Set hard timeout per call
const response = api.complete({
model: primaryModel,
prompt: prompt,
timeout: 30000,
headers: { "Idempotency-Key": idempotencyKey }
});
return response;

} catch (error) {
if (!isTransientError(error)) {
throw error; // Stop immediately for auth or invalid prompt errors
}

if (attempt === maxRetries) {
// All retries on primary model failed, initiate fallback
console.warn("Primary model failed, falling back to alternative...");
return callFallbackModel(fallbackModel, prompt, idempotencyKey);
}

// Calculate exponential backoff with jitter
const delay = (baseDelayMs * Math.pow(2, attempt)) + randomJitter(0, 500);
sleep(delay);
}
}
}

By consistently applying these patterns, you prevent hiccups in your applications and ensure a stable user experience. Want to know more about the broader context of scalable cloud solutions? Check out our insights at consultancy.llmnet.nl for advanced architecture patterns.