Designing graceful degradation for LLM outages
When an application depends on external Large Language Model (LLM) APIs, unavailability is not an occasional event but a given. Network failures, capacity problems at providers and unexpected error responses are part of daily practice. Once standard mechanisms such as retries or temporary waiting periods no longer help, a graceful degradation strategy keeps the entire application from grinding to a halt.
Graceful degradation means that a system falls back to a more limited mode in a controlled way when a dependency fails. Instead of showing a generic error message or returning a 500 error code to the user, the application delivers a partially functional answer. The art of designing robust integrations lies in deciding up front which functionality is essential and which parts can be temporarily simplified or left out.
Note: This article covers the architecture that kicks in once retries, timeouts and network recovery have been exhausted. For the basic setup of network recovery, see the articles on retries and backoff and timeouts and cancellation.
Failure modes in LLM dependencies
Building a robust degradation mechanism starts with distinguishing the different ways an LLM endpoint can fail. Each type of error requires a specific response in the application architecture.
1. Complete provider outage
During a complete outage the API endpoint no longer returns a controlled response. This can show up as an HTTP 500, 502 or 503 error code, or as the domain being entirely unreachable (DNS or TLS errors). In this scenario it is immediately clear that the endpoint is unavailable and the application can switch to an alternative route right away.
2. Sharply increased latency
A more treacherous failure mode is an extremely slow response from the model. The endpoint gives no explicit error, but the answer takes tens of seconds. For the end user this is practically the same as an outage. If the response time exceeds the configured limits, the system should abort the request and call a faster alternative.
3. Rate limiting (HTTP 429)
When application traffic exceeds the configured limits per minute (RPM) or tokens per minute (TPM), the provider returns an HTTP 429 status code. Although the service itself is operational, that specific API key cannot handle any more requests at that moment. This calls for temporarily rerouting requests to another provider or another account.
4. Truncated or stalled streams
When using streaming responses a connection can break halfway through the generated text. The system receives the first tokens correctly, but the stream goes silent without a valid closing JSON chunk or stop token arriving. The application must be able to capture the text already received and determine whether it is sufficient, or else complete the missing part via a secondary model.
5. Silent quality degradation
One of the hardest failure modes is quality degradation without any HTTP error code. Under heavy load a provider may switch to a heavily compressed quantization of a model, or internally reroute requests to a smaller model. The system does return HTTP 200 OK, but the result contains hallucinations, lacks structure or refuses to follow instructions. This requires validation logic on the receiving side that checks the output against minimum quality and structure requirements.
The hierarchy of fallback layers
An effective degradation design uses a layered structure of alternatives. When the primary route fails, the request drops down to the next available layer. This order has to be put together carefully to keep the balance between quality, latency and functionality.
| Priority | Fallback layer | Quality impact | Latency impact |
|---|---|---|---|
| 1 (Primary) | Large model at the primary provider | None (0%) | Standard |
| 2 | Another model at the same provider | Small | Equal or lower |
| 3 | Comparable model at another provider | Small to moderate | Variable |
| 4 | Local or smaller hosted model | Noticeable | Very low |
| 5 | Semantically cached answer | Depends on the hit | Extremely low |
| 6 (Last resort) | Deterministic rule-based route | Functionally minimal | Negligible |
Local model as an emergency scenario
Local or dedicated hosted models play a crucial role in the fallback chain. When external API providers are unreachable, choosing a local model can help keep critical basic functions running. By using local models behind an API the application keeps a guaranteed floor of processing capacity, regardless of external network or provider failures.
Cached answers and rule-based safety nets
If small or local models are unavailable too, or take too much time, the system can fall back on caching LLM responses. For closely similar questions, a previously approved answer is served. The last layer in the pyramid is a fully deterministic fallback. Think of a fixed template, a simple database query, or a predefined answer that helps the user directly without using artificial intelligence.
When you should NOT fail over
Automatically rerouting requests to a secondary route always looks like a good choice, but there are situations where automatic failover damages data integrity or the user experience.
- Non-idempotent actions: When the LLM call is tied to an action that changes state (such as creating a reservation, sending an email or a financial transaction), an interrupted request must not be blindly resent to another model. If the first request was processed in the background after all, a failover leads to duplicate actions.
- Tool calls with side effects: If the LLM was busy executing external functions (function calling) and the process stalls halfway through, the exact state is unknown. Failing over to another model can cause steps that have already run to be called again.
- Complex logical tasks with strict quality requirements: If the task involves drafting a legal document or a complicated code analysis, a smaller fallback model may make subtle mistakes. In that case it is better to hand back an explicit error message than to generate an answer of insufficient quality.
Circuit breakers and health signals
To keep an application from letting every single request hang on a broken external provider, a circuit breaker pattern is applied. A circuit breaker monitors the number of errors and the latency of outgoing requests and has three states:
In the Closed state all requests go to the primary provider as normal. As soon as the error rate or the average latency rises above a preset threshold during a fixed time interval, the circuit flips to Open. In this state new requests are routed straight to the fallback route, without loading the primary provider.
After a set cool-down period the breaker switches to the Half-Open state. A small percentage of test traffic is sent to the primary provider. If these test requests are processed successfully, the circuit closes again and the normal route is restored. If the tests fail, the breaker returns to the Open state immediately.
// Voorbeeld van een eenvoudige toestandcontrole in pseudo-code
if (circuitBreaker.isOpen()) {
return runFallbackRoute(request);
}
try {
Response response = primaryProvider.send(request);
circuitBreaker.recordSuccess();
return response;
} catch (ApiException e) {
circuitBreaker.recordFailure();
return runFallbackRoute(request);
}Preventing flapping
A common problem with automated failover is flapping: constantly switching back and forth between the primary route and the fallback route. This happens when the thresholds are set too strictly or the test intervals are too short. Flapping causes unpredictable latency and varying answer quality for the user. You can prevent it by using sliding-window counts for error rates, hysteresis in the thresholds and exponential waiting times for the Half-Open state.
User experience in limited mode
Graceful degradation is not only about the technical infrastructure, but also about communication toward the end user. Being transparent about the status of the system prevents frustration.
When an application switches to a more limited mode, this has to be made visually clear. That can be done with a subtle notice in the interface stating that the system is currently running in a simplified mode. Functions that depend on the failed model (such as generating images or in-depth analyses) can be temporarily disabled or shown grayed out.
By also adjusting expectations — for example by indicating that answers may be shorter than usual — the user understands why the result differs from the normal situation. More background on analyzing performance differences can be found in the article on measuring speed.
Setting quality levels in advance and testing them
The success of graceful degradation stands or falls with preparation. It is unwise to start thinking about which compromises are acceptable only during an active outage. Teams have to establish per function, up front, what the minimum acceptable quality level is (the so-called Minimum Viable Service Level).
The process for setting this up covers four clear steps:
- Function inventory: Map out all LLM-dependent functions in the application and categorize them by how critical they are (for example critical, important, optional).
- Fallback definition: Determine per function which alternative route (smaller model, cache, static text) is used during an outage and what the impact on the output is.
- Automated evaluation: Set up automated test sets that continuously check the quality of the fallback routes against benchmark questions.
- Chaos engineering: Actively simulate outages in a test or staging environment. Block API keys, introduce artificial delays and cut network connections to validate whether the circuit breakers and fallback chains work correctly.
By testing these mechanisms periodically, the system stays prepared for real outages and a high level of availability can be guaranteed, even when the underlying AI providers are dealing with serious interruptions.
