Implementing circuit breakers for unstable LLM APIs

When a traditional REST API fails, it usually returns a clear HTTP 500 status code within a few milliseconds. With Large Language Model (LLM) endpoints, an outage behaves fundamentally differently. LLM calls are computationally heavy, inherently last several seconds, and under high load do not crash outright, but exhibit exponentially increasing latency, hanging streams, and periodic timeout spikes. When hundreds of concurrent threads keep waiting on a stalling upstream provider, internal application pools, worker processes, and socket connections become completely exhausted within seconds.

To prevent failing external AI services from dragging down your entire backend infrastructure in a cascading failure, a robust isolation mechanism is required. In this article, we cover designing and implementing a specialized circuit breakerpattern tailored to the asymmetric latency and failure behavior of modern LLM integrations. For a broader foundation on reliable architectural designs, see the overview on building robust integrations, which explains the interplay between retries, deadlines, and circuit isolation at the system level.

Why traditional circuit breakers fall short with LLMs

The classic circuit breaker pattern, known from distributed microservices, counts the number of consecutive failed network requests. Once a preset threshold is exceeded, the circuit trips open and subsequent requests are immediately rejected locally to relieve the underlying system. While this logic is effective for fast RPC calls, the default configuration fails for LLM gateways due to three specific characteristics of language models:

• Extreme latency variation: A successful generation can vary between 800 milliseconds and 45 seconds depending on prompt size and output length. A generic timeout triggers too quickly on long prompts or too slowly on actual network congestion.
• Streaming and time-to-first-token (TTFT): Errors often do not occur during the initial TCP handshake, but halfway through a Server-Sent Events (SSE) stream. A connection drop after generating 200 tokens requires a different error classification than an immediate rejection.
• Asynchronous rate limits (HTTP 429): Providers enforce separate limits for Requests Per Minute (RPM) and Tokens Per Minute (TPM). A 429 response does not necessarily mean the provider is offline, but requires a specific backoff procedure rather than a hard circuit block.

When an upstream provider degrades and response times rise from 2 seconds to 30 seconds, incoming requests keep piling up. Without a circuit breaker, worker threads remain blocked until the hard timeout expires. This leads to memory exhaustion in your API gateway and renders applications unreachable for end users, even for features that do not require AI.

The Three States: Closed, Open, and Half-Open in an LLM Context

A circuit breaker's state machine consists of three main phases. To function effectively with LLM endpoints, transitions between these states must account for the dynamics of token generation and latencies.

The critical difference with LLMs lies in the Half-Open state. Because a single model call can take several seconds, a circuit breaker in the Half-Open state must never immediately allow dozens of concurrent requests through. A single heavy prompt can overload the provider again. Instead, strict probe concurrency must be applied: only 1 or 2 requests at a time are granted access, strictly monitoring both the HTTP status code and the Time-To-First-Token (TTFT).

Error Criteria and Metric Tracking for LLM Calls

Not every failed API call should count toward tripping the circuit breaker. Categorizing errors prevents legitimate invalid user input from unnecessarily tripping the circuit.

When a request fails due to network errors, it is important to carefully regulate retries. Consult the article on retries and exponential backoff to see how individual retry mechanisms should interact with the overarching circuit breaker state, preventing a failing provider from being bombarded with requests.

Sliding window algorithms: Count-based versus Time-based

To determine whether the circuit should trip, the breaker evaluates requests within a sliding window. There are two dominant approaches: count-based and time-based.

Count-based sliding window

With a count-based window, the system analyzes the last N requests (for example, the last 100 calls). If more than a specific percentage (for example, 50%) fails or exceeds the deadline, the circuit opens. This mechanism works exceptionally well with continuous, steady traffic flows, but responds slowly to low traffic volumes outside office hours.

Time-based sliding window

A time-based window measures performance over the past T seconds (for example, the last 60 seconds), divided into smaller buckets of 5 or 10 seconds. For LLM gateways, the time-based approach with a minimum volume threshold is preferred. Because LLM outages often manifest suddenly during peak hours, a time window of 30 to 60 seconds ensures a rapid response without old historical errors continuing to affect the current state.

Additionally, it is essential to maintain tight deadline budgets within each window. The article on timeouts and cancellation explains how to use AbortController and context deadlines prevents 'zombie requests' from silently continuing to consume resources while the breaker is already tripped.

Calibrating state transitions and thresholds

Determining the right thresholds is a delicate balance between protection and availability. Overly aggressive settings trigger unnecessary provider switches (flapping), while overly conservative settings cause your own backend to become saturated.

The following parameters form the standard configuration for a robust production LLM circuit breaker:

• Minimum sample volume (Minimum Request Volume): Set this to a minimum of 10 requests within the window. This prevents the circuit from opening immediately if 2 out of 2 nighttime requests happen to time out.
• Failure threshold (Failure Rate Threshold): A rate of 40% to 50% failed calls within the time window is typical to tolerate occasional network hiccups, while intercepting structural outages in a timely manner.
• Slow call threshold (Slow Call Rate Threshold): A request taking longer than 3 times the provider's average p95 latency (e.g., > 15 seconds for a standard completion) should be flagged as a 'slow call'. If 60% of calls are 'slow', the circuit opens, even if the calls technically succeed in the end.
• Cooldown period (Wait Duration in Open State): The wait duration before the Half-Open state is activated. For LLM providers, 30 to 60 seconds is ideal. Shorter durations give overloaded clusters insufficient recovery time.
• Permitted Probes in Half-Open: Limit the number of test requests in the Half-Open state to exactly 2 to 5 sequential calls.

Fallback strategies for an open circuit

As soon as the circuit breaker reaches the status Open , the application must follow a defined fallback path. A circuit breaker that simply sends an error message to the end user only solves half the problem. The real power lies in seamlessly switching to alternative processing paths.

When the primary LLM provider fails, there are three proven patterns to maintain continuity:

1. Provider Failover (Multi-Provider Routing): Traffic is immediately routed to a secondary provider with a comparable model family (for example, from a primary inference cluster to an alternative provider).
2. Model Downgrading: The request switches to a smaller, faster, and local or more reliable model with lower latency and ample capacity.
3. Graceful Degradation: Complex agentic tasks are scaled back to deterministic searches, static responses, or asynchronous queues.

For a detailed breakdown of these architectures and how to keep application logic intact during partial outages, read the guide on graceful degradation during LLM outages. With automated failover, always ensure that idempotency keys are preserved, as described in the guide on idempotency in LLM calls, so that end users are never double-billed for a partially completed generation during a sudden provider switch.

In complex autonomous systems, open circuits can also trigger chain reactions within an agent's decision tree. To understand how this manifests in cyclical invocations, check out the analysis on debugging agentic loops on our community platform.

Implementation Example: Asynchronous Python Circuit Breaker

The production-oriented implementation below demonstrates an asynchronous circuit breaker in Python. This breaker monitors TTFT and status codes, utilizes a time window with lock synchronization, and supports safe probing in the Half-Open state.

import asyncio
import time
from enum import Enum
from typing import Callable, Any, Dict, Optional

class CircuitState(Enum):
  CLOSED = "CLOSED"
  OPEN = "OPEN"
  HALF_OPEN = "HALF_OPEN"

class LLMCircuitBreaker:
  def __init__(
    self,
    name: str,
    failure_threshold: float = 0.5,
    recovery_timeout: float = 30.0,
    min_samples: int = 5,
    sample_window: float = 60.0,
    slow_call_duration: float = 12.0
  ):
    self.name = name
    self.failure_threshold = failure_threshold
    self.recovery_timeout = recovery_timeout
    self.min_samples = min_samples
    self.sample_window = sample_window
    self.slow_call_duration = slow_call_duration

    self.state = CircuitState.CLOSED
    self.history = []  # tuples van (timestamp, is_failure, is_slow)
    self.last_state_change = time.time()
    self.lock = asyncio.Lock()
    self.half_open_in_flight = 0

  def _clean_history(self, now: float):
    cutoff = now - self.sample_window
    self.history = [h for h in self.history if h[0] > cutoff]

  async def can_execute(self) -> bool:
    async with self.lock:
      now = time.time()
      if self.state == CircuitState.OPEN:
        if now - self.last_state_change >= self.recovery_timeout:
          self.state = CircuitState.HALF_OPEN
          self.last_state_change = now
          self.half_open_in_flight = 1
          return True
        return False

      if self.state == CircuitState.HALF_OPEN:
        if self.half_open_in_flight < 2:
          self.half_open_in_flight += 1
          return True
        return False

      return True

  async def record_result(self, success: bool, duration: float):
    async with self.lock:
      now = time.time()
      is_slow = duration > self.slow_call_duration
      is_failure = (not success) or is_slow

      if self.state == CircuitState.HALF_OPEN:
        if success and not is_slow:
          self.state = CircuitState.CLOSED
          self.history.clear()
        else:
          self.state = CircuitState.OPEN
        self.last_state_change = now
        self.half_open_in_flight = 0
        return

      if self.state == CircuitState.CLOSED:
        self.history.append((now, is_failure, is_slow))
        self._clean_history(now)

        if len(self.history) >= self.min_samples:
          failures = sum(1 for h in self.history if h[1])
          rate = failures / len(self.history)
          if rate >= self.failure_threshold:
            self.state = CircuitState.OPEN
            self.last_state_change = now

  async def execute(self, func: Callable, fallback: Optional[Callable] = None, *args, **kwargs) -> Any:
    if not await self.can_execute():
      if fallback:
        return await fallback(*args, **kwargs)
      raise RuntimeError(f"Circuit '{self.name}' is OPEN: LLM upstream niet beschikbaar.")

    start = time.time()
    try:
      result = await func(*args, **kwargs)
      await self.record_result(success=True, duration=time.time() - start)
      return result
    except Exception as exc:
      await self.record_result(success=False, duration=time.time() - start)
      if fallback:
        return await fallback(*args, **kwargs)
      raise exc

In this example, the circuit breaker not only monitors whether the call crashes with an exception, but also whether the execution time exceeds the slow_call_duration . An LLM provider that takes 20 seconds for a simple prompt is thus effectively isolated before all available application sockets become blocked.

Production Pitfalls: Thundering Herds and Distributed State

Running circuit breakers at production scale introduces specific concurrency and architectural challenges. The two most common operational risks are:

1. The Thundering Herd on State Reset

When a circuit transitions from Open to Half-Open and subsequently recovers to Closed, thousands of waiting clients simultaneously see that the provider is 'healthy' again. If all these clients fire their deferred requests at once, the newly recovered provider crashes again immediately. This cyclical phenomenon (flapping) must be prevented by:

• Adding jitter to the cool-down timer, so that different worker nodes do not execute their Half-Open probe at the exact same second.
• Using a rate limiter (token bucket) that gradually ramps up throughput after recovery instead of immediately allowing 100% capacity.

2. In-Memory versus Distributed Circuit State (Redis)

If your application runs across dozens of Kubernetes pods or serverless containers, the state of the circuit must be coordinated. A local in-memory circuit per container works fine with high volumes per node, but in highly fragmented microservices, node A does not know that node B is continuously receiving 503 errors.

For centralized control, the state can be synchronized via Redis using fast Lua scripts or Redis hashes. However, take the network latency to Redis into account: the check should not take more than 1-2 milliseconds to avoid unnecessarily burdening the total processing time.

Monitoring, alerting, and integration with runbooks

A transformer-based circuit breaker must be visible in real-time within your observability stack. When a circuit changes state, it is by definition an operational event that warrants alerting.

Ensure the following metrics in Prometheus, Datadog, or OpenTelemetry:

• llm_circuit_breaker_state{name="openai-gpt4o"}: Gauge metric (0 = Closed, 1 = Half-Open, 2 = Open).
• llm_circuit_breaker_rejections_total: Counter metric for the number of locally rejected or rerouted requests.
• llm_circuit_breaker_transitions_total: Counter metric for the number of state transitions (to detect flapping).

Link alerts on status Open directly to your incident response procedures. When the circuit remains open for longer than 5 minutes, the monitoring platform should automatically escalate to the on-call engineers. This serves as the direct trigger for standardized incident handling, as documented in the runbook for an LLM outage.

Conclusion

Circuit breakers are not a luxury, but an absolute prerequisite for reliable LLM integrations in production. By explicitly managing latency thresholds, streaming interruptions, and strict probe volumes in the Half-Open phase, you prevent an unstable AI provider from bringing down your entire backend. Combined with automated fallbacks and accurate telemetry, you build a resilient architecture that guarantees operational continuity, regardless of the vagaries of external cloud models.