Streaming with Fallback: Handling Partial Responses Gracefully

Streaming via Server-Sent Events (SSE) has become the standard method for drastically reducing perceived wait time (Time to First Token) in interactive LLM applications. Where a complete generation can take several seconds, streaming lets the user see the first words appear on screen immediately. However, this interactive speed comes with a significant technical downside: as soon as the HTTP connection drops midway or the provider returns an error, the client has already received and displayed a partial response. A simple HTTP status code is no longer enough at that point to enforce clean error handling.

In classic stateless API architectures, a failed request can easily be handled with an automated retry to a secondary provider. Anyone who wants to understand the basic principles of fault tolerance can review retries, timeouts, and fallbacks in LLM integrations to see how circuit breakers and backoff mechanisms work for blocked calls. With streaming, that simple pattern breaks down: after all, the client has already consumed data. In this article, we analyze the four failure modes of streaming, design state-tracking mechanisms in the application layer, and show robust fallback patterns for continuing or completing partial responses in a controlled way.

The four failure modes of streaming connections

When an LLM call fails before the first byte is sent, the client receives a clear HTTP error status (such as 429, 500, or 503). With a streamed response, however, the HTTP status code is already fixed at 200 OK with Content-Type: text/event-stream. From that point on, four different failure situations can occur, each requiring a specific detection method.

The first category is the abrupt socket interruption. The TCP connection between the API gateway and the provider drops due to network congestion or a restarting reverse proxy. The stream stops without any prior signal and without a formal closing SSE message (such as data: [DONE]). The application only notices this because an I/O read error occurs or because a specific chunk timeout expires.

The second category involves a provider error midway through the generation process. Some infrastructures send a JSON payload with an error message partway through the SSE stream instead of a regular content-delta block. For example, a server might hit an internal memory limit after 150 generated tokens and pass through an event with {"error": {"type": "server_error", "message": "Inference backend unavailable"}} If the client naively interprets this payload as text, raw JSON code appears directly in the user interface.

The third failure mode arises from content filtering and safety moderation. Providers evaluate both the prompt and the generated output. When the active generation collides with a safety filter after a few sentences, the model stops abruptly and the last SSE block contains a finish_reason: "content_filter" instead of "stop" or "length". This is not an infrastructure failure but a policy-based interruption that should not simply be retried with an arbitrary fallback.

The fourth category is exceeding token limits. If the sum of the input tokens and the maximum requested output tokens exceeds the context window, or if the parameter max_tokens is reached before the thought is complete, the stream returns finish_reason: "length". The sentence then stops midway. To get a grip on the raw structure of these kinds of network streams, you can study how streaming responses work in LLM APIs, which breaks down the payload specifications of Server-Sent Events in detail.

The UX dilemma: flicker, clear, or continue

When a stream suddenly drops after 200 words, the user interface must decide what happens to the text already shown. In poorly designed applications, we often see one of two extremes: the text suddenly disappears and is replaced by a generic error message ("Something went wrong"), or the page reloads without warning, causing the text to vanish and start typing again from token zero. Both scenarios cause frustration and confusion for the end user.

Suddenly clearing text that's already been generated wastes the cognitive processing effort of the reader, who may have already been reading the first paragraphs. On the other hand, simply leaving an interrupted text as-is carries serious risks: an incompletely generated piece of legal advice or a half-cut-off software algorithm can be factually incorrect or dangerous. In critical applications, users must always be able to verify the validity of incomplete outputs; see the guidelines on systematically fact-checking AI responses to assess how human reviewers can validate partial or inconsistent outputs.

A well-designed architecture therefore uses a clear state machine on the frontend. A response is in one of four states: STREAMING, COMPLETED, INTERRUPTED or FAILED. On interruption, the text already shown is preserved in the state INTERRUPTED, accompanied by a visual status bar indicating that the connection was interrupted, including an interactive action button to resume generation from the last point.

Architecture of a state-aware gateway buffer

To enable recovery and fallback without burdening the client with complex routing logic, we place an application gateway between the client and the upstream AI providers. This gateway acts as a state-aware proxy that inspects, accumulates, and forwards every incoming chunk.

The gateway maintains a temporary session buffer in memory for each active streaming request (for example, via a local buffer structure or Redis for distributed gateways). This buffer stores three crucial values: the accumulated text (the merged content deltas), the number of chunks received, and the last received metadata (such as model name and finish reasons). If the upstream provider drops during the stream, the gateway knows exactly how much text has already been successfully passed on to the client.

Anyone considering housing such routing and failover in a central infrastructure layer can explore the architecture of an LLM aggregator to see how multi-provider gateways centrally handle load balancing, rate limits, and fallback rules. In a streaming context, this central layer makes it possible to redirect an interrupted request directly to an alternative provider.

Three strategies for recovering interrupted streams

When the backend determines that a stream has been permanently interrupted, three fundamental patterns are available to resolve the incident. The choice of pattern depends on the model type, the cost structure, and the desired user experience.

1. The Prefix Resume strategy (Continuation)

With this strategy, the gateway tries to let the generation continue seamlessly from the point of failure. The gateway takes the original system prompt and user input, and adds the already generated text as a provisional assistant response (also known as assistant prefill or prefix prompting). The request is then sent to a secondary provider with instructions to finish the response from that exact point.

The new chunks from the secondary provider are then pasted directly onto the already open SSE stream to the client. To the end user, it looks as if there was only a brief half-second pause in the typing, after which the text simply continues. The limitation here is that not all commercial APIs allow prefilling; for models that don't support this, the accumulated text must be injected into a repeated user message with an explicit instruction ("Continue directly with the text below, without an introduction: ...").

2. The Clean Slate Retry strategy (Fresh restart)

If the quality difference between models is too great to mix styles mid-paragraph, or if it concerns structured code, a full restart with a secondary provider is chosen. The gateway sends a special SSE control event to the client, for example event: reset_stream with payload {"reason": "provider_fallback"}.

This event tells the client application that the earlier buffer must be cleared or visually marked as 'restarted,' after which the new stream is built up from the beginning. This guarantees logical consistency across the entire response, but comes with higher token costs and a longer total wait time.

3. The Graceful Degradation strategy (Partial acceptance)

When the budget is limited or when repeated calls are undesirable, the gateway closes the stream cleanly right away with a event: partial_complete. The displayed text remains on screen with a warning: "The connection was interrupted. This response may be incomplete." A "Continue generating" button appears below the block, letting the user explicitly initiate a new request that carries the earlier context forward.

Implementation: A robust SSE proxy with fallback

The Python implementation below shows how an application layer consumes an upstream streaming response, detects event errors, and switches to a secondary model on failure without breaking the connection to the browser.

import asyncio
import json
import httpx
from typing import AsyncGenerator

async def stream_met_fallback(
    prompt: str,
    primaire_url: str,
    secundaire_url: str,
    api_sleutel_primair: str,
    api_sleutel_secundair: str,
    chunk_timeout: float = 3.5
) -> AsyncGenerator[str, None]:
    geaccumuleerde_tekst = []
    stream_succesvol = False
    
    # Poging 1: Primaire provider aanroepen
    headers_primair = {
        "Authorization": f"Bearer {api_sleutel_primair}",
        "Content-Type": "application/json"
    }
    payload_primair = {
        "model": "model-alpha",
        "messages": [{"role": "user", "content": prompt}],
        "stream": True
    }
    
    try:
        async with httpx.AsyncClient(timeout=httpx.Timeout(connect=5.0, read=chunk_timeout, write=5.0, pool=5.0)) as client:
            async with client.stream("POST", primaire_url, headers=headers_primair, json=payload_primair) as response:
                if response.status_code != 200:
                    raise httpx.HTTPStatusError("Primaire provider gaf foutcode", request=response.request, response=response)
                
                async for regel in response.aiter_lines():
                    if not regel.startswith("data: "):
                        continue
                    data_str = regel[6:].strip()
                    if data_str == "[DONE]":
                        stream_succesvol = True
                        yield "data: [DONE]\n\n"
                        break
                    
                    data = json.loads(data_str)
                    # Controleer op expliciete foutpayloads binnen de 200 OK stream
                    if "error" in data:
                        raise RuntimeError(f"Fout in stream payload: {data['error']}")
                    
                    delta = data.get("choices", [{}])[0].get("delta", {}).get("content", "")
                    if delta:
                        geaccumuleerde_tekst.append(delta)
                        yield f"data: {json.dumps({'content': delta})}\n\n"
                        
    except (httpx.TransportError, httpx.HTTPStatusError, asyncio.TimeoutError, RuntimeError):
        # Primaire stream faalde halverwege; we schakelen over naar de fallback
        pass

    if stream_succesvol:
        return

    # Poging 2: Terugval naar secundaire provider met geaccumuleerde context
    reeds_gegenereerd = "".join(geaccumuleerde_tekst)
    headers_secundair = {
        "Authorization": f"Bearer {api_sleutel_secundair}",
        "Content-Type": "application/json"
    }
    
    # We instrueren het secundaire model om exact aan te sluiten
    berichten = [
        {"role": "user", "content": prompt}
    ]
    if reeds_gegenereerd:
        berichten.append({"role": "assistant", "content": reeds_gegenereerd})
        berichten.append({"role": "user", "content": "Ga direct verder vanaf het exacte punt waar je vorige bericht stopte."})

    payload_secundair = {
        "model": "model-beta",
        "messages": berichten,
        "stream": True
    }

    try:
        async with httpx.AsyncClient(timeout=httpx.Timeout(connect=5.0, read=chunk_timeout, write=5.0, pool=5.0)) as client:
            async with client.stream("POST", secundaire_url, headers=headers_secundair, json=payload_secundair) as response:
                if response.status_code != 200:
                    yield f"event: error\ndata: {json.dumps({'message': 'Beide providers onbereikbaar'})}\n\n"
                    return
                
                async for regel in response.aiter_lines():
                    if not regel.startswith("data: "):
                        continue
                    data_str = regel[6:].strip()
                    if data_str == "[DONE]":
                        yield "data: [DONE]\n\n"
                        break
                    
                    data = json.loads(data_str)
                    delta = data.get("choices", [{}])[0].get("delta", {}).get("content", "")
                    if delta:
                        yield f"data: {json.dumps({'content': delta, 'fallback': True})}\n\n"
    except Exception as e:
        yield f"event: error\ndata: {json.dumps({'message': 'Onherstelbare streamingfout'})}\n\n"

Key management and authentication for cross-provider switching

A seamless switch during an active stream requires secondary API credentials to be immediately ready in runtime memory. If a gateway first has to consult a slow database call or an external secret manager at the moment a stream fails, the client experiences a noticeable stutter of hundreds of milliseconds. This often causes the client to exceed its own socket timeout.

The gateway should therefore keep all necessary tokens for both primary and secondary providers pre-warmed in a secure cache. When setting up such infrastructure, it's essential to apply strict separation of privileges; read more about how you can securely manage API keys for LLMs without risk of cross-tenant data leaks or unauthorized access. Rotate keys automatically in the background and make sure that failing authentication on a secondary key immediately triggers an alarm in the logging.

Timeouts and deadline budgets in streaming

For normal RPC calls, one usually applies a single total timeout (for example, 10 seconds). With streaming, this works fundamentally differently: a total response time of 30 seconds can be perfectly acceptable as long as a new word arrives every 50 milliseconds. That's why two separate timeouts need to be configured:

First, there is the Time to First Token (TTFT) timeout. If the provider doesn't send the very first chunk within, say, 3 to 5 seconds after accepting the connection, the call is canceled and the system switches directly to the fallback provider. After all, nothing has been sent to the end user yet, making a clean failover trivial.

Second, there is the Inter-chunk timeout (or idle read timeout). Once the stream is running, the time between two consecutive chunks should rarely exceed 2 to 4 seconds. If the stream exceeds this threshold, the upstream server is likely in a 'hanging socket' state, and the gateway must actively close the connection. To see how you mathematically divide deadline budgets across distributed subsystems, we refer to timeouts, cancellation and deadline budgets in LLM calls.

Peculiarities with structured JSON streams

Handling interrupted streams becomes considerably more complex when the model generates not free text but structured JSON data or function calls via SSE. If a JSON stream is interrupted midway, the client is left with an invalid data fragment, such as:

{"klant_id": 4921, "analyse": "De jaarcijfers tonen een stijging", "risico_factoren": ["liquiditeit", "valuta

Standard JSON parsers such as JSON.parse() in JavaScript or json.loads() in Python will crash immediately on these unbalanced brackets and unclosed strings. Specialized recovery measures are therefore necessary for structured streaming applications:

First, tolerant streaming parsers (such as streaming JSON tokenizers) can be used, which dynamically close partial objects. These libraries virtually fill in missing quotation marks, braces, and brackets, so the application can already render the already-validated fields (such as klant_id) in forms or tables.

Second, for function calling, an interrupted tool call must never be partially executed. If a model was in the process of generating the arguments for a function voer_betaling_uit({"bedrag": 500, "ontvanger": ...}) and the connection drops before the full JSON structure has been validated, the gateway must invalidate the entire call and refuse to invoke the backend function. Tool execution may only take place after a successful, complete retry.

Production checklist for streaming robustness

Before a streaming integration goes into production, it's advisable to check the implementation against the technical criteria below:

1. Configure aggressive chunk timeouts: Never rely on the operating system's default TCP timeouts, which can wait minutes for a dead socket. Use an inter-chunk timeout of at most 3 to 5 seconds.

2. Disable compression buffering on proxies: Make sure intermediate reverse proxies (such as Nginx, Cloudflare, or Traefik) disable response buffering directly (for example, via the HTTP header X-Accel-Buffering: no). If a proxy accumulates chunks to send them compressed in 4KB batches, the stream is delayed and the client timeout may fire incorrectly.

3. Use explicit SSE error events: On a backend failure, always send a formal SSE event with event: error and a JSON body instead of abruptly closing the TCP connection. This way, the frontend knows exactly what's going on and can display targeted error information.

4. Account for model inconsistency when prefixing: Test in advance whether the fallback model can smoothly continue building on text started by a different base model. Some models repeat the last three words of the prefix or respond with a different form of address (informal versus formal). A normalization layer in the gateway can filter out minor style breaks.

Trade-offs and conclusion

Streaming offers an unmatched interactive user experience, but transforms a simple request-response interaction into a distributed stateful transaction. By buffering chunks at the gateway level, strictly enforcing inter-chunk timeouts, and deploying well-designed fallback strategies such as prefix continuation, applications keep functioning reliably when an AI provider stumbles.

The choice between fully restarting a response and continuing in a controlled way with a secondary model ultimately comes down to a balance between token costs, latency, and editorial consistency. By explicitly building the failure modes into the frontend's state machine, end users never see a mysterious flickering screen again, but a stable, professional application that handles network errors transparently.