Timeouts, cancellation, and deadline budgets in LLM calls

In the world of traditional API development, most network requests take a few tens to hundreds of milliseconds. However, as soon as we start working with Large Language Models (LLMs), the rules of the game change drastically. A simple LLM call can easily take five seconds, and a complex generation task can run up to more than thirty seconds. This fundamental difference in response times introduces significant risks to the stability of your application.

If you don't account for these delays, slow LLM calls can block your entire architecture, exhaust connection pools, and lead to skyrocketing costs. In this article, we dive deep into the strategies for handling these unpredictable response times. We discuss why slow calls are so dangerous, how to set sensible timeouts per task type, the importance of correctly aborting streams, and how to manage a so-called 'deadline budget' across multiple calls.

Why slow LLM calls block your entire chain

When a web application or backend service receives an incoming request, the server typically assigns a "worker", thread, or connection to it. This resource remains reserved until the request is fully handled. This model works excellently when API calls are fast, but it fails spectacularly in the context of LLMs if you are not careful.

Suppose your server has a limit of 100 concurrent connections. If each LLM call takes 10 seconds, your server can process a maximum of 10 requests per second (100 / 10). If traffic exceeds this, queues fill up. New users no longer receive responses, and the server eventually denies service. This phenomenon is known as resource exhaustion or thread starvation.

Additionally, you have to deal with the ripple effect (cascading failure) in microservice architectures. If microservice A depends on microservice B, and B is slowly waiting for an LLM provider (such as OpenAI, Anthropic, or a local open-source model), then service A also hangs. Without tightly configured timeouts, the delay spreads to your entire infrastructure. This requires building robust integrations where failures are caught and isolated in time.

Sensible timeouts per task type

A common mistake is setting a single, global timeout for all LLM interactions. A generic number (for example, 30 seconds) is rarely appropriate. The nature of the prompt determines the expected computation time. We distinguish between two types of time measurements in LLMs: Time-To-First-Token (TTFT) and Total Generation Time.

The TTFT is the time the model needs to process the prompt and return the very first word. Once the first word is generated, the remaining time is directly dependent on the number of tokens in the output. A short classification task is finished quickly, while writing an extensive report takes much longer. Therefore, tailor your timeouts to the specific use case.

Aborting streaming: how and why

Because text generation by an LLM can take so long, almost all modern AI applications use streaming (often via Server-Sent Events, or SSE). This allows you to see the text appear on the screen immediately. You can read more about implementing this correctly in our article on streaming responses.

However, what happens if the user closes their browser, or clicks a 'Stop Generating' button while the stream is halfway through? In a naive implementation, your backend server remains connected to the LLM provider, generates all remaining tokens, and then discards them. This not only wastes network traffic, but you also pay for all generated tokens per API call (after all, with many models you pay per thousand tokens).

It is therefore essential to propagate a cancellation signal. When the TCP connection with the client breaks (for example, because the user navigates away), your backend must detect this. As soon as this happens, you should immediately close or cancel the underlying HTTP call to the LLM API. Almost all official SDKs from model providers now support so-called abort controllers (in JavaScript) or context signals (in Go/Python) to cleanly abort an ongoing stream on the API side.

Clean retries after a timeout

When an LLM call fails with a timeout, the instinctive reaction of many developers is to automatically resubmit the request (a "retry"). While retries are useful for temporary network outages (such as a 502 Bad Gateway), they are highly dangerous for timeouts without a well-thought-out plan.

Imagine that an LLM provider is temporarily overloaded. Requests take longer than the set 15 seconds and end in a timeout. If your application immediately (and repeatedly) retries the same request, you double or triple the load on the already struggling API. This leads to a so-called Thundering Herd or Retry Storm, which only worsens the situation.

Retries after a timeout must therefore be strictly controlled. Always use a mechanism with exponential backoff and "jitter" (randomness). Check out our detailed article on retries and backoff strategies for the exact implementation details. Additionally, with timeouts, it is crucial to remember that you cannot be sure if the API partially processed your request. For pure (read) generation tasks, this is not an issue, but be careful if your LLM also performs actions (such as calling a database in an agentic workflow).

The Concept of Deadline Budgets (Total Request Budget)

Now that we have discussed the basics of timeouts, we come to the most advanced, yet most crucial concept in AI architecture: the Deadline Budget.

In many modern AI systems, especially in basic RAG systems (Retrieval-Augmented Generation), a single user request consists of multiple sequential steps. A typical flowchart looks like this:

1. The user asks a question via the frontend (HTTP Request).
2. Your API Gateway allows a maximum of 10 seconds for the entire process.
3. You call an embedding model to vectorize the question (takes 1.5 seconds).
4. You search the vector database for relevant context (takes 0.5 seconds).
5. You send the context and the question to the main LLM.

If you set a hardcoded timeout of 10 seconds on the LLM call in step 5, you run into a major problem. The first two steps have already consumed 2 seconds. If the LLM takes 9 seconds, the call remains successful within the LLM timeout of 10 seconds. However, the total processing time is now 11 seconds (2 + 9). This exceeds the 10-second limit of your API Gateway. The gateway will abort the request to the user with a "504 Gateway Timeout", while your backend is uselessly burning money on the LLM API in the background.

To solve this, you use a Deadline Budget. At the start of the request, you calculate the absolute final deadline, and for each subsequent action, you calculate how much "budget" (time) is left.

Concrete Pattern: Context and Cancellation in Practice

To successfully implement a deadline budget, you must dynamically pass the remaining time to your LLM client. Here is a conceptual example of how to implement this in Python. The pattern ensures that we never wait longer for the LLM than the total budget we have available.

import time
import asyncio
from typing import Optional

class DeadlineBudget:
    def __init__(self, total_budget_seconds: float):
        self.start_time = time.time()
        self.total_budget = total_budget_seconds

    def remaining(self) -> float:
        elapsed = time.time() - self.start_time
        left = self.total_budget - elapsed
        return max(0.0, left)

    def is_expired(self) -> bool:
        return self.remaining() <= 0

async def handle_rag_request(user_query: str):
    # The API Gateway accepts a maximum of 10 seconds for the entire chain
    budget = DeadlineBudget(total_budget_seconds=10.0)

    # Step 1: Embeddings (takes some time)
    if budget.is_expired():
        return "Timeout before embedding"
    vector = await get_embeddings(user_query, timeout=budget.remaining())

    # Step 2: Database query
    if budget.is_expired():
        return "Timeout before database query"
    context = await fetch_from_vector_db(vector, timeout=budget.remaining())

    # Step 3: LLM Call with the REMAINING budget
    time_left_for_llm = budget.remaining()
    if time_left_for_llm <= 0.5:
        # If we have less than half a second left, it is useless to
        # start another heavy LLM call. Abort immediately.
        return "Not enough time left to generate a response."

    try:
        # We pass exactly the remaining time as the timeout for this call
        response = await call_llm_api(
            prompt=user_query, 
            context=context, 
            timeout=time_left_for_llm
        )
        return response
    except asyncio.TimeoutError:
        return "The LLM was too slow and exceeded the deadline budget."

Important insight: Note in the code above that in step 3 we check if we still have a realistic amount of time left (0.5 seconds). Even the fastest models have a TTFT of hundreds of milliseconds. If there is almost no budget left, the chance of success is zero. It is cheaper (and faster for the user) to fail immediately.

Conclusion & Best Practices

Working with LLM APIs requires a fundamentally different approach to network timeouts than you might be used to. In summary, these are the most important guidelines for a healthy, cost-efficient application architecture:

• Differentiate your timeouts: Use short timeouts for simple routing tasks and longer (but strict) timeouts for complex reasoning tasks or large blocks of text.
• Actively abort unwanted streams: Link the closing of the client connection to an abort signal to the LLM provider to prevent unnecessary token costs.
• Be careful with blindly executing retries: A timeout often means the service on the other side is struggling. Do not hammer it; use exponential backoff.
• Integrate Deadline Budgets: In a chain of operations, always pass the remaining time as the timeout value for the next step, instead of using fixed values per step.

By applying these strategies, you prevent failing AI models from causing a domino effect in your backend, keep your API costs under control, and significantly improve reliability and user experience.