Share:𝕏LinkedInRedditFacebookCopy link

Idempotency in LLM API Calls

By Ivo Donker — compiled with AI assistance (Claude & Gemini)

In distributed systems, network error handling is a constant challenge. When your application communicates with external Large Language Model (LLM) APIs, slow response times and high compute costs amplify this problem. Blindly retrying a failed HTTP request can lead to duplicate billing, inconsistent system states, and unwanted actions in external systems.

Idempotency is the design principle that guarantees that executing the same operation multiple times yields exactly the same result as a single execution. This article covers why LLM calls are inherently non-idempotent, how to build a robust idempotency pattern, and how to handle side effects and partially failed streaming responses.

Why Duplicate Processing Is a Real Problem

Duplicate processing in API interactions rarely occurs because a developer accidentally calls the same function twice. It is almost always the result of underlying infrastructure and human behavior:

Why an LLM Call Is Inherently NOT Idempotent

Unlike an HTTP GET request or a SQL UPDATE that sets a fixed value, a standard HTTP POST to a /v1/chat/completions endpoint is inherently non-idempotent in three respects:

1. Direct Financial Damage

Every processed prompt costs tokens. When your system sends a 4,000-token prompt three times due to a network outage, you pay three times for the same input. At scale, this leads to a significant leak in your API budget.

2. Non-Deterministic Output

Unless the temperature parameter is strictly set to 0.0 (and even then, not all providers guarantee determinism due to parallel GPU computations), each execution of the same prompt produces different text. If two parallel workers execute the same task, they generate two divergent responses. Which of the two do you store in the database? This causes race conditions and data inconsistency.

3. Side Effects (Function Calling and Agentic Workflows)

When an LLM is deployed as an agent with 'function calling' capabilities, a duplicate call can have far-reaching consequences. A model instructed to "send the customer a confirmation email and create a ticket" will create two tickets and send two emails upon duplicate processing.

Key rule: Treat every LLM API call as an operation with side effects. Never rely on the provider to automatically deduplicate the call for you, unless the provider explicitly supports an idempotency header.

The Idempotency Key Pattern

To make a non-idempotent LLM call idempotent, we introduce a unique Idempotency Key. This pattern works on the principle that the requesting party sends a unique identifier with the request. The receiving or intermediate layer checks this identifier before the LLM is called.

Some LLM providers support a custom HTTP header for idempotency (such as Idempotency-Key: <key>). The exact header name and support vary by provider. If the provider does not support this, you must implement this layer in your own application architecture or API gateway.

How Do You Choose or Derive the Key?

The key must represent the unique intent of the operation:

Retention and Storage

Store the status of idempotency keys in a fast, central key-value store such as Redis. The time-to-live (TTL) of the key depends on the use case, but 24 to 48 hours is sufficient for most LLM applications to absorb network retries and queue replays.

Implementation: Deduplication in Your Own Application Layer

The pseudocode below illustrates how to build an idempotency structure around an LLM API call using Redis. This prevents two concurrent requests from executing the same LLM call and ensures that a later retry receives the already generated response.

import redis
import time
import json

db = redis.Redis(host='localhost', port=6379, db=0)

def execute_idempotent_llm_call(idempotency_key, prompt_data):
    lock_key = f"lock:{idempotency_key}"
    response_key = f"response:{idempotency_key}"
    
    # 1. Check whether the result already exists
    cached_response = db.get(response_key)
    if cached_response:
        return json.loads(cached_response), "CACHE_HIT"

    # 2. Try to acquire a distributed lock (atomic SETNX)
    # TTL of 60 seconds prevents a crashed worker from blocking the key forever
    acquired = db.set(lock_key, "PROCESSING", nx=True, ex=60)
    
    if not acquired:
        # Another worker is currently processing this request. Wait and poll.
        return wait_for_concurrent_request(response_key)

    try:
        # 3. Execute the actual LLM API call
        llm_result = call_external_llm_api(prompt_data)
        
        # 4. Store the result with a TTL of 24 hours (86400 sec)
        db.set(response_key, json.dumps(llm_result), ex=86400)
        
        return llm_result, "EXECUTED"

    finally:
        # Remove the lock once the work is done (or has failed)
        db.delete(lock_key)

def wait_for_concurrent_request(response_key, timeout=30):
    start = time.time()
    while time.time() - start < timeout:
        cached_response = db.get(response_key)
        if cached_response:
            return json.loads(cached_response), "CACHE_HIT_WAIT"
        time.sleep(0.5)
    raise TimeoutError("LLM processing took too long for concurrent request.")

Note that this pattern differs from general response caching. While caching is intended to improve performance and reduce costs for identical questions, this pattern is specifically designed for correctness and preventing duplicate processing in error situations. Consult our article on caching LLM responses for the differences in implementation.

Deduplicating API Calls vs. Deduplicating Side Effects

Securing the HTTP call to the LLM provider is only half the solution. In modern software architectures, an LLM doesn't just process text; it generates structured data that drives follow-up actions.

There is an essential difference between two levels of deduplication:

  1. Transport level (API call): Preventing the HTTP request to the LLM provider from being executed twice.
  2. Domain level (Side effects): Preventing the action resulting from the LLM output (for example, updating a database or sending a push notification) from being executed twice.

Suppose the LLM call succeeds and stores the response in the cache, but the worker crashes just before the follow-up information is written to the CRM database. On the next queue retry, the worker reads the response directly from the cache (transport idempotency succeeded), but still executes the CRM update for the first time.

To achieve domain idempotency, you must perform database operations using upserts (INSERT ... ON CONFLICT DO UPDATE) and call external systems with unique transaction IDs derived from the original idempotency_key.

Handling Partial Failures and Streaming Responses

Streaming responses (via Server-Sent Events or WebSockets) pose a particular challenge to idempotency. Because the data arrives in chunks, a connection can break halfway after 500 of the 1000 tokens have already been received.

If a streaming call breaks off halfway and you perform a simple retry, this leads to problems:

Strategies for Streaming Idempotency

For streaming scenarios, there are two common approaches:

1. Buffer until completion: Store streaming chunks in a temporary in-memory buffer (or Redis). Only when the stream reaches the explicit end marker (such as [DONE] or finish_reason: "stop") do you mark the idempotency_key as complete and write the result to permanent storage. Does the stream fail halfway? Then you discard the buffer and the retry performs a clean new call.

2. Append-only with session tracking: Store each chunk in the database linked to a unique chunk_index and the idempotency_key. Upon resumption, the client retrieves the current max(chunk_index) and sends the LLM prompt again with an instruction to continue from that point. However, this requires specific support from the model and the provider.

Interaction with Retries and Exponential Backoff

Idempotency and retries are two sides of the same coin. Retries without idempotency are dangerous; idempotency without retries is useless during network outages.

When you implement a retry mechanism with exponential backoff and jitter, the idempotency_key must remain constant across all consecutive attempts of the same logical action.

Checklist for Idempotency in LLM Integrations

Use this checklist when designing your LLM integration layer:

  1. Unique identification: Do you generate a unique key at the source (frontend/client) for each user action or task?
  2. Provider support: Do you check whether the specific LLM provider supports an Idempotency-Key header and what the exact conditions are?
  3. Own storage layer: Is a fast key-value store (such as Redis) available with distributed locks (SETNX) to block concurrent requests?
  4. TTL management: Do the stored idempotency records and locks have an appropriate automatic expiration (TTL)?
  5. Domain deduplication: Are the follow-up actions (database writes, emails, webhooks) also protected against duplicate execution using unique identifiers?
  6. Streaming handling: Are aborted streaming responses properly cleaned up or buffered before being marked as 'completed'?

By incorporating idempotency into your API architecture from the start, you not only prevent unexpected costs at LLM providers, but you also build a system that withstands network outages and unexpected background restarts.

Would you like to dive deeper into testing the robustness of your LLM infrastructure or are you looking for advice for your specific architecture? Check out the options at LLMNet Consultancy or join the discussions in our LLMNet Community.