# Idempotency in LLM API Calls

Share:[𝕏](https://twitter.com/intent/tweet?url=https%3A//api.llmnet.nl/idempotentie-bij-llm-calls&text=Idempotentie%20bij%20LLM-API-calls)[LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A//api.llmnet.nl/idempotentie-bij-llm-calls)[Reddit](https://www.reddit.com/submit?url=https%3A//api.llmnet.nl/idempotentie-bij-llm-calls&title=Idempotentie%20bij%20LLM-API-calls)[Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A//api.llmnet.nl/idempotentie-bij-llm-calls)[Copy link](#)

 
# Idempotency in LLM API Calls

 
 By Ivo Donker — compiled with AI assistance (Claude & Gemini)

 In distributed systems, network error handling is a constant challenge. When your application communicates with external Large Language Model (LLM) APIs, slow response times and high compute costs amplify this problem. Blindly retrying a failed HTTP request can lead to duplicate billing, inconsistent system states, and unwanted actions in external systems.

 Idempotency is the design principle that guarantees that executing the same operation multiple times yields exactly the same result as a single execution. This article covers why LLM calls are inherently non-idempotent, how to build a robust idempotency pattern, and how to handle side effects and partially failed streaming responses.

 
## Why Duplicate Processing Is a Real Problem

 Duplicate processing in API interactions rarely occurs because a developer accidentally calls the same function twice. It is almost always the result of underlying infrastructure and human behavior:

 
 
- Timeouts on successful calls: The LLM provider processes the prompt successfully, but the network between the provider and your server drops before the HTTP 200 OK reaches the client. Your application sees an HTTP timeout and doesn't know whether the request was processed. For more details on correctly setting network limits, see our article on [timeouts and cancellation](/en/timeouts-en-cancellation).
 
- Automatic retries after network errors: Client libraries or API gateways often automatically retry the call on a 503 Service Unavailable or a TCP reset. If the first request was nevertheless accepted in the background, the provider processes the prompt twice. You can find more about this in the guide on [retries and backoff](/en/retries-en-backoff).
 
- Restart of background workers and at-least-once queues: Message queues such as RabbitMQ, AWS SQS, or Apache Kafka typically guarantee 'at-least-once' delivery. If a worker crashes just before acknowledging (ACK) the processed message, another worker picks up the same work again. When setting up [webhooks and asynchronous tasks](/en/webhooks-async-taken), this is a fundamental point of attention.
 
- User behavior: An end user double-clicks the "Generate report" button because the interface doesn't provide immediate visual feedback.
 

 
## Why an LLM Call Is Inherently NOT Idempotent

 Unlike an HTTP GET request or a SQL UPDATE that sets a fixed value, a standard HTTP POST to a /v1/chat/completions endpoint is inherently non-idempotent in three respects:

 
### 1. Direct Financial Damage

 Every processed prompt costs tokens. When your system sends a 4,000-token prompt three times due to a network outage, you pay three times for the same input. At scale, this leads to a significant leak in your API budget.

 
### 2. Non-Deterministic Output

 Unless the temperature parameter is strictly set to 0.0 (and even then, not all providers guarantee determinism due to parallel GPU computations), each execution of the same prompt produces different text. If two parallel workers execute the same task, they generate two divergent responses. Which of the two do you store in the database? This causes race conditions and data inconsistency.

 
### 3. Side Effects (Function Calling and Agentic Workflows)

 When an LLM is deployed as an agent with 'function calling' capabilities, a duplicate call can have far-reaching consequences. A model instructed to "send the customer a confirmation email and create a ticket" will create two tickets and send two emails upon duplicate processing.

 
 Key rule: Treat every LLM API call as an operation with side effects. Never rely on the provider to automatically deduplicate the call for you, unless the provider explicitly supports an idempotency header.

 

 
## The Idempotency Key Pattern

 To make a non-idempotent LLM call idempotent, we introduce a unique Idempotency Key. This pattern works on the principle that the requesting party sends a unique identifier with the request. The receiving or intermediate layer checks this identifier before the LLM is called.

 Some LLM providers support a custom HTTP header for idempotency (such as Idempotency-Key: <key>). The exact header name and support vary by provider. If the provider does not support this, you must implement this layer in your own application architecture or API gateway.

 
### How Do You Choose or Derive the Key?

 The key must represent the unique intent of the operation:

 
 
- Explicit UUID: Generate a UUIDv4 at the source (for example, in the frontend at the user action or when creating the task in the queue). This is the most robust method.
 
- Deterministic Hash: Derive the key by creating a cryptographic hash (SHA-256) of the stable input parameters: hash(user_id + task_type + input_payload). Note that you should not include dynamic parameters such as `timestamp` in the hash, otherwise the deduplication effect is lost.
 

 
### Retention and Storage

 Store the status of idempotency keys in a fast, central key-value store such as Redis. The time-to-live (TTL) of the key depends on the use case, but 24 to 48 hours is sufficient for most LLM applications to absorb network retries and queue replays.

 
## Implementation: Deduplication in Your Own Application Layer

 The pseudocode below illustrates how to build an idempotency structure around an LLM API call using Redis. This prevents two concurrent requests from executing the same LLM call and ensures that a later retry receives the already generated response.

import redis
import time
import json

db = redis.Redis(host='localhost', port=6379, db=0)

def execute_idempotent_llm_call(idempotency_key, prompt_data):
 lock_key = f"lock:{idempotency_key}"
 response_key = f"response:{idempotency_key}"
 
 # 1. Check whether the result already exists
 cached_response = db.get(response_key)
 if cached_response:
 return json.loads(cached_response), "CACHE_HIT"

 # 2. Try to acquire a distributed lock (atomic SETNX)
 # TTL of 60 seconds prevents a crashed worker from blocking the key forever
 acquired = db.set(lock_key, "PROCESSING", nx=True, ex=60)
 
 if not acquired:
 # Another worker is currently processing this request. Wait and poll.
 return wait_for_concurrent_request(response_key)

 try:
 # 3. Execute the actual LLM API call
 llm_result = call_external_llm_api(prompt_data)
 
 # 4. Store the result with a TTL of 24 hours (86400 sec)
 db.set(response_key, json.dumps(llm_result), ex=86400)
 
 return llm_result, "EXECUTED"

 finally:
 # Remove the lock once the work is done (or has failed)
 db.delete(lock_key)

def wait_for_concurrent_request(response_key, timeout=30):
 start = time.time()
 while time.time() - start 

 Note that this pattern differs from general response caching. While caching is intended to improve performance and reduce costs for identical questions, this pattern is specifically designed for correctness and preventing duplicate processing in error situations. Consult our article on [caching LLM responses](/en/caching-llm-antwoorden) for the differences in implementation.

 
## Deduplicating API Calls vs. Deduplicating Side Effects

 Securing the HTTP call to the LLM provider is only half the solution. In modern software architectures, an LLM doesn't just process text; it generates structured data that drives follow-up actions.

 There is an essential difference between two levels of deduplication:

 
 
- Transport level (API call): Preventing the HTTP request to the LLM provider from being executed twice.
 
- Domain level (Side effects): Preventing the action resulting from the LLM output (for example, updating a database or sending a push notification) from being executed twice.
 

 Suppose the LLM call succeeds and stores the response in the cache, but the worker crashes just before the follow-up information is written to the CRM database. On the next queue retry, the worker reads the response directly from the cache (transport idempotency succeeded), but still executes the CRM update for the first time.

 To achieve domain idempotency, you must perform database operations using upserts (INSERT ... ON CONFLICT DO UPDATE) and call external systems with unique transaction IDs derived from the original idempotency_key.

 
## Handling Partial Failures and Streaming Responses

 Streaming responses (via Server-Sent Events or WebSockets) pose a particular challenge to idempotency. Because the data arrives in chunks, a connection can break halfway after 500 of the 1000 tokens have already been received.

 If a streaming call breaks off halfway and you perform a simple retry, this leads to problems:

 
 
- If you ignore the already received text and start over, you have incurred double costs for the first 500 tokens.
 
- If you have already stored the first part in your database, a retry results in duplicate or corrupted content.
 

 
### Strategies for Streaming Idempotency

 For streaming scenarios, there are two common approaches:

 
 1. Buffer until completion: Store streaming chunks in a temporary in-memory buffer (or Redis). Only when the stream reaches the explicit end marker (such as [DONE] or finish_reason: "stop") do you mark the idempotency_key as complete and write the result to permanent storage. Does the stream fail halfway? Then you discard the buffer and the retry performs a clean new call.

 2. Append-only with session tracking: Store each chunk in the database linked to a unique chunk_index and the idempotency_key. Upon resumption, the client retrieves the current max(chunk_index) and sends the LLM prompt again with an instruction to continue from that point. However, this requires specific support from the model and the provider.

 
## Interaction with Retries and Exponential Backoff

 Idempotency and retries are two sides of the same coin. Retries without idempotency are dangerous; idempotency without retries is useless during network outages.

 When you implement a retry mechanism with exponential backoff and jitter, the idempotency_key must remain constant across all consecutive attempts of the same logical action.

 
 
- Retry pointers: Ensure that Attempt 1, Attempt 2, and Attempt 3 send exactly the same idempotency_key in the header or payload.
 
- Error codes with locks: If a retry occurs while the previous attempt is still in progress in the LLM pipeline, the API should not immediately retry but respect the lock status (for example, by returning an HTTP 409 Conflict or 429 Too Many Requests to the caller).
 

 
## Checklist for Idempotency in LLM Integrations

 Use this checklist when designing your LLM integration layer:

 
 
- Unique identification: Do you generate a unique key at the source (frontend/client) for each user action or task?
 
- Provider support: Do you check whether the specific LLM provider supports an Idempotency-Key header and what the exact conditions are?
 
- Own storage layer: Is a fast key-value store (such as Redis) available with distributed locks (SETNX) to block concurrent requests?
 
- TTL management: Do the stored idempotency records and locks have an appropriate automatic expiration (TTL)?
 
- Domain deduplication: Are the follow-up actions (database writes, emails, webhooks) also protected against duplicate execution using unique identifiers?
 
- Streaming handling: Are aborted streaming responses properly cleaned up or buffered before being marked as 'completed'?
 

 By incorporating idempotency into your API architecture from the start, you not only prevent unexpected costs at LLM providers, but you also build a system that withstands network outages and unexpected background restarts.

 Would you like to dive deeper into testing the robustness of your LLM infrastructure or are you looking for advice for your specific architecture? Check out the options at [LLMNet Consultancy](https://consultancy.llmnet.nl/en/) or join the discussions in our [LLMNet Community](https://community.llmnet.nl/en/).