Batch Processing via the LLM API: Smartly Processing Large Volumes of Data

Processing a single prompt via a Language Model API is simple. But as soon as you need to analyze thousands of documents, classify customer reviews, or migrate entire databases, the dynamics change fundamentally. Real-time API calls quickly fall short. You run into hard limits, applications crash due to memory issues, and an unexpected network error can ruin your entire workflow.

To process large volumes of data stably, a well-thought-out architecture is required. This article discusses how to set up batch processing, how to handle rate limits, why idempotency is crucial, and how to keep costs manageable. For insight into basic costs and limits, you can also refer to rate limits and costs.

Real-Time versus Batch Processing

In traditional web applications, API requests are handled synchronously: the user waits for the response. For large-scale background tasks, this pattern is unsuitable. The table below illustrates the key differences:

Feature	Real-Time (Synchronous)	Batch Processing (Asynchronous)
Latency	Immediate (milliseconds to seconds)	Variable (hours to days depending on volume)
Error Handling	Immediate feedback to client	Automatic retries and isolation of errors
Resource Usage	High peak load, sensitive to spikes	Evenly distributed over time
Costs	Standard rates	Often discounted with providers' native batch endpoints

When you opt for asynchronous processing, the design principle shifts from "immediate response" to "reliable status management". This aligns closely with setting up robust systems, similar to the principles you read about in RAG for beginners for indexing large document collections.

The Processing Pattern: Status Management per Item

A robust batch engine relies on a database that accurately tracks the status of each individual request. Without such management, you are completely blind in the event of a crash and cannot distinguish between successful items and failed tasks.

Each item in your queue goes through a fixed lifecycle:

Pending: The item has been created but not yet picked up.
Processing: A worker has locked the item and sent it to the API.
Completed: The LLM response has been successfully received and stored.
Failed: The request has definitively failed after all attempts.

Below you can see a schematic representation of a database schema in SQL that supports these statuses:

CREATE TABLE batch_jobs (
    id SERIAL PRIMARY KEY,
    payload JSONB NOT NULL,
    status VARCHAR(20) DEFAULT 'pending',
    attempts INT DEFAULT 0,
    response TEXT,
    error_message TEXT,
    updated_at TIMESTAMP DEFAULT NOW()
);

Limiting Parallelism and Avoiding Rate Limits

Blindly firing off thousands of parallel HTTP requests immediately leads to 429 Too Many Requests errors. LLM providers enforce strict limits based on *Requests Per Minute* (RPM) and *Tokens Per Minute* (TPM).

To stay within these limits, you use a worker pool with a rate limiter (for example, a Token Bucket algorithm). This ensures that the number of concurrent requests is dynamically throttled as you approach the limit. Should you still hit a limit, a proper retry strategy is essential. Read more about how to handle this via retries and backoff.

Concurrency Control in Code

Instead of starting infinite async threads, you limit the number of active workers. This prevents your memory from filling up and keeps your network traffic predictable.

import asyncio

async def worker(queue, semaphore, api_client):
    while True:
        item_id, payload = await queue.get()
        async with semaphore:
            try:
                # Process LLM call
                result = await api_client.generate(payload)
                await save_success(item_id, result)
            except Exception as e:
                await handle_failure(item_id, str(e))
            finally:
                queue.task_done()

Idempotency: Safely Restarting Without Duplicate Work

What happens if your batch script crashes at 80% of the dataset and you restart the application? Without idempotency, you risk resending all previous items. That costs double the tokens and therefore double the money.

Idempotency means that an action can be executed multiple times with exactly the same result as a single execution. You achieve this by:

Generating a unique hash (for example, a SHA-256 of the input text) as a primary key or idempotency key.
Checking before each API call whether the database already has a completed status for that specific hash.
Directly linking incoming results to that unique identifier.

Saving Intermediate Results Periodically

Saving results only at the very end of a batch of 10,000 items is a recipe for frustration. If the process crashes after 9,000 items due to an unexpected memory leak, you lose everything.

Therefore, write results in small batches (chunks of, for example, 50 or 100 items) directly to your database or an object store. This minimizes the risk of data loss and gives you immediate insight into the progress via your dashboard.

Handling Partial Failure

With large-scale processing, there is a one hundred percent chance that some of the items will fail. Perhaps a specific prompt contains illegal characters, the model refuses to answer due to safety filters, or a temporary network outage occurs.

A professional batch pipeline never stops at an error in a single item. Instead:

Catch the specific exception within the worker loop.
Log the error message and the item ID in the database.
Increment the counter for the number of attempts (attempts).
Let the batch continue with the remaining items.
Export all failed items to a separate "dead-letter queue" afterwards for manual analysis.

Estimating Costs Upfront with a Sample

Before you unleash a massive batch on an expensive model, you want to know what the costs will be. Token counts are difficult to predict exactly beforehand because the length of both the input and output varies.

The solution is a representative sample:

Randomly select 1% to 5% of your total dataset (for example, 100 items).
Run this small batch and measure exactly the number of input and output tokens used.
Calculate the average number of tokens per item.
Multiply this average by the total number of items and the provider's rates.

This way, you prevent financial surprises afterwards and, based on the outcome, you can optionally choose a cheaper model or shorter prompts.

Conclusion

Batch processing of LLM calls requires a shift from ad-hoc scripts to a robust background architecture. By working with strict status management, limiting parallelism, applying idempotency, and saving intermediate results, you build systems that autonomously process thousands of requests safely. Combine this with a good upfront cost estimate, and you keep both the technology and your budget fully under control.