Embeddings APIs in Production: Limits, Costs, and Throughput

In many Retrieval-Augmented Generation (RAG) and search applications, the focus almost immediately shifts to the generative language model that formulates the final answer. Yet, the underlying vector representation of the data serves as the foundation of the entire retrieval system. Anyone scaling a proof-of-concept to a full-fledged production environment quickly discovers that generating vectors via external APIs brings its own operational challenges. This article covers the specific throughput constraints, rate limits, cost management, and error handling involved in deploying embeddings endpoints at scale.

While a standard chat call revolves around sequential tokens and streaming interfaces, embeddings instead require massive parallelization, strict batch processing, and careful consideration of dimension sizes. For those wanting to understand exactly how the query side connects to vector searches, the fundamentals can be explored in building semantic search with an embeddings API. Below, we focus on the architectural and infrastructural requirements to reliably and efficiently transform millions of text elements into vectors.

The Two Operational Profiles: Bulk Indexing versus Real-Time Queries

In a production system, embeddings calls exhibit two completely distinct traffic patterns that cannot be treated the same way: bulk indexing of documents and real-time transformation of end-user search queries.

Bulk indexing occurs when an initial dataset of thousands or millions of documents needs to be converted into vectors, or when periodic sync jobs run. Here, the traffic pattern is asynchronous, demands maximum throughput, and is relatively tolerant of latency lasting minutes or hours. The primary objective is to push as close as possible to the provider's maximum token limits without being throttled by HTTP 429 errors.

Real-time query processing occurs when a user types a search query. Here, round-trip latency is all that matters. The API call typically contains just a single sentence of 10 to 50 tokens, but the vector must be available within 50 to 150 milliseconds to keep the application's overall latency acceptable. If an administrator uses the same gateway configuration or queue for both bulk processing and real-time queries, a background job will inevitably cause unacceptable delays for active users.

Maximizing Throughput with Micro-Batching and Chunk Optimization

Most embedding endpoint providers allow an array of input texts to be sent in a single HTTP request. Sending individual text fragments (chunks) per HTTP call introduces massive overhead due to TLS handshakes, TCP roundtrips, and header parsing. Conversely, bundling too much text into a single request leads to timeouts or exceeding the maximum payload size.

The optimal batch size is a dynamic balance between three parameters:

• Number of items per batch: Providers often enforce a hard limit, ranging from 96 to 2048 text lines per request.
• Total number of tokens per batch: A combined ceiling per request often applies (for example, a maximum of 8,192 or 32,768 tokens per call).
• Network payload and serialization: Large JSON arrays require significant CPU capacity for string escaping and JSON parsing on both the client and server side.

In production, we therefore implement a worker pool with micro-batching. An ingestion service collects text fragments in a buffer and dispatches a batch as soon as either the token ceiling is reached (for instance, 90% of the maximum to maintain safety margins) or a short timeout (e.g., 50 ms) expires.

Rate limits and concurrency: TPM versus RPM

Embedding APIs apply stricter limits than often expected. Providers generally divide limits into two categories: Requests Per Minute (RPM) and Tokens Per Minute (TPM). With embeddings during bulk processing, you almost always hit the TPM limit long before the RPM threshold comes into view.

When an application launches parallel workers that directly pump data to the API, the server responds upon an overrun with a 429 Too Many Requests status code. Without structured handling, this can lead to a so-called retry storm, where all workers simultaneously attempt to resend and permanently clog the endpoint. How to manage general limits and pricing models holistically within an architecture can be read in the overview on rate limits, tokens, and costs.

For embedding pipelines, running a client-side concurrency limiter or token bucket is essential. The Python example below demonstrates a robust worker featuring dynamic micro-batching, local token counting, and exponential backoff with jitter:

import time
import random
import urllib.request
import json

def batch_embed_texts(chunks, token_limit=8000, max_retries=5):
    """
    Verstuurt batches naar een embeddings-API met foutafhandeling.
    chunks is een lijst van dicts: [{'id': str, 'text': str, 'tokens': int}]
    """
    url = "https://api.provider.example/v1/embeddings"
    api_key = "SECURE_API_KEY"
    
    batches = []
    current_batch = []
    current_tokens = 0
    
    # 1. Dynamische micro-batching op basis van tokens
    for item in chunks:
        if current_tokens + item['tokens'] > token_limit and current_batch:
            batches.append(current_batch)
            current_batch = []
            current_tokens = 0
        current_batch.append(item)
        current_tokens += item['tokens']
    if current_batch:
        batches.append(current_batch)
        
    results = []
    
    # 2. Uitvoeren per batch met retry- en backoff-patroon
    for batch in batches:
        payload = json.dumps({
            "model": "text-embedding-3-small",
            "input": [b['text'] for b in batch]
        }).encode('utf-8')
        
        headers = {
            "Content-Type": "application/json",
            "Authorization": f"Bearer {api_key}"
        }
        
        attempt = 0
        success = False
        
        while attempt < max_retries and not success:
            req = urllib.request.Request(url, data=payload, headers=headers, method="POST")
            try:
                with urllib.request.urlopen(req, timeout=30) as resp:
                    if resp.status == 200:
                        body = json.loads(resp.read().decode('utf-8'))
                        for idx, data_item in enumerate(body.get('data', [])):
                            results.append({
                                "id": batch[idx]['id'],
                                "embedding": data_item['embedding']
                            })
                        success = True
            except urllib.error.HTTPError as e:
                attempt += 1
                if e.code == 429:
                    # Exponentiële backoff met decorrelated jitter
                    sleep_time = (2 ** attempt) + (random.uniform(0.1, 1.0))
                    time.sleep(sleep_time)
                elif e.code >= 500:
                    time.sleep(1.0 * attempt)
                else:
                    # Client-fouten (400, 401, 403) direct escaleren
                    raise RuntimeError(f"API client-fout {e.code}: {e.read().decode('utf-8')}")
            except Exception as ex:
                attempt += 1
                time.sleep(1.5 * attempt)
                
        if not success:
            raise TimeoutError(f"Batch mislukt na {max_retries} pogingen.")
            
    return results

Cost analysis and dimensionality reduction (Matryoshka Embeddings)

The costs of embedding APIs are almost universally billed per million tokens. Although the unit price per token is significantly lower than for generative LLMs, total operational costs can escalate quickly with large datasets, regular re-indexing, or frequent document mutations.

Furthermore, API costs are only one part of the total cost structure. The chosen embedding dimension directly impacts the memory costs (RAM/VRAM) of the vector database and the network latency during vector transfer. Modern embedding models (such as Matryoshka Representation Learning models) offer the ability to truncate the output dimension (for example, from 1536 or 3072 dimensions down to 512 or 256) without a substantial loss in search relevance.

Lowering the dimension via the API parameter does not directly reduce the API invoice, but it doubles or triples the vector database throughput and drastically cuts index hosting costs. The impact of model selection on the overall retrieval pipeline and when an additional reranker becomes necessary is detailed extensively in the guide on embedding, reranker, or hybrid retrieval models.

What does mitigation cost? Latency, complexity, and memory

Every technical solution deployed to address embedding failure modes comes with its own price. It is essential to explicitly weigh these trade-offs:

• Client-side caching of embeddings: By pairing a hash of the input text with the generated vector, you avoid duplicate API costs for identical phrases or common search queries. The trade-off: Increased architectural complexity (such as a Redis instance) and memory consumption. In addition, a cache lookup adds 1 to 3 milliseconds to each query.
• Local fallback models: When the external API experiences an outage, the system can fall back to a locally hosted embedding model (such as a compact ONNX or HuggingFace model). The cost: Local compute resources (CPU/GPU load on application servers) and semantic incompatibility. A vector generated by model A can never be directly compared to a vector from model B within the same index space.
• Dynamic batching and queuing: Buffering requests significantly increases efficiency towards the external provider. The cost: Increased p99 latency for individual queued items and additional points of failure if worker processes crash before the buffer is flushed.

Failure behavior and data integrity in production

In a continuous production pipeline, partial failures inevitably occur. Consider a batch of 100 documents where a single document contains invisible binary data that triggers a 400 Bad Request, or a network connection that drops halfway through a stream. If the pipeline is not designed to be transaction-safe, this leads to 'silent corruption': documents marked as processed in the relational database, but whose vectors are missing from the vector index.

To ensure data integrity, the ingestion application must use a two-phase synchronization:

1. First, generate the vector via the API and temporarily store the result in a staging table or cache.
2. Only then commit the vector to the vector database with a unique document version hash.
3. After that, update the status in the source system to 'indexed'.

When a document changes, the old vector must be invalidated immediately to prevent stale search results. The full lifecycle of vector storage, updates, and cleanup processes is covered in the guide on maintaining a vector index: storing, updating, and deleting embeddings.

Observability and monitoring of embeddings endpoints

Because embeddings APIs are often called as black boxes, performance degradations frequently go unnoticed until end users complain about sluggish search interfaces. A mature production architecture monitors at least the following four telemetry dimensions:

• P95 and P99 Latency by batch size: Provides insight into server-side queues at the provider and signals when the network connection stutters.
• Token consumption by task type: Distinguishes the token usage of background ingestion from consumption by real-time search queries.
• HTTP status code distribution: An increase in 429 errors points to an inadequate client-side token bucket or unexpected spikes in data ingestion.
• Vector quality metrics (dimension and norm validation): Always verify prior to storage whether the returned array has the expected length (for example, exactly 1536 numbers) and that the vector does not contain NaNor nullvalues.

By treating embeddings endpoints not as simple utilities but as mission-critical network components with their own limits, costs, and failure modes, the retrieval layer of your AI application remains stable, predictable, and scalable under heavy production load.