Retrieval traffic through the gateway: routing and budgeting embeddings

Many production architectures route only chat and generation tasks through a central gateway. Retrieval traffic — such as creating vectors via an embeddings API for document indexing and search queries — is often called directly from application layers instead. This creates a blind spot: vector APIs have their own rate limits, provider outages, and unpredictable cost spikes. Anyone who wants to build reliable RAG systems must treat retrieval traffic as full-fledged API traffic.

In this guide, we examine how to integrate embeddings calls into your central gateway architecture. We look at failure behavior for bulk versus real-time traffic, smart routing strategies across provider pools, per-user budget monitoring, and protecting vector integrity. As a starting point, it helps to understand how you build semantic search with an embeddings API, so you can precisely place the data flow from text to vector.

The anatomical difference: generation versus embeddings traffic

The traffic profile of embeddings endpoints differs fundamentally from generative chat endpoints. Where chat requests are characterized by long-running streams, asymmetric I/O (few input tokens, many output tokens), and high per-token latency, embeddings requests show the opposite behavior. They have no streaming, deliver a strictly deterministic float array, and require fast responses for individual search queries, but cause extreme peak loads during batch indexing.

When a batch of 10.000 document fragments is indexed, the application fires off millions of tokens within seconds. If this traffic runs uncontrolled over the same API keys as a real-time search bar, it inevitably leads to HTTP 429 Too Many Requestserrors for end users. The gateway must be able to make this distinction based on headers, paths, or payload size.

Routing embeddings: the pitfall of heterogeneous vectors

With generative LLMs, failover is relatively simple: if Provider A goes down, the gateway forwards the prompt to an equivalent model at Provider B. With embeddings, this is extremely dangerous. A vector generated by model X cannot be directly compared (via cosine similarity or dot product) to a vector from model Y, even if both models happen to have the same dimensionality (such as 1536 or 3072). The semantic spaces are incompatible.

Routing retrieval traffic in the gateway therefore requires strict routing rules. Fallback may only occur between exactly the same underlying model weights. A useful approach is deploying the power of an LLM API aggregator to set up abstraction layers, provided you guarantee that the aggregation layer delivers exactly the same model version via alternative endpoints (for example OpenAI direct versus Azure OpenAI Service).

The gateway implements a Model Family Groupfor this. Instead of a random fallback, the gateway defines specific replica endpoints:

// Voorbeeld gateway route-mapping voor embeddings
{
  "virtual_model": "text-embedding-3-small",
  "strategy": "priority_with_failover",
  "targets": [
    {
      "provider": "openai_direct",
      "endpoint": "https://api.openai.com/v1/embeddings",
      "model": "text-embedding-3-small",
      "priority": 1,
      "timeout_ms": 350
    },
    {
      "provider": "azure_eastus",
      "endpoint": "https://company-eastus.openai.azure.com/openai/deployments/text-embedding-3-small/embeddings?api-version=2024-02-01",
      "model": "text-embedding-3-small",
      "priority": 2,
      "timeout_ms": 500
    }
  ]
}

If you manage the infrastructure yourself, consult the principles on how to set up a self-hosted LLM gateway with failover rules and health checks to detect latency spikes in time.

Throughput monitoring: token buckets for bulk versus real-time

To prevent heavy indexing tasks from clogging users' interactive search experience, the gateway must distinguish between interactive queries (low volume, high priority) and batch ingestion (high volume, low priority). We solve this with separate Token Bucketqueues on the gateway.

When a request comes in, the gateway inspects the header X-Traffic-Type: interactive or X-Traffic-Type: background_ingest. Both streams draw from their own rate-limit quota:

• Interactive Queue (Search Queries): Gets 70% of the guaranteed provider capacity (for example 700.000 TPM). Requests pass through immediately and fail fast if the platform is overloaded, with immediate notification to the client.
• Bulk Queue (Document Parsing & Indexing): Gets a maximum of 30% of capacity during quiet periods, but can scale up dynamically into unused tokens as long as interactive latency stays under a threshold (e.g. 40 ms). If the interactive bucket is drawn on, the gateway throttles the bulk queue dynamically (HTTP 429 with Retry-After headers).

This prevents a developer re-indexing a dataset locally from taking down the production website with rate limit errors.

Budgeting and cost allocation per tenant for vector traffic

Although embeddings are cheaper per token than generative tokens, uncontrolled loops and re-indexing can waste tens of thousands of euros. A document collection of 500.000 PDFs that gets re-vectorized weekly without chunk caching adds up fast.

The gateway acts as bookkeeper and gatekeeper. Every incoming embedding request must carry metadata (such as tenant_id, project_id or environment). The gateway counts the input tokens upfront (using a local tokenizer such as tiktoken) and immediately checks whether the monthly budget of the tenant in question has not been exceeded.

For a detailed financial model on how to pass such usage on to customers, see the article on how to attribute API costs per end user in a SaaS product. By setting thresholds at 80%, 95%, and 100% of the budget, the gateway can automatically pause non-critical indexing tasks while interactive search queries for existing data remain operational.

Security, key isolation, and data integrity

Indexing company documents carries significant privacy and security risks. Sensitive data (PII, financial reports, internal communications) flows through the gateway to the embedding provider in plain text. The gateway must therefore strictly enforce key management and zero data retention.

Developers and microservices must never have direct access to provider keys (such as OpenAI or Cohere API keys). They communicate exclusively with the gateway via short-lived internal API tokens with strict scopes (for example embeddings:write or embeddings:read-only). Consult the guidelines on how to manage LLM API keys securely to prevent leaks via environment variables or build pipelines.

In addition, the gateway must apply payload hashing. By storing a SHA-256 hash of the input fragment linked to the generated vector, the gateway can maintain a deterministic Exact Embedding Cache . If the same paragraph is submitted for indexing again, the gateway returns the cached vector immediately without making an external API call. This saves up to 40% on bulk indexing costs.

Vector drift, index maintenance, and quality assurance

A frequently overlooked failure mode is vector drift: a vendor tweaks an embedding model (or silently updates tokenizer artifacts), causing newly generated vectors to slowly drift away from vectors stored in the database six months earlier. The result is that semantic searches suddenly return less relevant documents, without any error appearing.

The gateway can detect this by periodically running a Canary Benchmark Suite . This is a fixed set of 50 test sentences whose resulting vectors are compared to a golden reference vector via cosine distance. If the score deviates by more than a minimal tolerance threshold (for example 1e-5), the gateway raises an alarm and blocks automatic indexing requests.

For structural management of your vector store, read the guide on how to maintain a vector index and update embeddings. After all, quality in RAG pipelines isn't just about retrieval speed, but above all about content reliability; see also the importance of fact-checking and verifying AI answers when retrieval errors lead to hallucinations in the final generation step.

Architecture pattern: the gateway implementation in code

Below is a robust pseudocode pattern for gateway middleware that handles embeddings traffic. The script validates the budget, checks the local hash cache, selects the primary or secondary replica endpoint, and handles timeouts gracefully without crashing the client.

// TypeScript / Node.js Gateway Middleware Voorbeeld
import { Request, Response, NextFunction } from 'express';
import { createHash } from 'crypto';

interface EmbeddingTarget {
  name: string;
  url: string;
  apiKey: string;
  timeoutMs: number;
}

export async function handleEmbeddingRoute(req: Request, res: Response) {
  const { input, tenant_id, traffic_type } = req.body;

  if (!input || !tenant_id) {
    return res.status(400).json({ error: 'Missing input text or tenant_id' });
  }

  // 1. Controleer tenant budget
  const hasBudget = await checkTenantBudget(tenant_id, input);
  if (!hasBudget) {
    return res.status(429).json({ 
      error: 'Budget exceeded for tenant',
      tenant_id,
      retry_after_billing_cycle: true 
    });
  }

  // 2. Hash-gebaseerde cache check (SHA-256)
  const inputHash = createHash('sha256').update(input).digest('hex');
  const cachedVector = await redisClient.get(`emb:${inputHash}`);
  if (cachedVector) {
    return res.json({ 
      data: [{ embedding: JSON.parse(cachedVector) }], 
      source: 'gateway_cache' 
    });
  }

  // 3. Fallback endpoints definiëren (identieke semantische ruimte)
  const targets: EmbeddingTarget[] = [
    {
      name: 'openai-primary',
      url: 'https://api.openai.com/v1/embeddings',
      apiKey: process.env.OPENAI_API_KEY!,
      timeoutMs: 800
    },
    {
      name: 'azure-secondary',
      url: 'https://gateway-eu.openai.azure.com/openai/deployments/text-embedding-3-small/embeddings?api-version=2024-02-01',
      apiKey: process.env.AZURE_API_KEY!,
      timeoutMs: 1200
    }
  ];

  // 4. Uitvoeren met failover en strikte timeout
  for (const target of targets) {
    try {
      const controller = new AbortController();
      const timeout = setTimeout(() => controller.abort(), target.timeoutMs);

      const response = await fetch(target.url, {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'Authorization': `Bearer ${target.apiKey}`
        },
        body: JSON.stringify({
          model: 'text-embedding-3-small',
          input: input
        }),
        signal: controller.signal
      });

      clearTimeout(timeout);

      if (response.ok) {
        const payload = await response.json();
        const vector = payload.data[0].embedding;

        // Asynchroon cachen voor toekomstige queries
        await redisClient.setex(`emb:${inputHash}`, 86400 * 7, JSON.stringify(vector));
        await trackTenantUsage(tenant_id, payload.usage.total_tokens);

        return res.json({
          data: payload.data,
          source: target.name,
          usage: payload.usage
        });
      }
    } catch (err) {
      console.warn(`Target ${target.name} gefaald of timed-out, probeer volgende target...`);
    }
  }

  return res.status(502).json({ 
    error: 'All embedding upstream replicas failed or timed out.' 
  });
}

What does a robust embeddings gateway cost?

Adding a gateway layer between your application and the vector endpoints comes with trade-offs. Here's an overview:

• Network latency: An internal gateway adds an average of 2 to 8 milliseconds to each round-trip (tokenization, header inspection, cache lookup). For real-time search bars, this is negligible compared to the external network latency of 80-250 ms to the API provider.
• Operational complexity: You need to maintain a Redis or memcached instance for the hash cache and persistent data management for rate limits and tenant budgets (e.g. Postgres or Redis Token Buckets).
• Sharp cost savings: Through exact caching of paragraphs and deduplicating batch requests, you save significantly on API bills in real-world setups, while the continuity of the interactive search function remains guaranteed during large-scale re-indexing.

Conclusion

By explicitly bringing retrieval and embeddings traffic under the central LLM gateway, you prevent heavy indexing processes from disrupting your interactive search functions. With separate token buckets, strict fallback rules between identical model versions, tenant-based budget caps, and automatic integrity checks, you transform a fragile RAG prototype into a reliable and cost-controlled production system.