Integrating reranking APIs into a multi-stage RAG pipeline

In a standard retrieval-augmented generation (RAG) architecture, vector search is often the first line of defense. By converting a query into an embedding vector and retrieving the nearest document fragments through cosine similarity, the system tries to supply relevant context to a language model. For a deeper look at the fundamentals of vectors and semantic distance measures, see the basic concept of building semantic search with an embeddings API to understand how bi-encoders generate vectors. In production environments, however, this single-stage approach proves structurally inadequate for complex queries. Bi-encoders compress an entire text fragment into a single static vector, which loses subtle relationships, negations and exact word combinations.

A multi-stage retrieval architecture solves this by splitting the search process into two phases: a fast, broad initial selection (high recall) followed by precise reordering with a cross-encoder (high precision). This article covers how specialized reranking APIs are integrated into such a chain, how to handle payload sizes and latency budgets, how score thresholds are calibrated, and which failure patterns appear when an external reranker slows down or fails.

The fundamental difference: bi-encoders versus cross-encoders

To understand why a reranking step is necessary, we have to look at the mathematical interaction between query and document. With a bi-encoder, the query $q$ and each candidate document $d_i$ are passed through a neural network independently. This yields two vectors: $\vec{u} = f(q)$ and $\vec{v}_i = g(d_i)$. The relevance score is simply the dot product or the cosine similarity $\cos(\vec{u}, \vec{v}_i)$. Because the document vectors are precomputed and indexed in a vector database, comparing thousands of vectors takes only a few milliseconds. The drawback is that the attention mechanisms (self-attention) in the language model never look at the query and the specific document at the same time.

A cross-encoder works fundamentally differently. Here the query and the document are joined into a single shared input sequence, separated by a separator token: [CLS] query [SEP] document [EOS]. The model computes full cross-attention over all tokens simultaneously. Every word in the query can interact directly with every word in the document. This model produces a continuous relevance score between 0 and 1 directly, with no vectors in between.

Computing cross-attention is orders of magnitude heavier ($O(N^2)$ complexity in token length). Passing a million documents through a cross-encoder in real time is impossible. By using the bi-encoder to filter from a million down to 50 candidates, and unleashing the cross-encoder only on those 50 fragments, you combine the scalability of vector indexes with the precision of full attention models.

The architecture of a multi-stage retrieval chain

A robust production pipeline typically consists of three consecutive steps before the final prompt is sent to the generative language model: candidate retrieval, document preparation and reranking scoring.

When sizing stage 1, it is important to allow enough margin. If the right document fragment is not in the initial selection of 50 documents, the reranker can never surface it (the recall ceiling). Sending too many fragments to the reranker, on the other hand, raises network latency and API costs proportionally. To keep the operational throughput and volume costs of the first stage under control, the overview of embeddings APIs in production: limits, costs and throughput offers practical guidance on sizing batches and rate limits.

Structuring payloads and making the API call

Reranking APIs generally expect a JSON payload with the query and an array of documents or text strings. A common mistake in integrations is passing superfluous metadata to the reranker. The reranker judges textual relevance; fields such as internal database IDs, timestamps, tenant identifiers and JSON attributes pollute the cross-encoder model's context window and drive up token consumption.

Below is an implementation example in TypeScript/Node.js showing how to strip and validate candidate documents and send them to a reranking endpoint with an explicit timeout and error handling:

interface CandidateDocument {
  id: string;
  text: string;
  metadata: Record<string, unknown>;
}

interface RankedResult {
  id: string;
  text: string;
  score: number;
  originalIndex: number;
}

async function rerankCandidates(
  query: string,
  candidates: CandidateDocument[],
  apiKey: string,
  topN: number = 5,
  timeoutMs: number = 400
): Promise<RankedResult[]> {
  if (!candidates.length) return [];

  // Strip metadata en stuur alleen noodzakelijke tekst door
  const documents = candidates.map(c => c.text);
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), timeoutMs);

  try {
    const response = await fetch("https://api.rerank-provider.example/v1/rerank", {
      method: "POST",
      headers: {
        "Authorization": `Bearer ${apiKey}`,
        "Content-Type": "application/json"
      },
      body: JSON.stringify({
        model: "rerank-v3.5",
        query: query,
        documents: documents,
        top_n: topN,
        return_documents: false
      }),
      signal: controller.signal
    });

    if (!response.ok) {
      throw new Error(`Reranker HTTP ${response.status}: ${response.statusText}`);
    }

    const data = await response.json();
    
    // Koppel de gerangschikte indices terug aan de originele objecten
    return data.results.map((r: { index: number; relevance_score: number }) => ({
      id: candidates[r.index].id,
      text: candidates[r.index].text,
      score: r.relevance_score,
      originalIndex: r.index
    }));
  } catch (err: unknown) {
    if (err instanceof Error && err.name === "AbortError") {
      // Faalmodus: Latency-overschrijding
      console.warn(`Reranker timeout na ${timeoutMs}ms. Val terug op Stage 1 ranking.`);
    } else {
      console.error("Reranking API fout:", err);
    }
    // Fallback: retourneer de top N uit Stage 1 met neutrale score
    return candidates.slice(0, topN).map((c, i) => ({
      id: c.id,
      text: c.text,
      score: 0.0,
      originalIndex: i
    }));
  } finally {
    clearTimeout(timer);
  }
}

Note the flag return_documents: false in the payload. By instructing the provider to return only the index and the relevance score, you save substantial network bandwidth on the response path. The original text is already in the calling application's memory, after all.

Score thresholds and dynamic context filtering

One important advantage of rerankers over raw cosine scores is the scaling of the relevance score. Cosine distances from bi-encoders are notorious for their narrow dynamic range: a score of 0.78 can be excellent with one model and pure noise with another. Cross-encoders, by contrast, produce a more reliable sigmoid probability distribution.

This makes it possible to work with dynamic score cutoffs instead of passing a fixed number of fragments ($k$) to the language model. If a query yields 50 candidates of which only one really answers the question, the reranker will return a score of, say, 0.94 for document 1 and scores below 0.20 for documents 2 through 50.

Blindly injecting a fixed $top\_k = 5$ into your prompt pollutes the context with four irrelevant chunks of text. That raises the risk of hallucination and drives up the generative model's token costs needlessly. A dynamic filtering algorithm applies two rules:

• Absolute threshold: Any document with a relevance score below $T_{min}$ (0.35, for instance) is rejected outright, whatever its rank.
• Relative gap analysis: If the difference between score $N$ and score $N+1$ is greater than $\Delta_{drop}$ (0.40, for instance), all documents from $N+1$ onward are cut off.

As a result, the generation model receives just one razor-sharp fragment for specific questions, while broad overview questions automatically get four or five relevant pieces of context.

Failure behavior and circuit breakers during network outages

Every external API call introduces a new point of failure in the architecture. Reranking sits squarely on the critical path of the user experience: as long as the reranker does not respond, the prompt cannot be assembled and the generative model cannot start streaming.

The main failure mode of reranking APIs is not a hard 500 error but tail latency — hanging connections caused by overload at the provider. If an application has a strict SLA of at most 1000 ms total response time, the reranker may consume no more than 250 to 300 ms of that. To keep slow API calls from blocking the whole chain, a deadline budget and a circuit breaker are required. How to configure strict time limits and deadline propagation across successive microservices is covered in the article on timeouts, cancellation and deadline budgets in LLM calls.

When the circuit breaker opens after repeated timeouts, the pipeline must immediately fall back to graceful degradation. In the case of RAG, that means the application uses the top 5 fragments straight from the vector search (stage 1). The quality of the generated answer may dip fractionally for a while, but the user keeps a working interface with acceptable response times.

Throughput, batching and routing through a gateway

In applications with high concurrency, a bottleneck around the reranking provider's rate limits appears quickly. Where embeddings are relatively cheap and can be computed massively in parallel, most reranking services apply stricter limits on documents processed per minute (DPM) and queries per second (QPS).

To keep individual microservices from hitting provider limits directly, all retrieval and reranking traffic can be centralized behind a gateway. An overview of architectures for managing throughput, budgeting and throttling in retrieval processes can be found in the guide to retrieval traffic through the gateway: routing, limiting and budgeting embeddings calls.

Routing reranking calls through a gateway brings three operational advantages:

• Key and tenant isolation: Different departments or customers cannot exhaust each other's rate limits.
• Multi-provider fallback: If provider A reports an outage or becomes structurally slower than 300 ms, the gateway switches transparently to an alternative reranking API (from Cohere to Jina, or to a self-hosted TEI instance).
• Semantic document caching: Identical queries with overlapping candidate sets can be served straight from an in-memory cache without calling the external API again.

Privacy, data minimization and compliance

Forwarding document fragments to an external reranker raises a compliance issue that is often overlooked. In a standard RAG setup, chunks are usually stored locally in your own vector database. The moment you activate stage 2, you are potentially sending hundreds of confidential text fragments per search to a third party's API.

Because rerankers need the full plain text to compute their cross-attention, sensitive personal data (PII), medical records or trade secrets can end up in the reranking vendor's logs. When selecting a SaaS reranker, therefore, explicitly check whether the processor offers a zero data retention (ZDR) agreement and where the inference servers are located geographically. More on selecting processors and safeguarding data integrity can be found in the overview of AI models and privacy: choices for GDPR compliance.

If data sovereignty is a hard requirement, you can opt for a self-hosted cross-encoder using frameworks such as Text Embeddings Inference (TEI) or vLLM on your own infrastructure. This eliminates external data transfer, but shifts operational responsibility for GPU capacity and latency management to your own team.

What the mitigation costs: trade-offs in production

Adding a reranking layer is not a free optimization. Anyone designing a multi-stage pipeline has to weigh the trade-offs between accuracy, delay and infrastructure costs carefully.

In practice, investing in a reranker pays for itself many times over in systems where wrong answers carry significant business risk, such as legal search systems, technical manuals and internal knowledge bases. By cutting the number of context tokens ultimately sent to the generative model sharply — from 15 to 3 high-quality fragments, for example — you also save on the final language model's input costs. In many architectures that offsets a considerable share of the reranker costs.

Conclusion and implementation roadmap

Integrating a reranking API turns a fragile vector search setup into a reliable, multi-stage information flow. The key to a successful implementation lies not only in choosing the most powerful cross-encoder model, but above all in the operational preconditions: tight timeouts, robust circuit breakers, dynamic score filtering and centralized throughput management.

Always start by establishing a baseline evaluation of the first stage. Make sure the bi-encoder's recall is high enough to capture relevant documents within the top 50, then add the reranker with a strict timeout of at most 350 ms, and implement dynamic score thresholds to keep noise out of the generative context.