Home / Reranking / Reranking in RAG Pipelines

Reranking in RAG Pipelines

Reranking in RAG is a second retrieval stage that rescores the passages found by search with a more precise model, so that only the most relevant few reach the language model’s prompt. The first stage, vector or hybrid search, is fast because it compares precomputed vectors, but it judges relevance coarsely. A reranker, usually a cross-encoder, reads the question and each candidate passage together and scores how well the passage answers it. The RAG system then keeps the top results of the new order. This matters because a prompt holds only a handful of passages, and the answer can only be as good as those passages. Reranking adds latency and cost per query and cannot recover passages the first stage missed. Weaviate runs reranking inside vector, keyword and hybrid queries through reranker integrations.

What does reranking do in a RAG pipeline?

Reranking in a RAG pipeline reorders the candidate passages from the first search so that the passages most likely to answer the question move to the top before the prompt is built. It sits between retrieval and generation and changes order, not membership.

A typical RAG system answers a question in three steps: retrieve passages, build a prompt from the best of them, and ask a language model to answer from that context. Without a reranker, the prompt takes the first search’s top results as they come. With a reranker, the system retrieves a wider candidate list, rescores every candidate against the question, and keeps the best few of the new order. What is reranking covers the general idea outside RAG.

Weaviate’s documentation describes the same split. Scoring every query and object pair with a reranker would be prohibitively slow, so reranking runs as a second stage on a smaller set. Because that set is small, a more expensive model can be used. The pattern is often called two-stage or multi-stage retrieval.

The split exists because the two stages use different kinds of model, and the difference explains both the gain and the cost.

Why does reranking improve RAG answers?

Reranking improves RAG answers because the prompt holds only a few passages, and a cross-encoder judges question-passage relevance more precisely than the vector similarity used to retrieve them. Better passages in the prompt give the model better evidence.

The first stage usually relies on a bi-encoder: the question and each passage are embedded separately, and relevance is the cosine of two vectors. That compression is what makes search fast, since passage vectors are computed once at indexing time. It also loses detail. A passage that mentions the right topic but answers a different question can sit close to the query vector, while the passage with the exact answer sits a few places lower.

A cross-encoder avoids that compression. It feeds the question and the passage into one transformer, so every word of the question can attend to every word of the passage. It can tell that a passage about mortgage interest answers “can I deduct interest on a home loan” while one about credit card interest does not. Cross-encoder rerankers explains the architecture in depth.

Position in the prompt also matters. Liu and colleagues’ 2023 paper “Lost in the Middle” found that language models used information at the start or end of a long context more reliably than information in the middle. Putting the strongest passages first is a side benefit of reranking. The next question is how the stages fit together in practice.

How does a reranker work inside a RAG pipeline?

A reranker works inside a RAG pipeline by taking the first stage’s candidate list, scoring each candidate against the original question, sorting by that score, and passing the top few to the prompt builder. Each step has a setting that changes the result.

1. Retrieve candidates: run vector or hybrid search with a limit larger than the number of passages the prompt will hold.
2. Choose the text to score: pick the passage field the reranker reads, such as the chunk body rather than its title.
3. Score each pair: send the question and each candidate to the reranker, which returns one relevance score per pair.
4. Sort and cut: order by the new score and keep as many passages as the prompt allows.
5. Build the prompt: place the kept passages, best first, with the question and instructions.

The steps hide two choices that matter most. The first is the candidate count: too few and the right passage is never seen; too many and latency grows with every extra pair. The second is the query the reranker scores against. Weaviate’s reranker accepts its own query text, which defaults to the search query, so a system can retrieve with a rewritten query and rerank against the user’s original question.

Rerankers also have an input limit. Most cross-encoders read a few hundred tokens of question and passage together and truncate the rest, so a long chunk is judged on its opening. Chunk size and reranking should be tuned together, as chunking for semantic search explains. The candidate count deserves a closer look.

How many candidates should a RAG reranker see?

A RAG reranker should see enough candidates that the right passage is usually among them, typically a few times the number of passages the prompt holds, and no more than the latency budget allows. The right number comes from measuring first-stage recall, not from a fixed rule.

The reasoning runs in both directions. If the prompt holds five passages and the reranker sees only five, it can reorder them but cannot change which five reach the model. Widening the list to 20 or 50 gives it room to promote a passage the first stage ranked lower. Each extra candidate costs one more model pass, and the gain shrinks once the list already contains the relevant passages most of the time.

The way to choose is to measure recall at depth for the first stage on real questions. Take a set of questions with known relevant passages, and check how often a relevant passage appears in the first-stage top 10, 20, 50 and 100. Pick the smallest depth where that rate stops climbing. Recall at k covers the measurement.

In Weaviate, the depth is the query’s limit. Weaviate’s academy course on reranking states that there is no separate, wider candidate pool: the limit is both the reranker’s input set and the number of results returned. To rerank 50 and keep 5, set the limit to 50 and take the first 5 of the response. That detail shapes the code.

How do you add a reranker to a Weaviate RAG query?

You add a reranker to a Weaviate RAG query by configuring a reranker integration on the collection and passing a Rerank object, naming the property to score, in the query. The reranker then reorders the results before they are returned.

Weaviate’s Python client v4 lists reranker integrations for Cohere, Contextual AI, Jina AI, NVIDIA, Voyage AI and locally hosted Transformers models. The example configures a Cohere reranker, which needs a Cohere API key passed when connecting. It retrieves 50 candidates with hybrid search, reranks them against the question, and keeps the top 5 for the prompt.

import os
import weaviate
from weaviate.classes.config import Configure, Property, DataType
from weaviate.classes.query import Rerank

client = weaviate.connect_to_local(headers={"X-Cohere-Api-Key": os.environ["COHERE_APIKEY"]})

chunks = client.collections.create(
    name="DocChunk",
    properties=[Property(name="body", data_type=DataType.TEXT)],
    vector_config=Configure.Vectors.text2vec_cohere(),
    reranker_config=Configure.Reranker.cohere(),
)

question = "Can I deduct interest on a home equity loan?"
response = chunks.query.hybrid(
    query=question,
    limit=50,                                   # reranker input and output
    rerank=Rerank(prop="body", query=question),
)
context = [o.properties["body"] for o in response.objects[:5]]
client.close()

The five passages in context then go to the language model. Weaviate’s generate methods also accept a rerank argument with a grouped task, which runs retrieval, reranking and generation in one call. Because the limit sets both the reranker’s input and the returned set, that single call passes every reranked object to the model. When the prompt should hold fewer passages than the reranker sees, the two-step version is the one to use.

Configuration applies per collection, and a reranker cannot rescue a poor first stage. Best reranking models compares the hosted and open models available, and hybrid search for RAG covers the first stage. Other kinds of reranker exist too.

Which types of reranker are used in RAG?

The types of reranker used in RAG are cross-encoders, LLM-based rerankers, and diversity rerankers such as maximal marginal relevance. They differ in what they optimise and what they cost.

• Cross-encoders score each question-passage pair with a fine-tuned transformer; they are the default choice for relevance.
• LLM-based rerankers prompt a general language model to score or order passages; they handle complex questions but cost far more per query, as covered in LLM reranking.
• Diversity rerankers penalise passages that repeat each other, so the prompt covers more ground; maximal marginal relevance is the standard method.

The list runs from cheapest to most expensive per candidate for the first two, while diversity reranking solves a different problem. A RAG prompt filled with five near-copies of one passage wastes four slots. That happens often with chunked documents that repeat boilerplate. Some systems therefore apply a cross-encoder for relevance and then a diversity step for coverage.

Hosted rerankers, such as those from Cohere, Jina AI and Voyage AI, are cross-encoder style models served over an API. They remove the need to run a model, at the price of a network call and per-query fees. Every type adds time to each answer, and that cost decides how far reranking can go.

What does reranking cost in a RAG system?

Reranking costs one model pass per candidate per query, which adds latency that grows with the candidate count, passage length and model size, plus API fees when the reranker is hosted. The first stage stays cheap; the reranker becomes the slow step.

A vector search over millions of passages can finish in milliseconds because it compares precomputed vectors. A cross-encoder cannot precompute anything, since its input depends on the question. Fifty candidates mean fifty forward passes through a transformer, run in batches. On a GPU that is usually acceptable for interactive use. On a CPU, a large reranker over 50 long passages can take longer than the language model’s first token.

Three levers control the cost. Fewer candidates cut the work directly. Smaller reranker models trade some precision for speed. Shorter passages reduce tokens per pass. Caching helps for repeated questions. Reranking latency and cost covers budgeting in detail, and the cost of semantic search places reranking in the full bill.

The cost is only worth paying when reranking actually improves the passages that reach the prompt, which is not guaranteed.

When does reranking fail to help RAG?

Reranking fails to help RAG when the first stage misses the relevant passage, when the reranker was trained on a different domain, or when passages are truncated before the part that answers the question. In those cases it adds latency without better context.

Recall is the first limit. A reranker can only reorder what it receives, so a question whose answer sits outside the candidate list gains nothing. Hybrid search, query rewriting and better chunking improve first-stage recall; reranking does not. Domain is the second. Many open cross-encoders are trained on web search questions, and on specialised text such as legal, scientific or medical writing they can rank worse than the first stage. Weaviate’s documentation names those fields as places where nuanced reranking may be needed, which is also where a general reranker is most likely to be wrong.

The safeguard is measurement. Build a set of real questions with known relevant passages, and compare the passages that reach the prompt with and without the reranker, using a metric such as recall in the top 5 or nDCG. Keep the reranker only if it wins on your data. A reranker fine-tuned on in-domain question-passage pairs usually closes the domain gap.

Truncation is the quiet failure. If chunks run longer than the reranker’s input window, the deciding sentence may never be read. Shorter chunks or a reranker with a longer window fix it.

Reranking in RAG adds a precise second look at the passages search returns. The first stage retrieves a wide candidate list quickly, a cross-encoder scores each candidate against the question, and only the best few reach the language model’s prompt. It works because a prompt holds few passages and a cross-encoder reads question and passage together. It costs one model pass per candidate and cannot recover what the first stage missed. In Weaviate, set the limit to the number of candidates to rerank, attach a Rerank object, and keep the top few for the prompt. Measure the passages that reach the model with and without reranking before keeping it.