Home / Reranking / Reranking in Semantic Search: What It Is and Why It Works

Reranking in Semantic Search: What It Is and Why It Works

Reranking is a second search stage that rescores the top results from a first retrieval pass with a stronger model and reorders them by that new score. The first stage, keyword, vector, or hybrid search, is fast and broad. It finds a candidate set that probably contains the right answers. The reranker then reads the query and each candidate together and decides which belong at the top. That split exists because accurate scoring models are too slow to run on a whole collection. Reranking improves the order of what was retrieved, often sharply, but it cannot add anything the first stage missed. It also adds latency and cost for every candidate it scores, so its depth is a design decision rather than a default.

What is reranking in semantic search?

Reranking in semantic search is the step that scores each retrieved candidate against the query again and sorts the list by that score. Weaviate’s reranking documentation describes it as computing a relevance score between the query and each data object, then returning objects from most to least relevant. The term also appears as re-ranking or re-rank. All three name the same operation.

The key word is “re.” A reranker never searches the collection. It receives a list that another method already produced and changes only its order. Weaviate’s search documentation puts it plainly: a reranker reorders initial retrieval results with a more complex model or different criteria.

That makes reranking a refinement stage rather than a retrieval method. The first stage might be keyword search, vector search, or hybrid search. Weaviate’s reranker modules work with all three. The reranker then applies a slower, more careful judgment to a list short enough to afford it.

A reranker can even use a different query from retrieval. Weaviate’s Rerank option accepts its own query string and falls back to the search query when none is given. A broad query can gather candidates, and a sharper one can order them. Why a second pass is worth the trouble comes down to how first-stage retrieval trades accuracy for speed.

Why does semantic search need a reranking stage?

Semantic search needs a reranking stage because the fastest retrieval methods compare precomputed representations, while the most accurate methods must read the query and document together. Those two goals pull in opposite directions. Reranking lets a system use each where it fits.

First-stage vector search embeds documents once, ahead of time, and compares a query vector against them. That is fast enough to search millions of items. Elastic’s semantic reranking documentation calls this the bi-encoder approach: query and document are encoded separately, which is faster and cheaper but less precise. The bi-encoder versus cross-encoder guide covers the difference in depth.

A cross-encoder reads the query and the document as one input and outputs a relevance score. It sees how each query word relates to each document word, so it judges relevance more accurately. Weaviate’s training material notes the catch: cross-encoder scores cannot be computed in advance, because they depend on the query. Running one over a whole collection for every search would be far too slow.

Weaviate’s reranking documentation draws the conclusion. Scoring every query and object pair is usually prohibitively slow, so reranking runs as a second stage on a smaller retrieved set. Because that set is small, the system can afford a more expensive model. In retrieval-augmented generation the same logic answers a common question: reranking in RAG puts the most relevant passages first in a limited context window. The pipeline that results has a fixed shape.

How does a two-stage retrieve-and-rerank pipeline work?

A two-stage pipeline retrieves a candidate set with a fast method, rescores those candidates with a reranker, and returns the top of the new order. The first stage is responsible for recall: getting the right items into the set. The second stage is responsible for precision: putting them first.

The pipeline runs in four steps:

1. Retrieve candidates with keyword, vector, or hybrid search.
2. Send the query and each candidate’s text to the reranker.
3. Score every query and candidate pair with the reranking model.
4. Sort by the new score and keep the top results.

Candidate depth is the main design choice. Weaviate’s training material is specific about its own behavior: the search limit is both the reranker’s input set and the number of results returned. There is no separate, wider pool. To let the reranker choose from more candidates, raise the limit and keep the top results yourself.

Weaviate’s reranker property sets which text field the reranker reads, such as a title or body. Elasticsearch offers the same stage through its text_similarity_reranker retriever or the ES|QL RERANK command. Different products expose the step differently, and the model inside it varies too.

What types of rerankers are there?

Rerankers fall into four broad types: cross-encoders, large language model rerankers, score-based rerankers, and diversity rerankers. Each one optimizes something different. Choosing among them starts with what the final list needs to achieve.

The table separates relevance from other goals. Cross-encoder rerankers are the common default. Weaviate’s search documentation notes that models such as Cohere Rerank and Hugging Face rerankers are typically cross-encoders. Cohere’s documentation says its Rerank models return a relevance score per document and are trained for more than 100 languages.

LLM reranking asks a large language model to order the candidates. Sun and colleagues’ 2023 EMNLP paper found that properly instructed models such as GPT-4 can match or beat supervised rerankers on standard benchmarks. Maximal marginal relevance takes a different aim. Carbonell and Goldstein’s 1998 SIGIR paper defines it as favoring documents that are relevant and dissimilar from those already selected, with a weight that balances the two. The best reranking models differ mainly within the first two types. The gains from relevance rerankers are well measured.

How much can reranking improve results?

Reranking can roughly double ranking quality on some benchmarks, as Nogueira and Cho showed when a BERT reranker lifted MS MARCO MRR@10 from 16.7 for BM25 to 36.5. Their 2019 paper, “Passage Re-ranking with BERT,” reranked BM25 candidates with a fine-tuned BERT Large cross-encoder. The abstract reports a 27 percent relative gain over the previous best leaderboard entry.

The measure explains the size of the gain. MRR@10 rewards placing the first relevant passage near the top of ten results. The information retrieval metrics guide defines it fully. Reranking targets exactly that: order within a short list.

The result needs context. The baseline was untuned BM25, and the reranker was trained on MS MARCO itself. A tuned hybrid first stage would leave less room to gain, and a reranker used outside its training domain may gain less. Nogueira and Cho also truncated each query and passage pair to 512 tokens. A team should treat the 2019 figure as evidence that reranking can matter, not as a forecast for its own data. The gain also comes with a cost.

What does reranking cost in latency and compute?

Reranking costs one model evaluation per candidate per query, so its latency and compute grow with candidate depth and query volume. Weaviate’s training material states the trade directly: better quality for top results, but added latency and additional cost. Retrieval cost barely changes with depth. Reranking cost grows in step with it.

Three settings control the bill. Candidate depth sets how many pairs each query scores. Document length matters because cross-encoders read the full text of each candidate up to their token limit. Model size sets the cost of each pair, from small open cross-encoders to large language models.

Hosting adds a fourth choice. A hosted API such as Cohere Rerank bills per request and needs no infrastructure. A self-hosted model, such as Weaviate’s Transformers reranker module, runs on your own hardware. Weaviate also integrates rerankers from Cohere, Contextual AI, Voyage AI, Jina AI, and NVIDIA, so the trade can be tested without changing the search code.

The practical method is a latency budget. Decide the slowest acceptable response time, measure the reranker’s time per candidate on representative documents, and set depth to fit. Reranking latency and cost covers batching and model choice in detail. Cost is one reason to skip reranking. Failure is another.

When does reranking fail or not help?

Reranking fails when the right item never reaches it, when documents are too long for its input, or when the first stage already orders results well. Each failure has a different cause. None is fixed by a stronger reranker.

Missing candidates are the hard limit. Weaviate’s search documentation states that reranking and boosting only reorder first-stage results and cannot add an object absent from them. If recall at the candidate depth is low, the fix belongs in retrieval: hybrid search, better chunking, or a deeper candidate set.

Long documents are the quiet failure. Elastic’s documentation warns that many cross-encoders truncate input to their token window, which can cut off the most relevant part of a document. Chunking content into passages before indexing keeps the relevant text inside the window.

Some systems gain little. A strong hybrid first stage on a narrow, well-structured collection may already rank well. A RAG pipeline that sends every retrieved chunk to the generator may care less about order. Weaviate suggests reranking when top results matter most, latency can grow, and first-stage retrieval is broad. Legal, medical, and scientific text are its examples of subtle collections that benefit. The only reliable answer is a measured test.

How do you add reranking to a search system?

Adding reranking means choosing a model, setting candidate depth, and measuring the change on judged queries before shipping it. The code change is small in most search systems. The decision needs evidence.

In Weaviate, a reranker is set in the collection configuration and applied per query. The Python client passes rerank=Rerank(prop="content", query="...") to a search, and the search limit sets how many candidates are reranked. Elasticsearch uses its text_similarity_reranker retriever with an inference endpoint. Cohere’s Rerank API takes a query, a list of documents, and top_n, and returns a relevance score and index for each document. Cohere recommends formatting structured records as YAML strings.

Then test it. Freeze the collection and the first stage. Measure recall at the candidate depth to confirm the right items are present. Compare MRR or nDCG with and without the reranker on the same judged queries. Record the added latency at the expected load. Keep the reranker only if the gain is worth the time. The evaluation guide explains how to build the judgments, and how semantic search works shows where reranking sits in the full pipeline.

Reranking is a second search stage that rescores the top results with a stronger model and reorders them. It exists because the most accurate models must read query and document together, which is too slow for a whole collection. The first stage sets recall and the reranker sets precision. A 2019 BERT reranker more than doubled MRR@10 over untuned BM25, but no reranker recovers a missed item, and every candidate costs time. The decision is to measure recall at the candidate depth, then keep a reranker only where it improves judged ranking within the latency budget.