Home / Reranking / Reranking Latency and Cost: How Many Results to Rerank

Reranking Latency and Cost: How Many Results to Rerank

Reranking latency and cost both grow roughly in proportion to the number of candidates you rerank, so the key decision is the smallest candidate count that still captures nearly all of the quality gain. A cross-encoder reranker scores every candidate against the query, so 100 candidates cost about twice as much as 50, in tokens, compute and time. The gain, however, flattens: once the first-stage search has placed most relevant documents in the top N, reranking more candidates adds cost without improving the top results. The practical method is to measure first-stage recall and reranked quality at several values of N on real queries, pick the knee of the curve, and check it against the latency budget. In Weaviate, the query limit is both the reranker’s input and its output, so you set the limit to N and keep the top results yourself. Caching, document truncation and reranking only the queries that need it reduce cost further.

What does reranking latency and cost depend on?

Reranking latency and cost are determined by the number of candidates, the length of each candidate and the query, the size of the reranking model, and where the model runs. The candidate count is the lever you control on every query.

A cross-encoder reranker reads the query and one candidate together and outputs a relevance score. Weaviate’s reranking documentation explains why this is a second stage: computing the score for every query and object pair would be prohibitively slow, so reranking runs on a smaller set retrieved first. The mechanics are covered in what is reranking and cross-encoder rerankers.

Each candidate adds work. Hosted rerankers typically charge per token or per request, and self-hosted ones consume GPU or CPU time per pair. Batching candidates together makes better use of hardware, but the total work still grows with the number and length of candidates. Larger models score better and cost more per pair, a trade-off compared in best reranking models.

Weaviate’s search overview summarises the trade-off in one line: reranking costs one model call per query and is worth it when precision at the top of the list matters more than latency.

How much does reranking more candidates cost?

Reranking cost scales linearly with the candidate count: tokens per query equal the number of candidates times the combined length of the query and each candidate. Monthly cost then scales with query volume.

Take an illustrative case of 20-token queries and 300-token candidates. Reranking 20 candidates sends 20 × 320 = 6,400 tokens per query; 50 candidates send 16,000; 100 send 32,000. At 100,000 queries a month, that is 640 million, 1.6 billion and 3.2 billion tokens respectively. Doubling N doubles the bill.

Candidate length matters as much as count. Reranking full documents instead of short passages multiplies tokens, and most cross-encoders truncate long inputs anyway. Reranking on a focused property, such as a passage or summary, keeps both cost and truncation under control. Weaviate’s Rerank setting takes a prop argument for exactly this choice.

How does reranking affect latency?

Reranking adds a second step after retrieval, and its duration grows with the number and length of candidates and the size of the model, often making it the slowest part of a search request. Network calls add more when the reranker is hosted.

First-stage retrieval with an approximate nearest neighbour index is fast because document vectors are precomputed. A cross-encoder cannot precompute anything: every candidate must pass through the model together with the query at request time. That is why the Weaviate academy course on search types lists added latency as the main trade-off of reranking.

Hosted rerankers add a network round trip, and self-hosted ones depend on available hardware. Measure the reranking step separately from retrieval; Weaviate offers query profiling for per-shard timing of searches, which helps separate the database part from the model call. A full request budget, stage by stage, is laid out in search latency.

How many candidates should you rerank?

You should rerank the smallest number of candidates after which reranked quality stops improving, found by testing several values of N on real queries and comparing first-stage recall with final quality. The first stage sets the ceiling.

A reranker cannot find documents that were not retrieved. Weaviate’s search overview states this directly: boost and rerank are second-stage operations that reorder what the first search returned, and neither can pull in an object that is not in the initial result set. If a relevant document sits at position 150, reranking the top 50 will never surface it.

So measure two curves. First-stage recall at N, described in recall at k, shows how many relevant documents are within reach. Reranked quality at the depth users see, such as nDCG at 10, shows what the reranker actually delivers. The best N is where the second curve flattens. Improving the first stage, for example with hybrid search, often lets a smaller N achieve the same quality.

How do you control the candidate count in Weaviate?

You control the candidate count in Weaviate through the query’s limit, which sets both how many results the reranker receives and how many come back, so you retrieve N and keep the top k yourself. There is no separate candidate pool.

Weaviate’s academy course spells this out: the initial search retrieves exactly as many results as the limit asks for, the reranker scores each of them, and the same results come back reordered. If you want the reranker to choose from more candidates, raise the limit and take the top results from the response.

from weaviate.classes.query import Rerank

def rerank_search(collection, query, n=50, k=10):
    res = collection.query.hybrid(
        query=query,
        limit=n,                                   # reranker input = n candidates
        rerank=Rerank(prop="passage", query=query),
    )
    return res.objects[:k]                         # keep the top k after reranking

def rerank_tokens(n, query_tokens=20, doc_tokens=300):
    return n * (query_tokens + doc_tokens)         # 50 -> 16,000

The query pattern follows Weaviate’s reranking documentation and requires a reranker integration, such as Cohere, Voyage AI or a locally hosted transformers model, configured on the collection; it was not run against a reranker locally. The token function was run as an arithmetic check and reproduces the values in the figure. Weaviate added its first reranker modules, for Cohere and sentence-transformers cross-encoders, in version 1.20.

How can you reduce reranking cost and latency?

You can reduce reranking cost and latency by reranking fewer and shorter candidates, caching results for repeated queries, using a smaller model, and skipping reranking for queries that do not need it. Each saves a different part of the bill.

Shorter candidates help most cheaply: rerank a passage or summary property rather than whole documents. Caching helps for popular queries, since the same query and candidate set give the same scores. A smaller cross-encoder trades some quality for speed, and a cascade can apply a cheap reranker to many candidates and an expensive one to a few. LLM-based reranking sits at the expensive end, as covered in LLM reranking.

Not every query benefits. Exact lookups, such as product codes, are usually already correct after keyword search, so reranking them wastes money. Routing by query type, or reranking only when first-stage scores are close together, keeps the reranker for queries where order is uncertain. Diversity is a separate concern, handled more cheaply by maximal marginal relevance.

How do you measure reranking latency and cost?

You measure reranking latency and cost by timing the reranking step separately from retrieval, counting tokens or requests sent to the reranker, and repeating both at each candidate count you are considering. Averages hide the slow tail.

Record latency percentiles, such as the median and the 95th or 99th percentile, rather than a single average, because users notice the slow requests. Run the test with realistic query lengths and the property you actually rerank, since long passages change both numbers. Log the reranker’s token or request counts next to quality metrics for the same queries, so each candidate count has a cost, a latency and a quality score in one row.

Repeat the measurement when anything changes: a new reranking model, a different property, longer documents or a shift in traffic. A candidate count chosen once can drift out of budget as the corpus grows.

How does reranking cost change in RAG systems?

In RAG systems reranking often pays for itself, because sending fewer, better passages to the language model can save more generation tokens than the reranker costs. The comparison depends on prices and passage counts.

Without reranking, a RAG pipeline may pass many retrieved chunks to the model to be safe, and every chunk costs generation-side input tokens. With reranking, the pipeline can retrieve more candidates, rerank them and pass only the best few. Weaviate’s advanced RAG article describes this overfetch-then-rerank pattern as a way to put the most relevant documents first for the language model.

Work out both sides with your own prices: the reranker’s cost for N candidates against the language model’s cost for the passages you no longer send. The pattern is covered in reranking in RAG, and broader cost planning in the cost of semantic search.

Reranking latency and cost grow with the number and length of candidates, because a cross-encoder scores each one against the query: with 20-token queries and 300-token passages, 50 candidates mean 16,000 tokens per query. Quality gains flatten once first-stage recall stops rising, and a reranker can never surface documents that were not retrieved. Test several candidate counts on real queries and choose the smallest that captures the gain. In Weaviate, set the query limit to that count and keep the top results yourself, rerank short passages, cache popular queries and skip reranking where order is already clear.