Home / Reranking / Cross-Encoder Rerankers: How to Use Them

Cross-Encoder Rerankers: How to Use Them

A cross-encoder reranker is a transformer model that reads a query and one candidate passage together and outputs a relevance score, used to reorder the top results of a fast first-stage search. Because it sees both texts at once, it judges relevance more precisely than the vector similarity used for retrieval. It also cannot precompute anything, so it runs one model pass per candidate and is used only on a short list. Using one takes three decisions: which model, how many candidates to rerank, and where to run it. Open models from the Sentence Transformers library run in a few lines of Python. Hosted rerankers are called over an API. Weaviate can run either inside a query through its reranker integrations, including a locally hosted Transformers container. Latency grows in step with the candidate count, so the right number is the smallest that keeps the relevant passage in the list.

What is a cross-encoder reranker?

A cross-encoder reranker is a model that takes a query and a passage as one combined input and predicts how relevant the passage is to the query. It is called a cross-encoder because attention crosses between the two texts inside the model.

The model is usually a BERT-style transformer fine-tuned on pairs of queries and passages labelled relevant or not. The query and passage are joined with separator tokens, the model reads the whole sequence, and a small head on top outputs a single number. Higher means more relevant. In the Python example later on, a relevant passage scored 6.19 while unrelated ones scored -5.55 and -11.04, which shows the scores are unbounded logits.

The contrast is with the bi-encoder used for retrieval, which encodes the query and passage separately into vectors. Bi-encoder vs cross-encoder compares the two designs, and what is reranking covers reranking in general. The joint reading is both the strength and the cost.

Why are cross-encoders more accurate than vector search?

Cross-encoders are more accurate than vector search because they compare the query and passage word by word inside one model, instead of compressing each into a single vector before comparing. Detail that a vector averages away stays visible.

A bi-encoder must squeeze a whole passage into a few hundred numbers before it ever sees the query. Two passages about the same topic can end up near each other even if only one answers the question. A cross-encoder reads “refund for a duplicate charge” against each passage and can tell that one about duplicate charges matters and one about duplicate accounts does not, even though both share the word “duplicate”.

The price is speed. A bi-encoder computes passage vectors once, at indexing time, so a search costs one query encoding plus an index lookup. A cross-encoder must run once for every query-passage pair at search time. Running it over a million passages per query is impractical, which is why it reranks a short list from the first stage. Choosing the model is the first decision.

Which cross-encoder models can you use?

You can use open cross-encoder models from the Sentence Transformers and Hugging Face ecosystems, or hosted reranking APIs from providers such as Cohere, Jina AI and Voyage AI. Open models run on your hardware; hosted ones trade control for convenience.

Open models trained on the MS MARCO passage ranking dataset, such as cross-encoder/ms-marco-MiniLM-L6-v2, are small and fast and a common starting point for English web-style questions. Larger open rerankers, such as the BAAI bge-reranker family, trade speed for accuracy and add multilingual support. Hosted rerankers are called over HTTP with a query and a list of documents and return scores. Best reranking models compares the options.

Domain fit matters more than leaderboard rank. A reranker trained on web search questions can rank scientific, legal or internal text worse than the first stage did. Test any candidate model on your own queries before committing, as described in reranking in RAG. Once chosen, the model takes a few lines to run.

How do you use a cross-encoder reranker in Python?

You use a cross-encoder reranker in Python by loading the model with the Sentence Transformers CrossEncoder class and calling rank with the query and the candidate passages. It returns the candidates sorted by score.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
query = "how do I get a refund for a duplicate charge"
candidates = [
    "Refunds for duplicate charges are issued to the original card within 5 days.",
    "You can change the card used for future payments in billing settings.",
    "Duplicate accounts can be merged by contacting support.",
    "Invoices list every charge made to your account.",
]
for hit in reranker.rank(query, candidates, top_k=3):
    print(round(float(hit["score"]), 2), candidates[hit["corpus_id"]])

Running it printed the refund passage first at 6.19, then the duplicate accounts passage at -5.55 and the invoices passage at -11.04. The model correctly separated the duplicate charge from the duplicate account. In a real pipeline, the candidates come from a vector or hybrid search, and the reranked top few go to the user or to a language model.

Two settings matter in production. Batch size controls how many pairs run together; larger batches use hardware better. Maximum sequence length controls truncation; most small cross-encoders read a few hundred tokens of query plus passage, so long chunks are judged on their opening. Running the model inside the database is the other option.

How do you run a cross-encoder reranker in Weaviate?

You run a cross-encoder reranker in Weaviate by enabling a reranker integration, configuring it on the collection, and adding a Rerank object to the query. Weaviate then reranks the search results before returning them.

For a self-hosted open model, Weaviate’s documentation describes the reranker-transformers module, which runs the model in a separate inference container. In Docker Compose, the Weaviate service sets ENABLE_MODULES to reranker-transformers and points RERANKER_INFERENCE_API at the container, whose image tag selects the model, for example cross-encoder-ms-marco-MiniLM-L-6-v2. ENABLE_CUDA switches the container to a GPU. The documentation notes this integration is not available on Weaviate Cloud, since it needs its own container.

from weaviate.classes.config import Configure, Property, DataType
from weaviate.classes.query import Rerank

client.collections.create(
    "HelpArticle",
    properties=[Property(name="body", data_type=DataType.TEXT)],
    reranker_config=Configure.Reranker.transformers(),
)
articles = client.collections.use("HelpArticle")
response = articles.query.hybrid(
    query="refund for a duplicate charge",
    limit=25,                                   # candidates reranked and returned
    rerank=Rerank(prop="body", query="refund for a duplicate charge"),
)

Hosted rerankers use the same query pattern with a different configuration, such as Configure.Reranker.cohere or voyageai, and an API key header. Weaviate’s documentation states that a collection’s reranker configuration can be changed after creation from versions 1.25.23, 1.26.8 and 1.27.1. The limit in the query is both the reranker’s input and the number of results, so candidate count is set there.

How many candidates should a cross-encoder rerank?

A cross-encoder should rerank enough candidates that the relevant result is usually among them, and no more, because each extra candidate adds one more model pass. The right number comes from measuring first-stage recall.

The method is simple. For a set of labelled queries, check how often a relevant document appears in the first stage’s top 10, 25, 50 and 100. Pick the smallest depth where that rate stops rising. Reranking deeper than that adds cost without new chances to succeed, and reranking shallower caps quality at the first stage’s ordering. Recall at k covers the measurement.

The answer depends on the first stage. A strong hybrid search may place the right passage in the top 20 most of the time, while a weak vector search may need 100. Improving the first stage often lets the reranker see fewer candidates. The candidate count drives latency directly.

How much latency does a cross-encoder reranker add?

A cross-encoder reranker adds latency proportional to the number of candidates times the cost of one forward pass, which depends on model size, passage length and hardware. With a large model on a CPU, it can take longer than the rest of the pipeline combined.

Three levers cut it. A smaller model, such as a six-layer MiniLM instead of a large reranker, runs faster per pass. Shorter inputs, by reranking chunk text rather than whole documents, reduce tokens per pass. Fewer candidates reduce passes. Running on a GPU, which Weaviate’s Transformers container supports with ENABLE_CUDA, speeds every pass.

Hosted rerankers move the compute elsewhere but add network time and a per-query fee. Measure end-to-end latency at the percentiles your users feel, not only the average. Reranking latency and cost covers budgeting in detail. When a cross-encoder is too slow or too narrow, alternatives exist.

When should you use something other than a cross-encoder?

You should use something other than a cross-encoder when latency is too tight, when the task needs reasoning beyond passage relevance, or when results need diversity rather than precision. Each case has a better tool.

For tight latency, skip reranking and improve the first stage with hybrid search and better chunking, or use a late interaction model such as ColBERT, which keeps token-level detail at lower query cost. For complex questions that need multi-step judgment, an LLM-based reranker can compare passages against instructions, at much higher cost, as covered in LLM reranking.

For diversity, a cross-encoder can make things worse by ranking near-duplicate passages together at the top. Maximal marginal relevance rebalances results so they cover different points. Many pipelines rerank for relevance first and then apply a diversity step. Whatever the combination, measure it on your own queries.

A cross-encoder reranker reads each query and candidate passage together and scores relevance more precisely than vector similarity, at the cost of one model pass per candidate. Use it on a short list from vector or hybrid search, chosen by measuring how deep the first stage must go to include the relevant passage. Run it in Python with the Sentence Transformers CrossEncoder class, or inside Weaviate with the reranker-transformers container or a hosted integration. Pick the model by testing on your own domain, keep inputs short and candidates few for latency, and confirm the gain with a labelled query set before shipping.