Home / Retrieval models / What Is a Cross-Encoder?

What Is a Cross-Encoder?

A cross-encoder is a model that reads a query and a document together, as one input, and outputs a single score for how relevant the document is to the query. Because every query word can attend to every document word inside the model, cross-encoders capture fine details such as negation, word order and exact conditions. Models that encode the two texts separately miss more of these. That makes them among the most accurate relevance scorers available. The cost is speed. A cross-encoder produces no reusable vector, so it must run once for every query-document pair, which is far too slow to search a large collection directly. In practice cross-encoders rerank a shortlist of a few dozen to a few hundred candidates, retrieved first by a fast bi-encoder, keyword or hybrid search. They are also used as teachers to train better bi-encoders.

What is a cross-encoder?

A cross-encoder is a transformer model that scores the relationship between two texts by processing them jointly, rather than encoding each into its own vector. In search, the two texts are a query and a candidate document.

The name contrasts with a bi-encoder, which encodes each text separately and compares the resulting vectors. In a cross-encoder, the query and document are joined into one sequence and passed through the model together, so its attention layers can relate any part of one to any part of the other. The output is a number, often turned into a probability of relevance, rather than a vector.

Cross-encoders are used wherever a precise judgment about a pair matters: reranking search results, scoring sentence similarity, or deciding whether one sentence follows from another. In semantic search their main job is reranking, covered in cross-encoder rerankers and what is reranking.

How does a cross-encoder work?

A cross-encoder works by concatenating the query and document with separator tokens, running the combined sequence through a transformer, and mapping the output to a relevance score. Attention spans both texts.

For a BERT-style model the input looks like “[CLS] query [SEP] document [SEP]”, which is exactly how the tokenizer of the cross-encoder/ms-marco-MiniLM-L-6-v2 model formats a pair. The model’s representation of the first token, or a small head on top of it, produces the score. Because the query and document share every attention layer, a word such as “not” in the query can directly change how the document’s words are weighed.

Nogueira and Cho applied this design to passage ranking in 2019, fine-tuning BERT on MS MARCO query-passage pairs, and reported large gains over earlier reranking methods. Many current rerankers follow the same pattern with smaller or multilingual models. Transformers and BERT in search covers the architecture behind it.

Why are cross-encoders too slow for first-stage search?

Cross-encoders are too slow for first-stage search because they must run the full model once for every query-document pair, and nothing can be computed ahead of time. Cost grows with the number of candidates.

A bi-encoder embeds documents once, at index time, and compares vectors at query time with cheap arithmetic. A cross-encoder cannot: the score depends on the query, so it can only be computed after the query arrives, separately for each document. Scoring a million documents would mean a million model calls per query. Bi-encoder vs cross-encoder compares the two side by side.

The standard answer is a two-stage pipeline. A fast first stage, using dense retrieval, sparse retrieval or hybrid search, returns a shortlist. The cross-encoder then reorders only that shortlist. The shortlist size trades latency for the chance that the best document is in it, covered in reranking latency and cost.

How do you use a cross-encoder in practice?

You use a cross-encoder in practice by retrieving candidates with a fast search, scoring each candidate paired with the query, and sorting by those scores. Libraries and databases handle the plumbing.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
query = "how do I get my money back"
candidates = first_stage_search(query, k=50)          # passages from vector or hybrid search
scores = reranker.predict([(query, passage) for passage in candidates])
reranked = [p for _, p in sorted(zip(scores, candidates), reverse=True)]

The sentence-transformers library also offers a rank method that returns sorted results directly. In Weaviate, a reranker integration applies a cross-encoder to the results of a near_text, hybrid or BM25 query, so reranking happens inside the search call. Choosing a model is covered in best reranking models. Batch the pairs in one predict call, as shown, rather than scoring them one by one.

How are cross-encoders trained?

Cross-encoders are trained on labelled query-document pairs to predict relevance, learning to give high scores to pairs that match and low scores to pairs that do not. Hard negatives make them sharper.

Training data consists of queries with relevant and non-relevant passages, from datasets such as MS MARCO or from your own judged examples. Negatives that look relevant but are not, often taken from the top results of a first-stage search, teach the model the fine distinctions it is valued for. Because a cross-encoder sees both texts together, it usually learns those distinctions better than a bi-encoder trained on the same data.

Fine-tuning a cross-encoder on your own domain can help when general models misjudge specialist language, such as legal clauses or product compatibility. Measure the gain on held-out queries before deploying a fine-tuned model. Fine-tuning embedding models covers the related process for bi-encoders.

How do cross-encoders help train bi-encoders?

Cross-encoders help train bi-encoders by acting as teachers: they score many query-passage pairs, and a bi-encoder is trained to reproduce those scores or rankings. This is knowledge distillation.

The bi-encoder keeps its speed, because it still encodes texts separately, but it learns from labels that are richer than simple relevant or not-relevant marks. The teacher can also label pairs that no human judged, which multiplies the training data. Many strong retrieval models have been trained this way, with a cross-encoder or a larger model guiding a smaller, faster one.

The result is a better first stage, which in turn gives the reranker a better shortlist. Teams often use the same cross-encoder twice: once offline to train the retriever, and once online to rerank its results. Retrieval models explained places this among other training approaches.

Which tasks besides search use cross-encoders?

Tasks besides search that use cross-encoders include scoring sentence similarity, detecting duplicate questions, and judging whether one sentence follows from another. Any task that judges a pair suits them.

Semantic textual similarity benchmarks rate sentence pairs on a scale, and cross-encoders trained on them output that rating directly. Duplicate-question detection, used by forums and support sites, decides whether two questions ask the same thing. Natural language inference decides whether a premise supports, contradicts or is neutral to a hypothesis, which some systems use to check whether a retrieved passage supports a generated claim.

The same cost rule applies everywhere: one model call per pair. For pair tasks over small candidate sets that cost is fine. For anything that must compare a query against a whole collection, a bi-encoder or keyword search narrows the field first.

What are the limitations of cross-encoders?

The limitations of cross-encoders are their cost per candidate, their input length limit, their dependence on the first stage, and scores that do not transfer between models. None prevents their use as rerankers.

Each candidate costs a model call, so latency grows with shortlist size and document length. Inputs are capped by the model’s maximum sequence length, so long documents must be chunked or truncated. A cross-encoder can only reorder what the first stage found; a relevant document missing from the shortlist stays missing. Scores are calibrated only within one model, so thresholds need retuning after a model change.

Alternatives sit between bi- and cross-encoders. Late-interaction models such as ColBERT keep per-token vectors for finer matching at lower query cost, as covered in ColBERT and late interaction and multi-vector search. Learned sparse models such as SPLADE improve the first stage itself, and dense vs sparse retrieval compares those options.

Why do cross-encoders matter for RAG and agents?

Cross-encoders matter for RAG and agents because only a few passages reach the language model, so putting the best ones first has an outsized effect on answer quality. Reranking is often the cheapest quality gain.

A RAG system usually passes only a handful of passages to the model. If the first-stage search ranks a relevant passage twelfth, it never reaches the prompt. Reranking a top-50 shortlist with a cross-encoder can move it into the passages that count. Reranking in RAG covers this in detail.

Agents that search repeatedly, as described in agentic retrieval and agentic search, multiply the cost of every reranking call, so they often use small, fast cross-encoders. Measure answer quality and latency together when choosing the shortlist size and model.

A cross-encoder reads a query and a document together and outputs one relevance score, using attention across both texts to capture fine details such as negation and word order. That precision costs one model call per pair and leaves no reusable vector, so cross-encoders cannot search large collections directly. They rerank shortlists from fast bi-encoder, keyword or hybrid search, a pattern Nogueira and Cho popularised in 2019. They also train better bi-encoders through distillation. In Weaviate, reranker integrations apply them inside the search call.