Home / Retrieval models / Bi-Encoder vs Cross-Encoder: Differences and When to Use Each

Bi-Encoder vs Cross-Encoder: Differences and When to Use Each

Bi-encoders and cross-encoders are two transformer architectures for scoring a query against a document: a bi-encoder embeds each text separately, while a cross-encoder reads both texts together. Separate encoding lets a bi-encoder store document vectors in advance and search millions of them quickly. Joint encoding lets a cross-encoder compare every query token with every document token. That usually ranks more accurately, but it costs one model pass per document for every query. Nils Reimers and Iryna Gurevych measured the gap in 2019. Comparing all pairs among 10,000 sentences took about 65 hours with a BERT cross-encoder and about 5 seconds of embedding with Sentence-BERT. The standard pipeline therefore retrieves candidates with a bi-encoder and reranks a short list with a cross-encoder.

What is the difference between a bi-encoder and a cross-encoder?

The difference between a bi-encoder and a cross-encoder is where the query and the document meet. A bi-encoder encodes each text alone and compares the two output vectors with a similarity function such as cosine similarity. A cross-encoder feeds the query and the document into one transformer as a single input and outputs one relevance score. The Sentence Transformers documentation states the consequence directly. A cross-encoder produces no sentence embedding, and it cannot take a single sentence as input.

Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux and Jason Weston of Facebook AI Research described the same split in their 2019 poly-encoder paper. They call both designs fine-tuned architectures built on top of a pretrained BERT model. Bi-encoders run self-attention over the input and the candidate separately, so encoded candidates can be cached. Cross-encoders run full self-attention over the pair and often score better, but they recompute every pair at test time.

The comparison table sets the two architectures side by side on the attributes that decide where each one runs. Every row follows from one design choice: whether the document is encoded before the query exists.

Separate encoding makes document work reusable across queries, and joint encoding makes every score specific to one query and one document. A bi-encoder’s output is an embedding that a vector database can index. A cross-encoder’s output is a number that exists only for the pair it scored. The overview of retrieval models places both among sparse, dense and late-interaction retrievers. The bi-encoder half of that design starts with how one text becomes one vector.

How does a bi-encoder score a query against a document?

A bi-encoder scores a query against a document by embedding each one separately and measuring the similarity of the two vectors. The model reads the document alone and compresses it into one fixed-size vector. At search time it reads the query alone and produces a vector in the same space. Cosine similarity or a dot product between the two vectors gives the score. The full bi-encoder guide covers pooling and training.

Independence is the property that makes a bi-encoder fast. Document vectors do not depend on any query, so they are computed once and stored. Weaviate’s text2vec-transformers vectorizer works this way. Its documentation says it generates object embeddings at import time and converts each text query into an embedding at search time. An approximate nearest neighbor index then finds the closest stored vectors without comparing every one.

Bi-encoders power dense retrieval, where the same design is often called a dual encoder. Vladimir Karpukhin and seven coauthors at Facebook AI, the University of Washington and Princeton published Dense Passage Retrieval in 2020. Their dual-encoder retriever beat a Lucene-BM25 system by 9% to 19% absolute in top-20 passage retrieval accuracy. That result shows the bi-encoder’s real job: finding a good candidate set across a large collection.

The cost of independence is lost interaction. The document vector is fixed before the query exists, so the model cannot weight the words one particular query cares about. Two documents on the same topic land close together even when only one answers the question. A cross-encoder removes that limit by never encoding the document alone.

How does a cross-encoder score a query against a document?

A cross-encoder scores a query against a document by reading both texts as one input and predicting relevance from the joint representation. Rodrigo Nogueira and Kyunghyun Cho of New York University described the standard recipe in their 2019 paper, Passage Re-ranking with BERT. The query is fed as sentence A and the passage as sentence B. A single-layer network on BERT’s [CLS] output gives the probability that the passage is relevant.

Joint input also means a shared length budget. Nogueira and Cho truncated the query to 64 tokens and the whole query-passage pair to 512 tokens. Every candidate needs its own forward pass, because the passage representation changes with each query. Nothing about the document is reusable from one query to the next. The cross-encoder guide covers training data and model sizes.

A cross-encoder score is a ranking signal, not a calibrated similarity. The Sentence Transformers cross-encoder overview describes an output between 0 and 1. Its usage documentation, read October 8, 2026, adds that MS MARCO cross-encoders return logits instead. Loading the model with a sigmoid activation maps those logits to the range 0 to 1 without changing the ranking. Bi-encoder cosine scores and cross-encoder scores sit on different scales, so a pipeline never mixes them in one sort.

Reading the pair jointly is the source of the cross-encoder’s advantage. The model sees the query while it reads the document, so query words shape how every document word is represented. That interaction is what a fixed document vector cannot provide.

Why is a cross-encoder more accurate than a bi-encoder?

A cross-encoder is more accurate than a bi-encoder because every query token can attend to every document token before the model commits to a score. Humeau and coauthors describe the effect: concatenating the two texts applies self-attention at every layer. Every word in the candidate can attend to every word in the input, and the reverse. A bi-encoder must compress the document into one vector before it sees the query. The Sentence Transformers documentation states that cross-encoders achieve better performance than bi-encoders.

The original BERT re-ranking result shows the scale of what a cross-encoder adds. On the MS MARCO passage development set, Nogueira and Cho report MRR@10 of 16.7 for untuned Lucene BM25. Re-ranking BM25’s top 1,000 passages with BERT Large raised MRR@10 to 36.5. Their system topped the MS MARCO leaderboard at the time, 27% relative above the previous best. That comparison is a cross-encoder against BM25, so it measures reranking value rather than the architecture gap alone.

Joint reading matters most where small words or context change the answer. Negation, as in a policy stating that employees cannot approve their own expenses, changes relevance without changing the topic. An ambiguous name such as “apple” means a fruit or a company depending on its neighbours. A question asking for a date needs a passage that states a date, not one that only discusses the same event. A single topical vector tends to place all of these texts close together.

Higher accuracy is not automatic outside the training domain. Weaviate’s engineering blog on cross-encoders as rerankers notes their high in-domain accuracy and recommends training or fine-tuning for out-of-domain data. The same post says its MS MARCO example model was trained on Bing search queries. Accuracy also carries a price, and that price grows with every document scored.

Why is a bi-encoder faster than a cross-encoder?

A bi-encoder is faster than a cross-encoder because its expensive model work happens once per document, while a cross-encoder repeats model work for every query-document pair. A bi-encoder search runs one forward pass for the query and then compares vectors. A cross-encoder needs one forward pass for each document it scores. The Sentence Transformers usage documentation describes cross-encoders as requiring computation for each pair rather than for each text.

Nils Reimers and Iryna Gurevych quantified the gap in their 2019 Sentence-BERT paper at EMNLP-IJCNLP. Finding the most similar pair among 10,000 sentences takes n(n-1)/2, or 49,995,000, BERT inference computations. That took about 65 hours on a V100 GPU. Sentence-BERT embedded the same 10,000 sentences in about 5 seconds, and the cosine comparisons took about 0.01 seconds. The Sentence-BERT paper made bi-encoder search practical for that reason.

Search has the same shape, with a query in place of one sentence of the pair. The Sentence-BERT authors estimate that matching one new question against over 40 million Quora questions with BERT would take over 50 hours. With an optimized vector index, the same paper reports, the bi-encoder search drops to a few milliseconds. A cross-encoder therefore never scans a large corpus directly.

A model-pass count makes the trade-off concrete for any corpus. With N documents, Q queries and k reranked candidates per query, a bi-encoder needs N + Q passes. A cross-encoder over the whole corpus needs Q × N passes. Retrieve-then-rerank needs N + Q + Q × k passes. With a single query, Q × N equals N and undercuts N + Q, so an index pays off only once queries repeat. Pair passes read longer inputs than single texts, so equal counts do not mean equal seconds. Measured costs per candidate are covered in reranking latency and cost. The calculator applies the three formulas to any corpus size.

Documents (N):

1,000,000

Queries (Q):

1,000

Candidates reranked per query (k):

100

How does retrieve-then-rerank combine a bi-encoder and a cross-encoder?

Retrieve-then-rerank combines a bi-encoder and a cross-encoder by letting the bi-encoder find a short candidate list and the cross-encoder reorder it. The Sentence Transformers retrieve-and-rerank example retrieves 100 candidates for a query, scores each with a cross-encoder, and shows the top 5. Weaviate’s reranking documentation gives the reason for two stages. Scoring every query-object pair would typically be prohibitively slow, so reranking runs after retrieval on a smaller set.

Weaviate runs both stages in one query. Its text2vec-transformers vectorizer is the bi-encoder side, and its reranker-transformers module is the cross-encoder side. The reranker documentation lists three MS MARCO cross-encoders: ms-marco-MiniLM-L-6-v2, ms-marco-MiniLM-L-2-v2 and ms-marco-TinyBERT-L-2-v2. Weaviate’s reranker modules work after vector, BM25 or hybrid search. The Transformers reranker runs in its own inference container and is not available on Weaviate Cloud.

The first stage does not have to be a bi-encoder. Nogueira and Cho reranked the top 1,000 BM25 results, and the Sentence Transformers documentation lists lexical search as an alternative retriever. A bi-encoder earns the first-stage role when queries and documents use different words. The same documentation notes that lexical search does not recognize synonyms, acronyms or spelling variations.

Reranking cannot recover a document the first stage missed. The cross-encoder only sees the k candidates it receives, so a relevant document at rank k + 1 never reaches it. Candidate depth therefore sets the ceiling on final quality, and a larger k buys recall with more pair passes. The guides to reranking and reranking in RAG pipelines cover depth choices for result pages and LLM context. The same two stages fit in a short Python script.

How do you run a bi-encoder and a cross-encoder in Python?

You run a bi-encoder and a cross-encoder in Python with the Sentence Transformers library, using SentenceTransformer for retrieval and CrossEncoder for reranking. The script embeds six help-desk documents with multi-qa-MiniLM-L6-cos-v1 and retrieves the four nearest to a question. It then reranks those four with ms-marco-MiniLM-L6-v2. It was tested with sentence-transformers 5.1.2 and PyTorch 2.8.0 on October 8, 2026.

from sentence_transformers import CrossEncoder, SentenceTransformer, util

docs = [
    "Change your VPN password in the self-service portal under Account settings.",
    "The VPN client stops connecting when the laptop enters battery saver mode.",
    "Email password resets need approval from your manager.",
    "Request a new VPN certificate when the current one expires.",
    "The VPN password must be at least 12 characters long.",
    "The cafeteria closes at 3 pm on Fridays.",
]
query = "How do I change my VPN password?"

# Stage 1: bi-encoder. Documents are embedded once and reused for every query.
bi_encoder = SentenceTransformer("sentence-transformers/multi-qa-MiniLM-L6-cos-v1")
doc_embeddings = bi_encoder.encode(docs, convert_to_tensor=True)
query_embedding = bi_encoder.encode(query, convert_to_tensor=True)
hits = util.semantic_search(query_embedding, doc_embeddings, top_k=4)[0]
candidates = [docs[hit["corpus_id"]] for hit in hits]

# Stage 2: cross-encoder. Each (query, candidate) pair is read together.
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")
ranked = cross_encoder.rank(query, candidates, top_k=2)

for hit in hits:
    print(f"bi-encoder   {hit['score']:.3f}  {docs[hit['corpus_id']]}")
for item in ranked:
    print(f"cross-encoder {item['score']:.3f}  {candidates[item['corpus_id']]}")

Both stages put the self-service portal instruction first in the test run. The difference is the margin. The bi-encoder’s cosine scores for the top two documents were 0.766 and 0.704, close together although only one explains how to change the password. The cross-encoder returned logits of 6.326 and 0.780, separating the instruction from the password-length rule. Those logits are ranking scores, not probabilities, as the Sentence Transformers documentation notes for MS MARCO models.

CrossEncoder.rank sorts the candidates and returns each one’s corpus_id and score. CrossEncoder.predict returns raw scores for a list of pairs when ranking happens elsewhere. Production code embeds documents once and stores the vectors in a vector database such as Weaviate rather than in memory. Batch size, candidate count and model choice belong to the guide on cross-encoder rerankers. Two separate models are also not the only design available.

Which models are a middle ground between a bi-encoder and a cross-encoder?

ColBERT-style late-interaction models and poly-encoders are two middle grounds between a bi-encoder and a cross-encoder. Both keep document encodings precomputable, like a bi-encoder. Both add a cheap interaction step after encoding, so the query still influences how the document is matched. Neither reads the query and the document as one joint input, which is what defines a cross-encoder.

Omar Khattab and Matei Zaharia of Stanford University introduced ColBERT and the term late interaction at SIGIR 2020. ColBERT encodes the query and the document separately into one contextual vector per token. Each query vector finds its maximum similarity among the document’s vectors, and the score sums those maxima. Used as a reranker, ColBERT reported over 170 times speedup and 14,000 times fewer FLOPs than existing BERT-based models. Late interaction is therefore its own architecture, covered in ColBERT and late interaction and multi-vector search.

The poly-encoder from Humeau and coauthors caches candidate embeddings like a bi-encoder. It then scores each candidate through an attention step between the cached candidate embedding and several learned context vectors. Their experiments found poly-encoders faster than cross-encoders and more accurate than bi-encoders. Reimers and Gurevych noted one limit in the Sentence-BERT paper: the poly-encoder score is not symmetric, and clustering would still need a score for every pair. With three families available, the choice depends on corpus size, latency budget and whether stored embeddings are needed.

When should you use a bi-encoder, a cross-encoder, or both?

Use a bi-encoder when every query searches a large collection, a cross-encoder when the pairs to score are few and known, and both when a large collection needs a precise top list. The Sentence Transformers documentation gives the cross-encoder case as a predefined set of pairs, such as 100 pairs to score. It names semantic search and clustering as bi-encoder tasks, because both need an embedding in a vector space.

The decision table maps common situations to an architecture. Each reason comes from the cost and accuracy properties of the two designs.

The in-document case comes from the Sentence Transformers retrieve-and-rerank page. Its example splits one Wikipedia article into paragraphs and scores every paragraph with a cross-encoder, skipping retrieval entirely. Latency settles close cases elsewhere. A reranker adds k pair passes to every query, on top of query embedding and vector search. The search latency breakdown shows where those milliseconds go, and the comparison of reranking models covers smaller and larger cross-encoders. Whichever combination wins on paper still needs a test on real queries.

How do you measure whether a cross-encoder improves a bi-encoder?

Measure whether a cross-encoder improves a bi-encoder by scoring both stages separately on the same judged queries. Stage one needs recall@k at the candidate depth k, because recall at that depth fixes what the reranker can see. Stage two needs a rank-sensitive metric such as nDCG at 10 or mean reciprocal rank at 10. Nogueira and Cho reported MRR@10 on MS MARCO for the same reason.

Compare three configurations: the bi-encoder alone, the bi-encoder plus a cross-encoder at the chosen k, and the same pipeline at a larger k. If the larger k raises recall@k but not the final metric, the cross-encoder is not using the extra candidates. If the final metric matches the bi-encoder alone, the reranker adds latency without adding quality. Record latency at a high percentile beside each relevance number.

Domain shift is the failure to test for first. A cross-encoder trained on MS MARCO web queries may rank a legal or medical corpus worse than its benchmark suggests. A small judged set drawn from the real query log exposes that before launch. The guide to evaluating semantic search covers judgment collection and significance testing.

Bi-encoders and cross-encoders answer different questions about the same query and document. The bi-encoder asks which documents sit near the query in a vector space, and it answers across millions of documents because document vectors are precomputed. The cross-encoder asks how relevant one document is to one query, and it answers more precisely because it reads both texts together. The pattern documented by Sentence Transformers and Weaviate uses the bi-encoder for recall and the cross-encoder for the final order. Candidate depth k is the setting that joins them, and judged queries decide its value.