Home / Retrieval models / ColBERT and Late Interaction Retrieval Explained

ColBERT and Late Interaction Retrieval Explained

ColBERT is a retrieval model that keeps one vector for every token of a document and scores it by late interaction. Each query token finds its most similar document token, and those best matches are summed. The scoring step is called MaxSim. Because documents are encoded offline, ColBERT avoids running a model over every query-document pair, as a cross-encoder must. Because it compares tokens instead of one summary vector, it keeps far more detail than a single-vector bi-encoder. The price is storage. A 300-token passage with 128-dimension token vectors takes 153,600 bytes as float32, against 3,072 bytes for one 768-dimension vector. ColBERTv2 compresses token vectors with residual quantization, and encodings such as MUVERA squeeze many vectors into one. Weaviate supports ColBERT-style multi-vector embeddings natively from version 1.30.

What is the ColBERT retrieval model?

ColBERT, short for Contextualized Late Interaction over BERT, is a neural retrieval model that represents each query and document as a set of contextual token vectors rather than a single vector. Omar Khattab and Matei Zaharia published it at SIGIR in 2020.

A BERT encoder reads the text and outputs one vector per token. Each vector reflects the token in its context, so “bank” in a sentence about rivers and “bank” in a sentence about loans get different vectors. ColBERT projects each token vector down to a small size, 128 dimensions in the original paper, and normalizes it. A document of 300 tokens therefore becomes a matrix of 300 small vectors.

The model sits between two families covered elsewhere on this site. Bi-encoders compress a whole text into one vector, which is cheap to store and search but loses detail. Cross-encoders read the query and document together, which is precise but needs a model call per pair. ColBERT keeps the separate, offline document encoding of a bi-encoder and recovers much of the token-level matching of a cross-encoder. The trade-off between the two outer families is laid out in bi-encoder vs cross-encoder.

What does late interaction mean?

Late interaction means the query and document are encoded independently and only interact at the very end, in a cheap similarity step over their token vectors. The term contrasts with early interaction, where both texts pass through the model together.

In a cross-encoder, interaction happens early and everywhere: attention layers mix query and document tokens from the first layer on. That captures fine detail but means nothing can be precomputed for the document. In a bi-encoder there is almost no interaction: one query vector meets one document vector in a single dot product. Late interaction keeps every token vector until scoring time, then lets them meet through a simple operation that needs no neural network.

Because the document side is fixed, every document matrix can be computed once at indexing time and stored. At search time only the query is encoded, which is a single short model call. The interaction itself is just dot products and maxima, which hardware handles quickly. That combination is what lets ColBERT run over large collections, not just rerank a shortlist.

How does MaxSim scoring work?

MaxSim scores a document by taking each query token vector, finding its highest similarity to any document token vector, and summing those maxima across all query tokens. A document scores well when every part of the query is matched somewhere in it.

Take the query “refund policy for digital games”. Each of its tokens gets a vector. For the token “refund”, the model computes its similarity with every token vector in a candidate document and keeps only the highest. A document that says “money back on downloaded titles” may give “refund” a high match through “money back” and “games” a high match through “titles”. The document’s score is the sum of those best matches across all query tokens.

The design has useful properties. Each query term is judged on its own best evidence, so a long document is not penalised for also covering other topics. A document that matches only half the query loses the other half of the sum, so partial matches rank lower than complete ones. The per-token maxima also explain a ranking: one can see which document token supplied each query token’s score, which a single-vector similarity cannot show.

ColBERT also pads queries to a fixed length with special mask tokens. The original paper describes this query augmentation as a way for the model to add soft extra terms that expand the query. It is one reason ColBERT queries behave a little like learned query expansion.

How does ColBERT search a large collection?

ColBERT searches a large collection in two stages: an approximate nearest neighbour search over all token vectors gathers candidate documents, and exact MaxSim then ranks those candidates. Computing MaxSim against every document directly would be too slow at scale.

In the first stage, each query token vector retrieves its nearest document token vectors from an index that holds every token of every document. The documents those tokens belong to form the candidate set. In the second stage, the full MaxSim score is computed for each candidate using all of its token vectors. The original paper used FAISS for the first stage; later systems such as PLAID, also from the Stanford group, prune candidates with centroid scores to cut the work further.

Weaviate follows the same pattern inside the database. According to its documentation, the initial search phase uses compressed vectors for efficiency, and the MaxSim step uses uncompressed vectors for precise scoring. The general idea of searching one object through many vectors is covered in multi-vector search.

How does ColBERTv2 improve on ColBERT?

ColBERTv2 made late interaction cheaper to store and better trained, through residual compression of token vectors and supervision distilled from a cross-encoder. Keshav Santhanam, Omar Khattab and colleagues published it in 2022.

Residual compression rests on an observation: token vectors cluster tightly around a modest number of centroids. ColBERTv2 stores each token vector as the identifier of its nearest centroid plus a residual, the small difference from that centroid, quantized to one or two bits per dimension. With 128 dimensions at two bits each, the residual takes 32 bytes, and the paper describes about 36 bytes per vector including the centroid reference. That is a large cut from 512 bytes for the same vector in float32.

The training change, denoised supervision, uses a cross-encoder as a teacher. Hard negatives are mined with an earlier ColBERT model, scored by the cross-encoder, and the student learns to match those scores. The same distillation idea improves bi-encoders, as described in dense retrieval. The paper reported state-of-the-art quality on several benchmarks at the time, including out-of-domain sets.

Why is late interaction more accurate than one vector?

Late interaction is usually more accurate than a single vector because it keeps token-level detail that one pooled vector must blur, especially for long documents and rare terms. A single vector has a fixed size, however much text it summarises.

A 768-dimension vector for a 300-token passage has to average or pool everything the passage says. Specific details, such as a product code, an uncommon name or one condition among many, carry little weight in that average. With token vectors, that detail survives as its own vector, and a query token that needs it can find it through MaxSim. The BEIR benchmark paper by Thakur and colleagues in 2021 found ColBERT generalised to new domains better than the single-vector dense models it tested.

The gain is not free and not universal. On short, simple queries over short texts, a strong single-vector model may perform close to ColBERT at a fraction of the cost. Test on your own queries before paying the storage price; the method for doing so is in retrieval models explained.

How does ColBERT compare with sparse and dense retrieval?

ColBERT sits between dense and sparse retrieval: it uses dense neural vectors like a bi-encoder, but matches at the term level like sparse methods such as BM25 and SPLADE. It shares strengths with both and costs more than either.

Sparse retrieval scores exact term overlap with an inverted index, which is cheap and precise for rare words but blind to synonyms. SPLADE and learned sparse retrieval add learned term weights and expansion while keeping the inverted index. ColBERT matches tokens too, but through contextual vectors, so “money back” can match “refund” without a shared word. The broader trade-off is set out in dense vs sparse retrieval.

In practice ColBERT is often used as a reranker over candidates from cheaper retrieval, even though it can retrieve end to end. Reranking a few hundred candidates with stored token vectors is fast, and it avoids indexing every token of every document. That role overlaps with cross-encoder reranking, compared in cross-encoder rerankers.

How do you use ColBERT-style embeddings in Weaviate?

In Weaviate you use ColBERT-style embeddings by defining a multi-vector named vector, either filled by a model integration or by vectors you supply, and querying it with near_vector or near_text. Multi-vectors run on the HNSW index from version 1.30.

from weaviate.classes.config import Configure, Property, DataType

passages = client.collections.create(
    "Passage",
    vector_config=[Configure.MultiVectors.self_provided(name="tokens")],
    properties=[Property(name="text", data_type=DataType.TEXT)],
)
# each object's "tokens" vector is a list of token vectors (a 2-D array)
response = passages.query.near_vector(near_vector=query_token_vectors,
                                      target_vector="tokens", limit=5)

Weaviate scores multi-vector objects with MaxSim, so self-provided token matrices from a ColBERT checkpoint work directly. Its documentation also lists model integrations, such as a Jina AI ColBERT vectorizer, that create the token vectors at import. The same multi-vector support covers image models such as ColPali, which apply late interaction to patches of document page images.

Storage can be reduced two ways. Quantization (PQ, BQ, RQ or SQ) compresses each token vector, as covered in embedding quantization. MUVERA encoding, added in version 1.31, turns the whole token matrix into one fixed-length vector. Its size is repetitions times 2 to the power ksim times dprojections, which is 2,560 dimensions with the default settings. Weaviate’s documentation notes MUVERA can lower recall, partly recoverable with higher ef values.

Which search engines support ColBERT-style retrieval?

Several vector databases and search engines support ColBERT-style multi-vector retrieval natively, including Weaviate, Vespa and Qdrant, and others can run it through plugins or application code. Support differs in compression, indexing and whether MaxSim runs inside the engine.

Weaviate stores multi-vectors on HNSW with MaxSim scoring and optional MUVERA encoding. Vespa expresses ColBERT as a tensor ranking expression over stored token tensors. Qdrant offers multi-vector fields with a MaxSim comparator. Where an engine lacks native support, teams store token vectors elsewhere and run MaxSim in application code over a shortlist. That works but adds a round trip and memory outside the database.

The Stanford ColBERT repository and wrappers such as RAGatouille also provide standalone indexing and search. These suit experiments and moderate collections. For production systems that also need filters, hybrid search and access control, keeping late interaction inside the database is simpler. Evaluate each option on recall at your latency target, not on feature lists alone.

When should you use ColBERT?

Use ColBERT when single-vector retrieval misses relevant results because of long documents, rare terms or domain shift, and when you can afford the extra storage or a reranking stage. It is a quality upgrade, not a default.

Good fits include technical documentation with exact identifiers, legal or scientific text where one clause decides relevance, and visual document search with ColPali-style models. Poor fits include very large collections on a tight memory budget and short texts where a single vector already performs well. A middle path is to retrieve with a bi-encoder or hybrid search and rerank the top results with stored ColBERT vectors.

Agents that search repeatedly, described in agentic retrieval and agentic search, benefit from late interaction’s precision but multiply its cost per question. Measure recall, latency and storage together before committing, and compare against a tuned single-vector baseline.

ColBERT keeps one contextual vector per token and scores documents by late interaction. Each query token takes its best match among a document’s tokens, and MaxSim sums those matches. Documents are encoded offline, so search needs only a query encoding and cheap arithmetic. Accuracy usually beats single-vector retrieval on long and specialised text, at a large storage cost: 153,600 bytes for a 300-token passage uncompressed. ColBERTv2’s residual compression and Weaviate’s quantization and MUVERA encoding cut that cost, and Weaviate supports multi-vector embeddings natively from version 1.30.