Home / Query understanding / Multi-Query Retrieval Explained

Multi-Query Retrieval Explained

Multi-query retrieval is a technique that asks a language model to rewrite one question into several different phrasings, runs a search for each, and merges the results into one list. Each phrasing reaches a slightly different part of the index, so documents that one wording would miss can still be found. The method targets a known weakness of vector search: a single query embedding is one point in space, and relevant documents worded differently may sit just out of reach. Merging is usually done with a simple union of unique documents or with reciprocal rank fusion, which rewards documents found by several phrasings. The approach is popular in RAG pipelines, where missing a passage means a worse answer. Its costs are an extra language model call, several searches per question and a risk of drift when a rewrite changes the meaning. It works best combined with hybrid search and a reranker.

What is multi-query retrieval?

Multi-query retrieval is a retrieval method that turns one user question into several search queries, retrieves results for each, and combines them before ranking or generation. The language model acts as a query writer, not as the answer writer.

A user asks “how do I get my money back for a broken item”. A single search might find refund pages but miss the warranty replacement page, because “warranty” never appears in the question. A language model can propose variants such as “refund for a damaged product”, “return a defective item” and “warranty replacement policy”. Each variant searches separately.

The union of their results is broader than any single search. Documents that several variants find are probably central to the question, while documents found by one variant add coverage. That is the core trade: more searches for more recall.

Multi-query retrieval is one member of a family of query transformations. Rewriting turns a query into one better query, expansion adds terms, decomposition splits compound questions, and HyDE writes a hypothetical answer to embed. The family is introduced in query understanding and query rewriting. Multi-query is distinctive in keeping several views at once. It addresses a specific limit of vector search.

Why does one query embedding miss relevant documents?

One query embedding misses relevant documents because it places the question at a single point, and relevant passages written in other words or from other angles can sit outside its nearest neighbours. The top k is a small neighbourhood.

LangChain’s documentation for its MultiQueryRetriever describes the motivation directly: its default prompt says that generating multiple perspectives on the user question helps overcome some of the limitations of distance-based similarity search. A short or ambiguous question embeds close to whichever interpretation the model favours.

Vocabulary plays a part too. A question phrased in a customer’s words may embed far from a policy written in legal terms. Embeddings narrow that gap compared with keyword search, as covered in the vocabulary mismatch problem, but they do not remove it. Different phrasings cover different wordings.

Multiple facets are the third cause. “Broken item” could mean a refund, a return, a repair or a warranty claim. A single embedding averages these facets; separate variants can each target one. The approach is related to adding terms, covered in query expansion, but keeps each angle as its own search. The pipeline has four steps.

How does multi-query retrieval work step by step?

Multi-query retrieval works in four steps: generate variants with a language model, run a search for each variant, merge the result lists, and pass the merged list to ranking or generation. Searches can run in parallel.

Generation is a single prompt. LangChain’s default MultiQueryRetriever prompt asks for 3 different versions of the user question, separated by newlines. Many teams also keep the original question as one of the searches, which LangChain supports through an include_original option that is off by default.

Each variant then runs as an ordinary search. That can be vector, keyword or hybrid search. Running the searches in parallel keeps the added delay close to that of one search plus the generation call.

Merging decides what the extra searches are worth. A plain union collects every unique document. Rank fusion scores documents by how high they rank across lists. A reranker or the generating model then works from the merged set, as in reranking in RAG. The merge method has the largest effect on the final ranking.

How are multi-query results merged?

Multi-query results are merged either as a unique union, which keeps every distinct document once, or with reciprocal rank fusion, which scores each document by its ranks across all variant lists. Fusion gives an order; a union does not.

LangChain’s MultiQueryRetriever returns the unique union of documents from all variants. That suits pipelines where a reranker or the language model will judge every candidate anyway. It does not say which documents are most supported.

Reciprocal rank fusion adds 1 divided by k plus rank for every list a document appears in. In the illustrative example, with k = 60 and ranks from 1, document B appears at ranks 2, 1 and 2 and scores 1/62 + 1/61 + 1/62, about 0.048652. Document A, at ranks 1 and 3, scores about 0.032266. Documents found by one variant score about 0.016 each. This combination of multiple queries with rank fusion is often called RAG-Fusion.

Fusion rewards agreement, which usually means relevance, but it can also push down a document that only one variant found and that matters. Keep the merged list long enough for those to survive. The formula is explained in the reciprocal rank fusion calculator. Writing the pipeline takes little code.

How do you implement multi-query retrieval with Weaviate?

You implement multi-query retrieval with Weaviate by generating variants with any language model, running a hybrid query for each, and fusing the result lists with reciprocal rank fusion in application code. Weaviate handles each search; the application handles the merge.

from collections import defaultdict

def rrf_fuse(result_lists, k=60, top=5):
    """result_lists: one ranked list of document IDs per query variant."""
    scores = defaultdict(float)
    for ranked in result_lists:
        for rank, doc_id in enumerate(ranked, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)[:top]

def multi_query_search(collection, question, generate_variants, per_query=20):
    queries = [question] + generate_variants(question)   # keep the original as one search
    lists = []
    for q in queries:
        res = collection.query.hybrid(query=q, alpha=0.5, limit=per_query)
        lists.append([str(o.uuid) for o in res.objects])
    return rrf_fuse(lists)

generate_variants is any function that asks a language model for alternative phrasings and returns them as a list of strings. Each variant runs a hybrid query, so both keyword and vector matches count. The fusion step was checked on the illustrative lists and returns B, A, E, D, C, F in that order.

Fetching 20 candidates per variant before fusing keeps documents found by only one variant in play. The fused IDs can then be fetched in full, reranked, or passed to a generative query. Hybrid search itself is covered in hybrid search. The extra calls have a price.

What does multi-query retrieval cost?

Multi-query retrieval costs one language model call per question and one extra search per variant, plus the risk that a rewrite drifts from the user’s meaning. The search cost scales with the number of variants.

The generation call usually dominates added latency, since searches can run in parallel and are fast. Three variants plus the original mean four searches instead of one, which matters for systems near capacity. Caching variants for repeated questions reduces both costs.

Drift is the quality risk. A model may reinterpret “how do I cancel” as “how do I pause”, and that variant retrieves the wrong policy. Fusion limits the damage, since one off-target list adds little against several agreeing lists. Keeping the original question as one search also anchors the results.

Too many variants dilute the signal. Each adds candidates that the ranker must sift. A handful of distinct variants is the common pattern, in line with LangChain’s default of 3. Where precision matters most, a cross-encoder after fusion removes off-topic results, as described in cross-encoder rerankers. Knowing when the gain is worth the cost decides whether to use it.

When should you use multi-query retrieval?

You should use multi-query retrieval when questions are short, ambiguous or phrased very differently from the documents, and when missing a relevant passage is costly, as in RAG. It helps recall more than precision.

Good fits include customer support assistants, where users describe problems in their own words, and research assistants, where one question touches several topics. Questions with a single clear answer, such as a product code lookup, gain little; a keyword or hybrid search already finds it.

Compound questions that ask about several separate things are better handled by decomposition, which splits the question into sub-questions and answers each, as covered in query decomposition. Multi-query rephrases one question; decomposition separates several.

HyDE takes a different route to the same goal, embedding a hypothetical answer instead of the question, as explained in HyDE. Intent classification can decide which technique each query needs, as described in query intent classification. Measuring the gain is the final step.

How do you measure whether multi-query retrieval helps?

You measure whether multi-query retrieval helps by comparing recall and nDCG with and without it on a judged set of real questions, at the number of results your pipeline uses. Track latency and cost alongside quality.

Recall at the context size is the main check for RAG: does the passage with the answer now appear more often? nDCG shows whether fusion also improved order. Precision shows whether drift added noise. The metrics are covered in recall at k and nDCG explained.

Break results down by query type. Multi-query often helps vague, conversational questions and does nothing for precise ones. If the gain is concentrated, apply the technique only to the query types that benefit, using intent classification or query length as the switch.

Compare against cheaper alternatives too. Hybrid search alone, a better embedding model or spelling correction may close much of the same gap, as described in typo tolerance and spelling correction. Suggesting better queries while the user types is another option, covered in semantic autocomplete.

Multi-query retrieval rewrites one question into several phrasings, searches with each, and merges the lists so documents missed by one wording are still found. LangChain’s default asks for 3 variants and merges by unique union, while reciprocal rank fusion ranks documents found by several variants highest, as document B shows in the worked example. It costs a language model call and extra searches, and rewrites can drift, so keep the original query, fetch enough candidates per variant, and rerank after fusion. Use it for short, ambiguous or conversational questions, and confirm the gain on judged queries before turning it on for everything.