Home / Query understanding / Query Decomposition for Complex Questions

Query Decomposition for Complex Questions

Query decomposition is a technique that splits a complex, multi-part question into simpler sub-questions, retrieves information for each one separately, and combines the results into one answer. A question such as “what are the refund windows for digital and physical goods, and which ships faster to Canada” contains three needs, and a single search tends to satisfy only the most prominent one. Decomposition gives each need its own search. Sub-questions can be independent, run in parallel, or sequential, where the answer to one shapes the next, as in multi-hop questions. A language model usually does the splitting, and the final step either merges the retrieved passages for a generator or answers each part and composes the result. Weaviate’s Query Agent applies decomposition automatically to multi-intent requests. The technique raises coverage for compound questions at the cost of extra model calls and more places for errors to enter.

What is query decomposition in search?

Query decomposition in search is the process of breaking one complex question into several focused sub-questions so that each can be retrieved and answered on its own. The results are then recombined to answer the original question.

Search engines and retrievers work best on questions with a single focus. A compound question mixes several, and its embedding lands somewhere between them. The retriever returns passages close to that blend, which often covers one part well and the others poorly. Decomposition separates the parts before searching.

Weaviate’s article on RAG evaluation illustrates the idea with the question “What is Ref2Vec in Weaviate?”. Answering it needs to know what Ref2Vec is and what Weaviate is, so two separate searches retrieve context for each, and the two answers are combined into one. LlamaIndex calls this pattern a sub-question query engine.

Decomposition sits in a family of query transformations. Rewriting improves one query, multi-query retrieval runs several phrasings of the same question, and decomposition runs different questions for different parts. The family is introduced in query understanding. Compound questions fail in a predictable way.

Why do multi-part questions fail with a single search?

Multi-part questions fail with a single search because one query embedding can only sit near one region of the index, and the top results cluster around the part of the question that dominates the embedding. The other parts get few or no passages.

Take “what are the refund windows for digital and physical goods, and which ships faster to Canada”. Refund policy, digital goods, physical goods and Canadian shipping are four topics. A single search might return three refund passages and nothing about shipping. A language model given that context will answer the refund part and guess, or say nothing, about shipping.

Keyword search fails differently. A long query with many terms matches documents containing any of them, which spreads results across topics without guaranteeing one good passage per part. Neither method knows the question has separate parts.

Comparisons are a common case. “Which plan includes priority support, and how does its price compare with the basic plan” needs facts about two plans. Weaviate’s Query Agent documentation describes decomposition as breaking such multi-intent questions into discrete, concurrent searches so all facets are captured. Questions come apart in two different shapes.

What are the two kinds of query decomposition?

The two kinds of query decomposition are parallel decomposition, where sub-questions are independent and run at the same time, and sequential decomposition, where each sub-question depends on the answer to the previous one. Each needs a different pipeline.

Parallel decomposition handles questions made of separate facets. The refund and shipping question splits into “refund window for digital goods”, “refund window for physical goods” and “shipping time to Canada”. None depends on the others, so the three searches can run together, and their results are merged for one answer.

Sequential decomposition handles multi-hop questions, where facts must be chained. “Who manages the team that owns the billing service” needs the team first: “which team owns the billing service”. Only with that answer, say the payments team, can the second question run: “who manages the payments team”. Research datasets such as HotpotQA, from Zhilin Yang and colleagues in 2018, were built to test this kind of reasoning across documents.

Many real questions mix both, such as comparing two items found by a prior lookup. Agentic systems plan such chains dynamically, as described in Weaviate’s article on agentic workflows. The splitting step is usually a language model.

How does a language model decompose a question?

A language model decomposes a question by being prompted to list the separate pieces of information needed to answer it, each written as a standalone search query. Good sub-questions are specific and self-contained.

The prompt asks for sub-questions that each target one fact, use the terms likely to appear in documents, and include any context needed to stand alone. “Its price” is a poor sub-question; “price of the Pro plan” is a good one. For sequential decomposition, the prompt also asks the model to mark which sub-questions depend on others.

Prompting methods from research formalize this. Least-to-most prompting, from Denny Zhou and colleagues in 2022, asks a model to reduce a problem to simpler sub-problems and solve them in order. Self-ask, from Ofir Press and colleagues in 2022, has the model pose and answer follow-up questions before the final answer. Both map naturally onto retrieval, with a search behind each step.

Not every question should be split. A single-focus question gains nothing and pays for an extra model call. A classifier or a simple rule, such as checking for “and”, “compare” or several entities, can decide when to decompose, as covered in query intent classification. The sub-questions then go to retrieval.

How do you retrieve and combine results for sub-questions?

You retrieve for sub-questions by running a separate search for each, then combine either by merging all retrieved passages into one context or by answering each sub-question and composing a final answer. The choice depends on how independent the parts are.

Merging passages is simpler. Each sub-question retrieves a few passages, the passages are labelled with their sub-question and placed in one prompt, and the model writes one answer. This works for parallel questions and keeps generation to a single call.

Answering each part separately gives more control. Each sub-question gets its own short answer from its own passages, and a final call composes them. Sequential questions need this form, because the next search uses the previous answer. It also makes errors easier to trace, since each partial answer can be checked.

Either way, each sub-question needs enough of its own passages. A fixed total context split evenly across sub-questions is a sensible default. Reranking within each sub-question’s results keeps the best passages, as described in reranking in RAG. Building this yourself takes modest code.

How do you implement query decomposition with Weaviate?

You implement query decomposition with Weaviate either through the Query Agent, which decomposes automatically, or with your own code that asks a language model for sub-questions and runs a hybrid search for each. Custom code gives full control over prompts and merging.

def decompose_and_retrieve(collection, question, split_question, per_sub=4):
    """split_question: asks a language model for standalone sub-questions."""
    sub_questions = split_question(question)
    context = []
    for sub in sub_questions:
        res = collection.query.hybrid(query=sub, alpha=0.5, limit=per_sub)
        passages = [o.properties["text"] for o in res.objects]
        context.append({"sub_question": sub, "passages": passages})
    return context   # pass to a generator, labelled by sub-question

split_question is any function that prompts a language model and returns a list of strings. Each sub-question runs a hybrid search, so exact terms such as plan names still match. The returned context keeps passages grouped under their sub-question, which helps the generator cover every part. For sequential questions, call the function step by step and insert each answer into the next sub-question.

Weaviate’s Query Agent does this inside one call. Its documentation lists query decomposition among its features, breaking multi-intent questions into concurrent searches. A Query Agent recipe shows one natural-language question producing two searches against a brands collection and one aggregation against a products collection. Hybrid search is covered in hybrid search. Decomposition adds its own failure modes.

What can go wrong with query decomposition?

Query decomposition can go wrong when the model splits a question badly, when an early answer in a chain is wrong, or when the added calls make the system too slow. Each step adds a place for error.

Bad splits are the first risk. A model may drop a part of the question, merge two parts, or invent a sub-question the user never asked. Sub-questions that lose context, such as “what is its price”, retrieve poorly. Reviewing generated sub-questions on real queries catches these patterns early.

Error propagation is the second. In a sequential chain, a wrong first answer sends every later search in the wrong direction, and the final answer can look confident while being wrong. Showing intermediate answers, or citing the passages used at each step, helps users and developers spot it.

Cost and latency are the third. Decomposition adds a planning call, several searches and sometimes several generation calls. Parallel sub-questions can run concurrently to limit the delay; sequential ones cannot. Weaviate’s article on GPU acceleration notes that agentic query rewriting can expand one query into many sub-queries, multiplying load on the vector database. The related technique for single-focus questions is in multi-query retrieval. Measurement shows whether the cost pays off.

How do you evaluate query decomposition?

You evaluate query decomposition by checking the sub-questions, the retrieval for each sub-question, and the final answer separately, on a set of real compound questions. A failure at any step explains a bad answer.

Weaviate’s article on RAG evaluation proposes this structure for multi-hop query engines. First, judge whether the decomposition makes sense, which a language model can do with a prompt that shows the question and its sub-questions. Second, run a separate retrieval evaluation for each sub-question. Third, check whether the combined answer correctly uses the parts.

Retrieval per sub-question uses the usual metrics, such as recall at the number of passages kept, as described in recall at k. Compare against a single-search baseline on the same compound questions. If decomposition does not improve coverage of all parts, its extra cost is wasted.

Other query techniques may solve part of the problem more cheaply. Expansion and rewriting help with vocabulary, as covered in query expansion and query rewriting. HyDE helps with short questions, as explained in HyDE. Spelling fixes and suggestions help before the query is even sent, as described in typo tolerance and spelling correction and semantic autocomplete.

Query decomposition splits a complex question into focused sub-questions, retrieves for each, and combines the results, so every part of the question gets its own evidence. Parallel decomposition runs independent sub-questions together; sequential decomposition chains multi-hop questions, where each answer feeds the next search. A language model does the splitting, Weaviate’s Query Agent does it automatically for multi-intent requests, and custom code can run a hybrid search per sub-question. Decompose only questions that need it, keep sub-questions self-contained, and evaluate the split, each retrieval and the final answer separately.