Home / Query understanding / Query Rewriting With LLMs for Better Retrieval

Query Rewriting With LLMs for Better Retrieval

Query rewriting is the step that turns a user’s raw question into a better search query before retrieval, usually by prompting a language model to restate it. It fixes inputs that search engines handle badly: follow-up questions with pronouns, chatty phrasing, missing context and mismatched vocabulary. Ma and colleagues named the pattern Rewrite-Retrieve-Read in an EMNLP 2023 paper, putting the rewrite before the retrieve-then-read pipeline used in RAG. In this site’s test, rewriting 40 conversational follow-ups with a small local model raised hybrid search nDCG@10 from 0.392 to 0.736. On 300 queries that were already clear, rewriting changed almost nothing: 33 improved and 39 got worse. Rewrite when queries depend on context or wander from the corpus’s language, and measure before rewriting everything.

What is query rewriting in search and RAG?

Query rewriting in search and RAG is the transformation of the user’s input into a query that matches how the documents are written and indexed, done before any retrieval runs. The user still sees their own words; only the search system sees the rewrite.

The idea predates language models. Search engines have long corrected spelling, normalised terms and added synonyms. What changed is the rewriter. Ma, Gong, He, Zhao and Duan’s 2023 paper, “Query Rewriting for Retrieval-Augmented Large Language Models”, prompts an LLM to generate the search query and then retrieves with a web search engine. It also trains a small rewriter model with reinforcement learning, using the reader LLM’s feedback. The authors report consistent gains on open-domain and multiple-choice question answering.

Query rewriting sits inside query understanding, the wider set of steps that interpret what a user wants. Rewriting changes the query text. Related steps change other things: query intent classification routes the query, and typo tolerance and spelling correction repairs single words. The rewrite is the step that can change the whole sentence, which makes it powerful and risky at once.

In a RAG system the same rewritten query usually feeds retrieval only, while the original question still goes to the answering model. Keeping the two apart matters, because the best query for an index is rarely the best prompt for a model. That separation also explains why rewriting helps at all.

Why does query rewriting improve retrieval?

Query rewriting improves retrieval because users phrase questions for people, not for indexes, and the rewrite closes the gap between the two. The largest gap appears in conversation, where the key term often sits in an earlier message.

A follow-up like “Does it raise homocysteine in the blood?” means nothing to a search engine on its own. “It” carries no keyword weight in BM25, and its embedding drifts towards any document about blood chemistry. The user meant vitamin B12 deficiency, named one turn earlier. A rewriter that reads the history can restore that term, which is what most of the gain in this site’s test came from.

Vocabulary is the second gap. Users write “bones getting weak” while the corpus says “osteoporosis”. A Weaviate blog post on advanced RAG gives a medical example. A patient’s message, “My left shoulder hurts and I have numbness in my thumb and index finger”, becomes “Pain left shoulder. Numbness. Numbness index finger.” The rewrite strips narrative and keeps the clinical signal.

Noise is the third. Long chat messages mix the question with greetings, constraints and asides, and every extra word shifts a vector and adds BM25 terms. Rewriting can drop the padding. It can also turn “cheap ones under $70” into a price filter rather than search words, which Weaviate’s Query Agent does, as covered later. With the reasons clear, the mechanics follow.

How does query rewriting work in a search pipeline?

Query rewriting works in a search pipeline by sending the latest user message, plus any needed history, to a language model with instructions to return a standalone search query, and then searching with that output. The rest of the pipeline is unchanged.

The rewriter can be any instruction-following model. In this site’s test it was Qwen2.5-3B-Instruct in a 4-bit build, running locally on a laptop with greedy decoding. A hosted model adds network latency and cost per query, so many teams use a small model or cache rewrites for frequent inputs. The minimal version is a prompt and one function call.

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Qwen2.5-3B-Instruct-4bit")
PROMPT = ("Rewrite the user's last message as a standalone search query. Replace pronouns with "
          "what they refer to, using the earlier message. Keep every specific term. "
          "Output only the query.\n\nEarlier message: {t1}\nLast message: {t2}\n\nStandalone query:")

def rewrite(t1, t2):
    p = tokenizer.apply_chat_template([{"role": "user", "content": PROMPT.format(t1=t1, t2=t2)}],
                                      add_generation_prompt=True, tokenize=False)
    return generate(model, tokenizer, prompt=p, max_tokens=60).strip().split("\n")[0]

print(rewrite("What happens in vitamin B12 deficiency?", "Does it raise homocysteine in the blood?"))

The output was “Does vitamin B12 deficiency raise homocysteine in the blood?” The rewritten string then goes to BM25, vector or hybrid search exactly as a typed query would. Some systems go further and produce several rewrites, which leads to a family of related techniques.

Which query rewriting techniques are used for retrieval?

The query rewriting techniques used for retrieval are standalone rewriting, normalisation, expansion, multi-query generation, decomposition, step-back prompting and hypothetical document generation. They differ in how many queries come out and how far each one moves from the original.

• Standalone rewriting resolves pronouns and ellipsis from conversation history into one self-contained query.
• Normalisation strips filler, fixes spelling and maps informal words to the corpus’s terms.
• Expansion adds synonyms or related terms to the same query, covered in query expansion.
• Multi-query generation writes several phrasings and merges their results, covered in multi-query retrieval.
• Decomposition splits a compound question into sub-questions, covered in query decomposition.
• Step-back prompting asks a broader question first to fetch background, then the specific one.
• Hypothetical documents embed an LLM-written answer instead of the question, covered in HyDE.

The list runs roughly from conservative to aggressive. A standalone rewrite should keep the user’s meaning exactly. HyDE replaces the query with invented text whose facts may be wrong but whose vocabulary resembles the answer. The more a technique adds, the more it can help sparse queries and the more it can drift on clear ones. Semantic autocomplete attacks the same problem from the other side, by helping users type a better query before any rewrite. The conservative end is where the clearest evidence sits, starting with conversation.

Does query rewriting help conversational search?

Query rewriting helps conversational search a great deal, because follow-up messages lose their subject when searched alone. In this site’s test, it nearly doubled ranking quality for follow-ups.

We wrote 40 two-turn conversations by hand, each built on one claim from BEIR SciFact, a public set of 5,183 scientific abstracts with relevance labels. The first turn names the topic, such as “What is albendazole?”. The second leaves it out: “Is it used for lymphatic filariasis?”. Each follow-up was searched four ways and scored against the original claim’s labels. Retrieval used our own BM25 at Weaviate’s default parameters, all-MiniLM-L6-v2 vectors, and reciprocal rank fusion of the two.

The table shows two things. Rewriting nearly closed the gap to the original claim, especially for BM25, at 0.728 against 0.731. And the free baseline of joining both turns recovered most of the gain, at 0.719 for hybrid. Vector search suffered most from the bare follow-up, at 0.235, because a pronoun-heavy sentence embeds as a generic question.

The conversations are hand-written and omit assistant replies, which real histories include. They also share one domain. The direction is clear, but expect smaller gains when users already repeat the subject. The harder question is whether clear queries benefit too.

Does query rewriting help queries that are already clear?

Query rewriting does not reliably help queries that are already clear; on such queries it improves some, worsens others and averages out near zero. Rewriting every query by default adds latency and risk for little return.

We asked the same model to rewrite all 300 SciFact test claims as concise literature-search queries. Hybrid nDCG@10 went from 0.688 for the original claims to 0.683 for the rewrites. Per query, 33 rewrites helped, 39 hurt and 228 left the score unchanged. BM25 gained slightly, from 0.663 to 0.670, while vector search lost, from 0.648 to 0.633. Shorter keyword-style rewrites suit BM25 and starve the embedding of context.

Merging was the safer use. Searching with both the original and the rewrite, then fusing the two lists with reciprocal rank fusion, gave 0.698, a small gain over either alone. That is the logic of multi-query retrieval: the rewrite adds a second view instead of replacing the first. Whether a rewrite helps depends heavily on the prompt.

How do you write a query rewriting prompt?

You write a query rewriting prompt by stating the job narrowly, telling the model to keep every specific term, giving it the history it needs, and demanding the query alone with no explanation. Each rule blocks a failure seen in practice.

The two prompts in this site’s test differed only in purpose. The conversational prompt asked for a standalone query that replaces pronouns using the earlier message. The standalone prompt asked for a concise query for a scientific literature search engine. Both included “Keep every specific term” and “Output only the query”. Even so, the 3-billion-parameter model made recognisable mistakes, all visible in the saved rewrites.

• Copying history: 21 of 40 conversational rewrites repeated the first question in full instead of merging it into the follow-up.
• Adding words: 27 of 300 standalone rewrites gained the phrase “literature review”, which matches no abstract’s topic.
• Changing syntax: 33 rewrites used AND operators and 12 joined words with underscores; the test engines treat both as plain text.
• Losing the question: “Is it determined only by genes?” became “What determines breast cancer, genes or other factors?”, which loosens the claim.

Few-shot examples in the prompt reduce these errors, and a larger model reduces them further. A cheap guard is to check that every rare term from the original appears in the rewrite and to fall back to the original if not. Managed services hide this work, and Weaviate offers one.

How does Weaviate support query rewriting?

Weaviate supports query rewriting through its Query Agent, which turns a natural language request into one or more Weaviate queries with search terms, filters or both. Applications that want their own rewriter can also pass any rewritten string to a normal hybrid or vector query.

Weaviate’s documentation describes the Query Agent’s Search Mode as using query rewriting to produce Weaviate queries, then reranking the results against the original request. A filtering setting controls breadth. The default, “recall”, generates several queries across different filters and interpretations. The “precision” setting generates a single query for the most likely interpretation. For chat, the agent accepts the conversation as a list of user and assistant messages and treats the final user message as the query, which covers the follow-up case tested here.

The documentation lists limits. The Query Agent runs on Weaviate Cloud only, not on local instances. Because it makes several model and database calls, runs of about 10 seconds are not unusual for complex queries. The free allowance is 1,000 requests per month per organisation, as of October 2026, with Search counting as one request. For a self-managed rewriter, the earlier function’s output goes straight into a query such as hybrid search with a query string, as in any other search.

Either way, the rewrite is a new component that can fail, so it needs measurement like any other.

When does query rewriting hurt retrieval, and how do you measure it?

Query rewriting hurts retrieval when the rewrite drops or changes a key term, adds words the corpus lacks, or rewrites a query that was already good. The way to catch it is to score retrieval with and without the rewrite on the same labelled queries.

The test here shows both outcomes in one system. The same model that nearly doubled scores on follow-ups made 39 clear queries worse. Latency is the other cost: every rewrite adds a model call before search can start, and a slow hosted model can outweigh a fast index. Gating helps. Rewrite only when the message contains pronouns, is very long, or follows earlier turns, and search the original otherwise.

Measurement needs real queries with known relevant documents. Score nDCG@10 for the original, the rewrite and the merged lists, and look at the per-query wins and losses, not just the average. Save every rewrite so failures can be read, as this site’s test did. How to evaluate semantic search covers building that query set.

Recheck after any change of model, prompt or corpus. Rewriters drift with model updates, and a prompt tuned for one domain can damage another.

Query rewriting turns a user’s question into a query the index can use, most often by asking a language model for a standalone version. Its value is clearest in conversation. In this site’s test, rewriting 40 follow-ups lifted hybrid nDCG@10 from 0.392 to 0.736, close to searching with the full original claim. On clear queries, the same rewriter helped as often as it hurt, and merging the original with the rewrite was the safer choice. Rewrite when context is missing, keep every specific term, gate the rewriter so clear queries pass through, and measure the change on labelled queries before trusting it.