Home / Hybrid search / Hybrid Search for RAG: Better Retrieval for LLM Answers

Hybrid Search for RAG: Better Retrieval for LLM Answers

Hybrid search in RAG retrieves context for a language model by running keyword (BM25) search and vector search on the same question and fusing the results. The prompt then gets passages that match both the meaning and the exact terms. Vector-only retrieval handles paraphrased questions well but can miss chunks that hinge on a name, an error code, a config key, or a version number. A missed chunk cannot be fixed later, because the model never sees it. In this site’s 315-chunk test, vector retrieval failed to return two error-code answers anywhere in its top 20, and a reranker could not recover them. Hybrid retrieval followed by reranking put the right chunk in the top 3 for all 24 questions.

What is hybrid search in RAG?

Hybrid search in RAG is the retrieval step of retrieval-augmented generation done with two searches at once, keyword and vector, whose ranked lists are fused before the top passages go into the prompt. It replaces vector-only retrieval, the default in many RAG tutorials.

RAG itself has two halves. The retrieval half searches a knowledge base for passages relevant to the user’s question. The generation half sends those passages, with the question, to a language model that writes the answer. A RAG search is the retrieval half: a search whose results are read by a model rather than a person.

Hybrid retrieval changes only how candidates are found. Weaviate’s documentation describes hybrid search as running vector and BM25 searches in parallel, normalizing their scores, and combining them with a fusion method. The weight between the two sides is a parameter called alpha. What is hybrid search covers the general mechanism. The RAG case changes because a model, not a person, reads the results.

Three things are different in RAG. The model sees only the top few chunks, often three to five, so recall at that small depth matters more than a full ranked list. Questions are long and conversational, which helps vector search and adds noise for keyword search. And the cost of a miss is high: a model given the wrong context may answer confidently and wrongly.

Why does vector-only retrieval fail RAG on names and codes?

Vector-only retrieval fails RAG on names and codes because an embedding model compresses a passage into its general meaning, and passages that differ only in an identifier get nearly identical vectors. The model knows a passage is about a gateway error, but not reliably which one.

Anthropic’s contextual retrieval post gives the standard example. A user asks about “Error code TS-999” in a support database. An embedding model may find content about error codes in general and miss the exact TS-999 match. BM25 matches the literal string, which is why the post combines both.

Weaviate’s advanced RAG guide makes the same point for medicine. Medication names, anatomical terms, and diagnoses behave like keywords rather than concepts, and pure vector search might miss them. Its guide lists hybrid search first among techniques for raising retrieval recall, before query rewriting and fine-tuned embedding models.

The failure grows with the knowledge base. A small corpus with one error code per topic gives the embedding enough to work with. A real one, with hundreds of error codes, dozens of release notes, and many config keys written from the same templates, does not. Our test was built to show that difference.

How much does hybrid retrieval help a RAG knowledge base?

Hybrid retrieval helped little on a small, varied knowledge base and decisively once look-alike chunks were added, in our test of 24 RAG questions. We ran the same questions against two versions of a fictional company knowledge base.

The first version had 48 hand-written chunks: gateway runbooks, config reference, release notes, team ownership, and HR policy. We wrote 24 questions with one answer chunk each. Twelve contained a rare exact term, such as GW-471, max_idle_time, or “Kestrel team”. Twelve were paraphrased, such as “how long can I work abroad”.

Keyword search used BM25 with Weaviate’s default k1 of 1.2 and b of 0.75. Vector search used the open all-MiniLM-L6-v2 model. Hybrid runs fused each side’s top 20, with relative score fusion or reciprocal rank fusion. A reranking run passed the top 20 through the ms-marco-MiniLM-L-6-v2 cross-encoder.

On the 48-chunk version, vector retrieval put every answer chunk in the top 3, for all 24 questions. Hybrid search added nothing there, and the equal-weight and rank-based hybrids lost one or two paraphrased questions to keyword noise. A small, distinct corpus does not need hybrid search.

The second version added 267 generated look-alike chunks: 200 more gateway error codes, 47 more release notes, and 20 more config keys, each written from the same templates. That made 315 chunks, closer to a real knowledge base. The original 48 chunks and 24 questions stayed the same.

With look-alikes present, vector retrieval lost two rare-term questions. For “what does GW-471 mean” and “what is GW-714”, the answer chunk fell out of the vector top 20 entirely, crowded out by near-identical error passages. BM25 found both first. Hybrid at Weaviate’s default alpha of 0.75 ranked them 9th and 11th, outside a five-chunk prompt.

The test is hand-written and small, so read it as a demonstration of the mechanism rather than a benchmark. The direction matches published results, which show the same pattern at larger scale.

Can a reranker fix vector-only retrieval?

A reranker cannot fix vector-only retrieval when the right chunk is missing from the candidates, because it only reorders what the first stage returned. It improves order, not coverage.

In our test, reranking the vector top 20 still got 22 of 24. The two error-code chunks were not among the 20 candidates, so the cross-encoder never saw them. Reranking the hybrid top 20 got 24 of 24: hybrid retrieval brought the chunks into the candidate set, and the reranker fixed the paraphrase losses that keyword noise had caused.

Weaviate Academy states the constraint plainly. The re-ranker scores exactly the results your limit retrieves and performs no second retrieval. To let it choose from more candidates, raise the limit and take the top results yourself. That makes the first stage’s job recall and the reranker’s job precision.

The two stages therefore work best together. Hybrid retrieval widens the net so the right chunk is somewhere in the top 20. The reranker reads each candidate against the question and moves the right one into the top 3. Bi-encoder versus cross-encoder explains why the reranker can judge relevance more precisely.

What do published results show about hybrid retrieval for RAG?

Published results show hybrid retrieval reducing RAG retrieval failures, with the largest gains when it is combined with reranking. The clearest numbers come from Anthropic and from OpenSearch.

Anthropic’s contextual retrieval study, published in September 2024, measured the share of relevant chunks missing from the top 20, across codebases, fiction, ArXiv papers, and science papers. Standard embeddings missed 5.7 percent. Contextual embeddings, which prepend a short description of each chunk’s document, cut that to 3.7 percent. Adding contextual BM25 cut it to 2.9 percent, and adding reranking cut it to 1.9 percent.

OpenSearch’s BEIR benchmark points the same way for retrieval in general. Combining the TAS-B dense model with BM25 raised nDCG@10 on 9 of 10 datasets compared with the dense model alone. Hybrid search versus vector search covers that benchmark and a larger identifier test.

Weaviate’s own benchmarks use hybrid search as the baseline. Its Search Mode study compares an agentic retriever against hybrid search with Snowflake Arctic embeddings, BM25, and reciprocal rank fusion. The choice of hybrid as the reference reflects how standard it has become for RAG retrieval.

What does a reference hybrid RAG setup look like?

A reference hybrid RAG setup retrieves about 20 candidates with filtered hybrid search, reranks them with a cross-encoder, and passes the top three to five chunks to the model. Each number is a starting point to tune.

Filters matter more in RAG than in search, because a model will quote whatever it receives. Access rights and tenant boundaries belong inside the retrieval query, so forbidden chunks never become candidates. Version filters stop the model mixing instructions from different releases. Permission-aware search covers access control.

The cost of the reranking stage is real but bounded. In this site’s timing test, a cross-encoder scored 20 candidates in 42.5 ms on a laptop CPU. That is small next to a language model call. Semantic search architecture breaks down the full latency budget.

How should you set alpha and fusion for RAG?

You should set alpha and fusion for RAG by testing a few values on labeled questions from your own users, because the best balance depends on how often they use exact terms. The 315-chunk run illustrates why no default is safe.

Weaviate’s server default alpha is 0.75, leaning toward vector search. In our 315-chunk test that weight left two error-code chunks at ranks 9 and 11. An alpha of 0.5 brought every rare-term chunk into the top 3 but cost one paraphrased question. Reciprocal rank fusion found all rare-term chunks but cost two paraphrased questions.

Reranking changed the picture. Once a cross-encoder reordered the hybrid top 20, every question reached the top 3, so the fusion weight mattered much less. With a reranker in place, tune alpha for recall at 20 rather than order at 3. Without one, tune for the depth your prompt actually uses.

Weaviate’s documentation recommends setting alpha explicitly, since client libraries do not all leave it unset and the effective weight can differ from 0.75. Balancing keyword and vector scores covers tuning, and reciprocal rank fusion explains the rank-based method.

What is BM25 search in RAG?

BM25 search in RAG is the keyword half of hybrid retrieval: it scores each chunk by how often the question’s terms appear in it, weighted by how rare each term is across the knowledge base. Rare terms, like error codes, get the most weight.

That weighting is why BM25 helps RAG. A token such as “471” or “kestrel” appears in one chunk, so it carries a high inverse document frequency and dominates the score. Common words carry little weight, and Weaviate removes English stopwords from the query by default. Weaviate uses BM25F, a variant that scores several fields, such as title and body, with separate weights.

RAG questions bring one complication: they are long. A question like “how many people must approve a change to login code” contains several ordinary words that match many chunks. In our test, BM25 alone put the answer for that question third, behind chunks that shared incidental words. Weaviate’s BM25 operator can require a minimum number of matching tokens, which trims those weak matches.

from weaviate.classes.query import BM25Operator

response = docs.query.hybrid(
    query="how many people must approve a change to login code",
    alpha=0.5,
    bm25_operator=BM25Operator.or_(minimum_match=2),   # keyword side needs 2+ matching tokens
    limit=20,
)

The operator, added in Weaviate 1.31, affects only the keyword side of a hybrid query; the vector side still searches everything. The BM25 algorithm explains the scoring formula.

How does chunking affect hybrid retrieval for RAG?

Chunking affects hybrid retrieval because both searches score chunks, not documents, so a chunk that loses its identifiers or context becomes hard for either side to find. The split decides what each chunk can match.

A chunk cut from the middle of a runbook may say “raise the timeout or fix the slow service” without the error code it refers to. Neither BM25 nor the embedding can link it to a question about that code. Keeping headings and identifiers with each chunk, or prepending a short summary of the parent document, fixes this. Anthropic’s contextual retrieval applies that idea to both the embeddings and the BM25 index.

Chunk length also interacts with BM25. Its b parameter, 0.75 by default, penalizes long passages, so very uneven chunk sizes can bias keyword scores. Consistent chunk sizes keep the comparison fair. Chunking for semantic search covers strategies and sizes.

Does hybrid retrieval handle follow-up questions in a chat?

Hybrid retrieval handles follow-up questions in a chat only after the follow-up is rewritten into a standalone question, because both searches see just the text they are given. A follow-up like “and what about 714?” carries almost nothing to match.

In a conversation, users lean on context. After asking what GW-471 means, they type “and 714?” or “how do I fix that one?”. The vector side embeds a fragment with no subject, and the keyword side sees one number that also appears in release notes, config defaults, and other error codes. Neither search can recover the missing words, so retrieval returns noise and the model answers from the wrong chunks.

The standard fix is query rewriting. Before retrieval, a language model turns the latest message and the recent history into a full question. For example, it rewrites “and 714?” as what does error GW-714 mean. Weaviate’s advanced RAG guide lists query rewriting next to hybrid search as a way to raise retrieval recall, and the two work well together. Rewriting restores the exact terms, and the keyword side of hybrid search then matches them.

Rewriting has a cost and a risk. It adds a model call before every retrieval, and a careless rewrite can drop or change an identifier. Keep exact strings verbatim in the rewrite prompt, log rewritten queries beside the originals, and include follow-up questions in the labeled test set. A test set of only standalone questions will overstate how well a chat assistant retrieves.

Filters carry over the same way. If the user earlier said they run version 4.21, the rewrite step or the application should keep that version filter for later turns, so the follow-up does not retrieve instructions for a different release.

How do you build hybrid RAG in Weaviate?

You build hybrid RAG in Weaviate by running a hybrid query with a reranker and a generative task in one call, so retrieval, reranking, and generation happen together. The Python client’s generate.hybrid method takes the hybrid parameters plus a prompt.

import weaviate
from weaviate.classes.config import Configure, Property, DataType
from weaviate.classes.query import Filter, HybridFusion, Rerank

client = weaviate.connect_to_local()
client.collections.create(
    "KnowledgeChunk",
    vector_config=Configure.Vectors.text2vec_openai(source_properties=["text"]),
    generative_config=Configure.Generative.openai(),
    reranker_config=Configure.Reranker.cohere(),
    properties=[
        Property(name="text", data_type=DataType.TEXT),
        Property(name="source", data_type=DataType.TEXT),
        Property(name="product_version", data_type=DataType.TEXT),
    ],
)
chunks = client.collections.use("KnowledgeChunk")

question = "What does GW-471 mean?"
response = chunks.generate.hybrid(
    query=question,
    alpha=0.5,
    fusion_type=HybridFusion.RELATIVE_SCORE,
    filters=Filter.by_property("product_version").equal("4.21"),
    limit=20,                                   # candidates the reranker sees
    rerank=Rerank(prop="text", query=question),
    grouped_task=f"Answer using only these passages: {question}",
    grouped_properties=["text"],
)
print(response.generated)
client.close()

The reranker reorders the 20 candidates, and the grouped task sends them to the model. To send only the top few after reranking, run query.hybrid with the reranker, keep the top results, and call your model yourself. The reranker here needs a Cohere API key at connection time. Semantic search in RAG covers the generation side.

LangChain’s Weaviate integration also exposes hybrid search, with alpha and a reranker, so the same pattern fits a LangChain pipeline. Semantic search with LangChain and LlamaIndex shows the framework route.

How do you evaluate hybrid retrieval for RAG?

You evaluate hybrid retrieval for RAG by measuring context recall at the number of chunks your prompt uses, on labeled questions, split by question type. End-to-end answer scores alone hide whether retrieval or generation failed.

Start with 50 to 200 real questions from users or support logs. Mark the chunk or chunks that contain each answer. Tag each question as containing a rare exact term or not. Then measure, for each retrieval setting, how often the answer chunk lands in the top 3 or top 5, and how often it lands in the top 20 the reranker sees.

The split by question type is what makes the result useful. In our 48-chunk test, overall numbers suggested vector search was enough. Only the 315-chunk run, with realistic look-alikes, exposed the error-code misses. Weaviate’s RAG evaluation guidance lists retrieval metrics such as recall and mean reciprocal rank beside generation metrics such as faithfulness. How to evaluate semantic search and recall at k cover the methods.

What is the difference between semantic search and hybrid search in RAG?

The difference between semantic search and hybrid search in RAG is that semantic retrieval ranks chunks by embedding similarity alone, while hybrid retrieval also ranks them by exact keyword matches and fuses the two. Hybrid retrieval is semantic retrieval with a keyword safety net.

Semantic retrieval is enough when questions and documents use everyday language and the knowledge base has few look-alike passages. Hybrid retrieval earns its cost when questions contain codes, product names, people, versions, or jargon. Weaviate Academy’s guidance for user-typed, unpredictable queries is to start with hybrid search for that reason.

Hybrid search is also different from hybrid RAG in the broader sense, which sometimes means combining vector retrieval with a knowledge graph or a SQL database. That is a different design. Semantic search versus vector search untangles the related terms.

When is hybrid retrieval unnecessary for RAG?

Hybrid retrieval is unnecessary for RAG when the knowledge base is small and distinct, questions rarely contain exact identifiers, and an embedding model fine-tuned on the domain already retrieves well. Then the extra tuning buys little.

Our 48-chunk run is an example: vector retrieval already reached every answer, and badly weighted hybrids only added noise. Weaviate’s web developer guide makes the same point: with a model fine-tuned for your data, pure vector search might be good enough. Hybrid search also costs a second search per query, though Weaviate runs the two in parallel.

The safe path is to measure rather than assume. Run the labeled questions through vector-only and hybrid retrieval, with and without a reranker, and keep the simplest setup that reaches your recall target. Re-run the test as the knowledge base grows, because look-alike content accumulates over time.

Hybrid search for RAG pairs BM25’s exact matching with vector search’s grasp of meaning, so the chunks a model reads match both the wording and the intent of the question. The two test runs showed when that matters: not on a small, distinct knowledge base, but decisively once hundreds of look-alike codes and releases crowd the vector results. Retrieve about 20 candidates with filtered hybrid search, rerank them, pass the top few to the model, and check recall on your own questions.