Home / Use cases / Semantic Search as the Retrieval Step in RAG

Semantic Search as the Retrieval Step in RAG

In retrieval-augmented generation (RAG), semantic search is the retrieval step: it finds the passages most relevant to a user’s question so a language model can answer from them instead of from memory alone. The pattern is simple: embed and index your documents, search them with the question, and pass the top results to the model with the question. Because the model can only use what retrieval hands it, retrieval quality caps answer quality. The CRAG benchmark shows the stakes: leading language models answered at most 34% of its questions correctly alone, and straightforward RAG raised that to only 44%. Weaviate runs retrieval and generation in one query. The first things to tune are chunking, the embedding model, hybrid search, and reranking.

What role does semantic search play in RAG?

Semantic search plays the role of the retriever in RAG, finding the documents whose meaning best matches the user’s question and supplying them as context for the language model. Weaviate describes RAG as a way to combine the retrieval capabilities of semantic search with the generation capabilities of large language models.

Language models know only what they learned in training, which can be outdated, general, or wrong about your private data. RAG fixes that by fetching relevant text at question time. The model then writes an answer grounded in that text, and can cite it. The benefits are fresher answers, answers about private data, and fewer invented facts.

Semantic search suits the job because users ask questions in their own words. A question like “can I get my money back after 30 days?” should find a policy section titled “Refund eligibility window,” even though the words differ. A semantic search example like that is exactly what semantic search is built for. The retrieval step has one clear purpose.

What is the primary objective of the retrieval step in RAG?

The primary objective of the retrieval step in RAG is to put the information needed to answer the question into the model’s context, with as little irrelevant text as possible. Both halves matter: coverage and focus.

Coverage means the right passage must be among the results. If it is missing, the model cannot use it and may guess instead. Focus means the results should not drown the answer in noise. Extra passages cost tokens, slow responses, and can distract the model.

Research shows why focus matters. Liu and colleagues, in “Lost in the Middle,” found that language models use information best when it appears at the start or end of a long context, and worse when it sits in the middle. Passing 50 loosely related chunks is not a safe substitute for retrieving the right five. The process that serves this objective has a familiar shape.

How does retrieval augmented generation work?

Retrieval augmented generation works in two phases: an indexing phase that embeds and stores your documents, and a query phase that retrieves relevant passages and feeds them to a language model with the question. The retrieved documents become part of the prompt.

1. Chunk: split documents into passages small enough to retrieve precisely.
2. Embed and index: turn each chunk into a vector and store it in a vector database.
3. Retrieve: embed the user’s question and search for the most relevant chunks.
4. Augment: place the retrieved chunks in the prompt, with instructions to answer from them.
5. Generate: the language model writes the answer, ideally citing the chunks it used.

The original RAG paper, by Lewis and colleagues at NeurIPS 2020, combined a sequence-to-sequence model with what it called non-parametric memory: a dense vector index of Wikipedia searched by a neural retriever. It set new results on open-domain question answering and produced more specific and factual text than the model alone. Today’s systems follow the same pattern with larger models and vector databases. The retriever, though, remains the bottleneck.

Why does retrieval quality cap RAG answer quality?

Retrieval quality caps RAG answer quality because the language model can only ground its answer in the passages it receives. If retrieval misses the answer, the model either says it does not know or invents one. Better generation cannot recover missing evidence.

The CRAG benchmark, a Comprehensive RAG Benchmark with 4,409 question-and-answer pairs, measures the gap. Its authors report that the most advanced language models answered at most 34% of questions correctly on their own. Adding RAG in a straightforward way raised accuracy only to 44%. State-of-the-art industry RAG systems answered 63% of questions without hallucination.

The pattern is consistent with practice: a basic retriever helps, and careful retrieval engineering helps much more. Most of that engineering happens before the model sees anything. The first decision is what to search.

Where does retrieval augmented generation pull information from?

Retrieval augmented generation pulls information from whatever sources the retriever can search, most often a vector database of document chunks, but also keyword indexes, SQL databases, APIs, and the web. The source shapes the answers.

• Vector databases: chunks of documents, help articles, contracts, or tickets, searched by meaning.
• Keyword indexes: BM25 search over the same text, strong on codes and names.
• Structured databases: rows from SQL tables for questions about orders, prices, or counts.
• APIs and the web: live data such as weather, stock levels, or current news.

Semantic search is the common default but not a requirement. Simon Willison made the point in a 2025 PyCon workshop: RAG does not have to use semantic search, and any search that finds relevant text can feed the model. Agentic systems route between several sources, as agentic retrieval explains. The choice of search method is the next question.

How is RAG different from semantic search and BM25 search?

RAG is different from semantic search and BM25 search because it is a full pattern that retrieves and then generates an answer, while semantic search and BM25 are retrieval methods that return ranked documents. RAG can use either one, or both.

In practice, the strongest RAG retrievers combine both search types. Hybrid search for RAG runs BM25 and vector search together, so a question that mentions an error code and describes a symptom matches on both. Semantic search versus RAG compares the two in depth. With the pieces clear, the question is where to spend effort.

What should you tune first in RAG retrieval?

You should tune RAG retrieval in roughly this order: chunking, the embedding model, the number of results, hybrid search, reranking, and filters. Each step fixes a different failure.

1. Chunking: chunks should hold one complete idea. Too large and they dilute the match; too small and they lose context. See chunking for semantic search.
2. Embedding model: the best semantic search model is the one that scores best on your own questions, not the top of a leaderboard, as choosing an embedding model explains.
3. Number of results (k): enough to cover the answer, few enough to keep the context focused.
4. Hybrid search: add BM25 so exact terms, names, and codes are not missed.
5. Reranking: retrieve more candidates, then let a cross-encoder reorder them, as in reranking in RAG.
6. Filters: restrict by date, product, or permission so the model never sees outdated or forbidden text.

Weaviate’s evaluation guidance adds index-level knobs. Compression settings can trade recall for memory, and for corpora under about 10,000 vectors a brute-force index may be enough. Measure after each change rather than changing everything at once.

The order reflects cost and impact. Chunking and model choice shape every later step and are cheap to test on a sample. Hybrid search and reranking add moving parts, so they pay off once the basics are sound. Filters are often a correctness fix rather than a quality tweak, especially where permissions matter. Some systems go further and let the model judge its own retrieval.

What is corrective retrieval augmented generation?

Corrective retrieval augmented generation, or Corrective RAG, is a method that checks the quality of retrieved documents before generation and takes corrective action, such as a web search, when retrieval looks weak. It was proposed by Yan and colleagues in 2024.

Corrective RAG adds a lightweight retrieval evaluator that scores how well the retrieved documents fit the query. Depending on its confidence, the system keeps the documents, discards them and searches the web, or combines both. A decompose-then-recompose step then strips irrelevant parts of the documents before the model sees them.

The abbreviation causes confusion. Corrective RAG is often shortened to CRAG, but the CRAG benchmark cited above is a different paper: the Comprehensive RAG Benchmark, published in the NeurIPS 2024 Datasets and Benchmarks track. One is a method for fixing retrieval; the other is a test for measuring it. Both point to the same lesson. Reranking and agentic retrieval are related ways to judge evidence. Building the basic pattern comes first.

How do you build RAG on semantic search in Weaviate?

You build RAG on semantic search in Weaviate by configuring a vectorizer and a generative model on a collection, then calling generate.near_text with a prompt. Weaviate runs the search and the generation in one request.

import weaviate
from weaviate.classes.config import Configure, Property, DataType

client = weaviate.connect_to_local()

docs = client.collections.create(
    name="HelpArticle",
    properties=[
        Property(name="title", data_type=DataType.TEXT),
        Property(name="body", data_type=DataType.TEXT),
    ],
    vector_config=Configure.Vectors.text2vec_weaviate(),
    generative_config=Configure.Generative.openai(),
)
# ... import chunked articles with docs.data.insert_many([...]) ...

response = docs.generate.near_text(
    query="Can I get a refund after 30 days?",
    limit=5,
    grouped_task="Answer the question using only these articles. Cite the titles you used.",
    grouped_properties=["title", "body"],
)
print(response.generative.text)          # the grounded answer
for obj in response.objects:
    print("-", obj.properties["title"])  # the retrieved sources

client.close()

A grouped_task produces one answer from all retrieved objects, which is the usual RAG case. A single_prompt instead runs the model once per result, useful for summarizing or translating each item. Weaviate’s Academy notes that grouped_properties limits which fields go into the prompt, keeping it short. From version 1.30, the generative provider can also be set per query. Any of the earlier search options, such as hybrid or filters, can replace near_text. The last step is checking that it works.

How do you evaluate RAG retrieval and reduce hallucination?

You evaluate RAG retrieval by measuring whether the right passages are retrieved for a set of test questions, and reduce hallucination by grounding answers in those passages and checking that each claim is supported. Retrieval and generation should be measured separately.

For retrieval, build a set of real questions with the passages that answer them. Measure how often the right passage appears in the top k, using metrics such as recall at k and nDCG. If a passage is never retrieved, no prompt change will fix the answer.

For generation, check faithfulness: whether each statement in the answer is supported by the retrieved text. Grounding techniques help. Instruct the model to answer only from the provided passages, to cite them, and to say when the answer is not there. Filter out low-scoring results with a distance threshold, so weak context never reaches the model.

A small labeled set of 50 to 100 questions is enough to start. Re-run it after every change to chunking, models, or retrieval settings. How to evaluate semantic search and semantic search use cases go further.

In RAG, semantic search is the retrieval step that decides what the language model can use, so retrieval quality caps answer quality. The pattern, introduced by Lewis and colleagues in 2020, embeds documents, retrieves the passages closest to a question, and generates a grounded answer. Benchmarks such as CRAG show that simple RAG helps only modestly, while careful retrieval helps much more. Weaviate combines search and generation in one query. The practical step is to tune chunking and the embedding model first, add hybrid search and reranking, and measure retrieval on real questions before adjusting prompts.