Home / Retrieval models / Agentic Retrieval: When an Agent Decides How to Search

Agentic Retrieval: When an Agent Decides How to Search

Agentic retrieval is a retrieval pattern in which a language model agent controls the search process, deciding whether to search, where to search, what query to send, and whether the results are good enough. Standard RAG retrieves once and then generates an answer. Agentic retrieval turns that single step into a loop: plan, search, read, judge, and search again if needed. Weaviate describes it as retrieval that can retrieve, evaluate, re-retrieve, and validate context before the final answer. The main techniques are query decomposition, routing across sources, query rewriting, and evidence grading. Research such as IRCoT shows real gains on multi-step questions, with retrieval up to 21 points better. The price is more model calls, more tokens, and more latency, so a fixed pipeline is still the better choice for simple lookups.

What is agentic retrieval?

Agentic retrieval is the use of an AI agent, a language model with tools and a plan, to run the retrieval step of a search or RAG system instead of a fixed query. The agent treats search as a task to reason about, not a single function call.

Weaviate’s explainer on agentic RAG lists four decisions the retrieval agent can make. It can decide whether to retrieve at all. It can decide which tool to use. It can formulate the query itself. And it can evaluate the retrieved context and decide whether to retrieve again. A standard pipeline makes none of these decisions; it runs the same steps every time.

The tools vary. Weaviate names a vector search engine, web search, a calculator, and any API, such as email or chat programs. A single question might touch several of them. “What did our top customer order last quarter, and is that product still in stock?” needs a database query, a lookup in another system, and a way to join the answers.

The term is close to agentic RAG, and the two are often used as synonyms. Agentic RAG describes the whole system, retrieval plus generation, with agents inside it. Agentic retrieval names the part that changed: the retrieval step is now driven by an agent. Weaviate notes that agentic RAG most commonly means agents in the retrieval component, so in practice the two point at the same idea. The contrast with standard RAG makes the change concrete.

How does agentic retrieval differ from standard RAG?

Agentic retrieval differs from standard RAG in that it can use several tools, rewrite the query, retrieve in several steps, and check the evidence, while standard RAG retrieves once from one source. Weaviate summarizes the difference in four rows.

Weaviate’s article frames the two limits of standard RAG plainly. A naive pipeline considers only one knowledge source, and it is a one-shot solution with no reasoning about the quality of what it retrieved. Weaviate’s analogy compares standard RAG to a library visit before smartphones, and agentic RAG to having a phone with a browser, calculator, and email.

A concrete failure shows why validation matters. Weaviate’s legal RAG example asks for notice periods in the company’s 2024 service agreements. A plain retriever may return 2022 clauses, because their text is semantically similar. Without a reasoning step, nothing applies the date filter. An agent can see the year in the question and build the filter before searching, as metadata filtering explains.

The core of standard retrieval does not disappear. The agent still calls a vector or hybrid search underneath; it simply decides how and when to call it. That control runs as a loop.

How does the agentic retrieval loop work?

The agentic retrieval loop works by repeating five steps: plan the query, route it to a source, retrieve and read, grade the evidence, then answer or search again. The loop ends when the agent judges the evidence sufficient or hits a limit.

1. Plan: the agent reads the question and decides what information it needs, often splitting it into parts.
2. Route: for each part, it picks a source, such as a vector collection, a SQL table, or the web.
3. Retrieve: it sends a query, with filters if needed, and reads the results.
4. Grade: it checks whether the results are relevant and complete for the question.
5. Decide: it either writes the answer or goes back to plan a better search.

The loop descends from ReAct, a method introduced by Yao and colleagues and published at ICLR 2023. ReAct interleaves reasoning traces with actions, such as calls to a Wikipedia API. Weaviate describes the cycle as thought, action, and observation, repeated until the task is done. Each “thought” is the model’s reasoning about what to do next. Each “action” is a tool call, and each “observation” is the result fed back to the model.

Modern language models make the loop practical through function calling. The developer describes each tool, such as a search function, in a schema. The model returns a structured request to call a tool, the application runs it, and the result goes back into the conversation. Weaviate’s tutorial shows this with a hybrid search function exposed as a tool. The steps inside the loop correspond to a handful of named techniques.

What are the main agentic retrieval techniques?

The main agentic retrieval techniques are query decomposition, routing, query rewriting, structured query construction, evidence grading, and iterative multi-hop retrieval. Most systems combine several of them in one loop.

None of these techniques is new on its own. Query rewriting, query expansion, and multi-query retrieval already exist in fixed pipelines. What makes them agentic is that the model chooses when to apply them, based on the question and on what earlier searches returned.

Weaviate’s Query Agent shows several working together. It inspects the available collections, decides whether a question needs a search or an aggregation, writes one or more queries with filters, and reranks the results. Asked how many posts an author has published, it runs a count. Asked about a topic in one article, it filters to that article and searches. The first technique, decomposition, does the most for complex questions.

How does query decomposition help with multi-part questions?

Query decomposition helps with multi-part questions by splitting them into simple sub-questions, retrieving evidence for each one, and combining the answers. A single embedding of a compound question tends to match documents about only one of its parts.

Weaviate’s evaluation article gives a small example. To answer “What is Ref2Vec in Weaviate?” a system needs to know what Ref2Vec is and what Weaviate is. A sub-question engine makes two calls to the database, one per part, then combines the two answers into one response.

Comparison questions benefit most. “How do the refund policies of our three vendors differ?” becomes three searches, one per vendor, followed by a comparison. A single search might return three passages about the same vendor and miss the others. Decomposition guarantees each part gets its own retrieval.

Decomposition can also fail. If the agent splits the question badly, every later step builds on the wrong plan. Weaviate recommends checking the sub-questions directly: does the decomposition make sense for the original question? Query decomposition covers the method in more detail. Once a question is split, each part needs a destination.

How does an agent choose which source to search?

An agent chooses which source to search by reading descriptions of the available tools or collections and matching them to what each sub-question needs. Weaviate calls the simplest form of agentic RAG a router.

A router needs at least two knowledge sources. One might be a vector collection of support articles, another a SQL table of orders, a third a web search tool. The agent sees each tool’s name and description, often with its schema, and picks the one most likely to hold the answer. Weaviate notes the sources need not be databases; they can be APIs for Slack channels or email.

Good descriptions decide good routing. A tool described as “search” gives the model little to go on. A tool described as “search product manuals by model number and topic” lets the model route accurately. The Weaviate Query Agent reads collection schemas for this reason, so it knows which properties exist and what filters make sense.

Routing also covers the type of operation, not just the source. Weaviate’s Query Agent decides between semantic search and aggregation for each question. Other systems route between a vector database and SQL, a pattern Weaviate’s evaluation guide describes as a routing query engine. When internal data is not enough, web search is one more route, which is how research assistants extend their reach. After routing and retrieving, the agent must judge what came back.

How do agents grade evidence and decide to search again?

Agents grade evidence by asking a model, or a smaller evaluator, whether the retrieved passages are relevant and sufficient, then rewrite the query and search again when they are not. This check is what separates agentic retrieval from a pipeline that trusts its first result.

Research has formalized two versions of this step. Self-RAG, from Asai and colleagues, trains a model to emit reflection tokens. These tokens let the model retrieve on demand rather than always, and critique both the passages and its own output. Corrective RAG, or CRAG, from Yan and colleagues, uses a lightweight retrieval evaluator. The evaluator returns a confidence score that triggers different actions, including a web search when local results look poor.

In everyday systems, grading is usually a prompt. The agent asks itself whether the passages answer the question, which parts are missing, and what to search next. If the first search returns documents about the wrong product version, the agent adds the version to the query and tries again. If nothing relevant exists, a good agent says so rather than answering from weak context.

Weaviate’s Query Agent exposes this as a setting. With result evaluation turned on, an extra model step keeps only the sources used in the answer. It also reports whether the answer is partial and what information is missing. The setting raises cost and latency, which is the trade-off behind every grading step. Reranking in RAG covers a cheaper, model-based way to sort evidence. Research gives a sense of how much these loops help.

What does research show about agentic retrieval?

Research shows that interleaving retrieval with reasoning improves answers on multi-step questions, with gains of up to 21 points in retrieval and 15 points in question answering in one study. The gains are largest where a question needs several facts in sequence.

• ReAct (Yao and colleagues, ICLR 2023) showed that alternating reasoning with actions, such as Wikipedia lookups, gives a model a way to gather facts while it thinks.
• IRCoT (Trivedi and colleagues) interleaved retrieval with chain-of-thought reasoning. On HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC, it improved retrieval by up to 21 points and question answering by up to 15 points.
• Self-RAG (Asai and colleagues) let 7-billion and 13-billion parameter models retrieve on demand and critique themselves. The authors report it outperformed ChatGPT on open-domain question answering and fact verification.
• CRAG (Yan and colleagues) added a retrieval evaluator and web search fallback, designed to plug into existing RAG systems.

Industry results point the same way for research-style tasks. Anthropic reported that its multi-agent research system outperformed a single Claude Opus 4 agent by 90.2 percent on an internal research evaluation. It also found that token usage alone explained 80 percent of the performance variance in one browsing evaluation. Much of the benefit, in other words, comes from spending more effort on search.

These results come from multi-hop benchmarks and open research tasks. They do not show that agents help on single-fact lookups, where one good search already works. That distinction shapes which architecture fits.

Which architectures are used for agentic retrieval?

The architectures used for agentic retrieval range from a single agent acting as a router to multi-agent systems where a lead agent coordinates specialized retrieval agents. Weaviate describes both as the two basic forms.

A single-agent router holds the reasoning, retrieval, and answer generation in one loop. It is the natural starting point and covers most business search needs. Its limit is that one agent must handle every source and every step in a single context window.

A multi-agent system splits the work. Weaviate’s example has a master agent coordinating three specialists: one for internal data, one for personal accounts such as email or chat, and one for the web. Anthropic’s research system follows the same shape, with a lead agent spawning sub-agents that search in parallel. Anthropic notes the pattern fits tasks with heavy parallel work and information that exceeds one context window. It fits poorly where agents must share context or depend on each other.

Frameworks package these patterns. Weaviate’s article names LlamaIndex, LangGraph, and CrewAI, and LangChain offers agent tooling too. Weaviate’s own open-source Elysia is a decision-tree agent that chooses tools and decides when its goal is met. The same agent ideas also power products people call agentic search.

How does memory fit into agentic retrieval?

Memory fits into agentic retrieval as the store the agent searches and writes to across steps and conversations, so retrieval doubles as the agent’s long-term memory. Weaviate lists memory, short-term and long-term, as one of the four core parts of an agent.

Short-term memory holds what happened inside one task. It includes the sub-questions, the searches already run, and the evidence gathered so far. Without it, an agent would repeat the same search or lose track of which parts of a question it has answered. In most systems, short-term memory is simply the running conversation that the model sees on each step.

Long-term memory persists between tasks. Weaviate notes that retrieved results can act as a form of long-term memory, letting agents recall relevant information across steps. A support agent might store the outcome of each resolved ticket, then search those records the next time a similar problem appears. The memory store is usually a vector collection, searched the same way as any other source.

Weaviate offers pieces for both kinds. The Query Agent accepts a list of chat messages instead of a single string, so follow-up questions keep their context. Engram, a managed service, gives agents persistent and searchable memory across conversations and users. Elysia, Weaviate’s open-source agent, chunks documents at query time and caches the chunks, so documents an agent reads often become faster to use later.

Memory brings its own risks. A wrong fact stored once can be retrieved again and again. Weaviate’s guidance on multi-agent systems applies here too: context that fails a relevance or freshness check should be flagged rather than passed forward. Memory also raises privacy questions, since stored conversations may hold personal data. Products built on these ideas often go by a different name.

What is agentic search?

Agentic search is search in which an AI agent plans and runs multiple searches on a user’s behalf, then synthesizes the results; agentic retrieval is the same idea applied inside an application’s retrieval step. The terms overlap heavily, and the difference is mostly about audience.

Agentic search usually refers to user-facing products. A person asks a broad question, and the system searches the web or a document set many times, reads the results, and writes a report. Weaviate’s article on agentic workflows describes these “deep research” assistants. It notes that OpenAI, Perplexity, and Google have each released their own version. What sets them apart is synthesis: they analyze and combine information rather than return a ranked list.

An agentic search engine therefore behaves differently from a classic search engine. A classic engine returns links for one query. An agentic engine breaks the question into several queries, follows promising results, and produces an answer with sources. It is slower and more expensive per question, but it can handle questions that would take a person many searches.

Agentic retrieval is the developer’s view of the same mechanism. Inside a RAG application, the agent’s searches feed a language model’s answer rather than a report for a reader. In agentic search for RAG, the retrieval loop simply replaces the single search step. Semantic search versus RAG and semantic search in RAG cover the underlying pipeline. Building one is simpler than it sounds.

How do you build agentic retrieval with Weaviate?

You build agentic retrieval with Weaviate either by using the Query Agent, a managed agent that plans searches over your collections, or by exposing Weaviate searches as tools to your own agent. The first is faster to start; the second gives full control.

The Query Agent runs on Weaviate Cloud. It decides which collections to search, which filters and sorts to apply, and which search types to use. Ask mode returns a written answer; Search mode returns the matching objects for your own pipeline.

import os
import weaviate
from weaviate.classes.init import Auth
from weaviate.classes.query import Filter
from weaviate.agents.query import QueryAgent
from weaviate.agents.classes import QueryAgentCollectionConfig

client = weaviate.connect_to_weaviate_cloud(
    cluster_url=os.environ["WEAVIATE_URL"],
    auth_credentials=Auth.api_key(os.environ["WEAVIATE_API_KEY"]),
)

qa = QueryAgent(
    client=client,
    collections=[
        QueryAgentCollectionConfig(
            name="Contracts",
            # always combined with the agent's own filters (AND)
            additional_filters=Filter.by_property("status").equal("active"),
        ),
        "SupportArticles",
    ],
)

response = qa.ask("What notice periods apply in our 2024 service agreements?")
response.display()   # shows searches, filters, sources, and the answer

client.close()

The additional_filters setting matters for safety. Weaviate combines it with any filter the agent generates using a logical AND, so the agent can narrow results but never escape the rule. That is a simple guardrail for multi-tenant data or access control.

For self-hosted Weaviate, the usual route is function calling. Weaviate’s agentic RAG tutorial wraps a hybrid query in a Python function, describes it in a tool schema, and lets the model call it in a loop. Each collection, or each search type, becomes one tool. The agent framework handles the loop, and Weaviate handles fast retrieval with filters. Whichever route you choose, the loop has a cost.

What does agentic retrieval cost in tokens and latency?

Agentic retrieval costs several times more tokens and seconds than a single search, because every plan, grade, and retry is another language model call. The extra cost buys better answers on hard questions and adds little on easy ones.

Anthropic’s engineering team published rough multipliers from its research system. Agents typically use about 4 times more tokens than chat interactions. Multi-agent systems use about 15 times more. Those multipliers apply to every question the system answers, so they matter at scale.

A single Weaviate example makes the latency concrete. In the Query Agent launch post, one shopping question produced a response that made 3 model requests and used 9,177 tokens in total. It took just under 14 seconds. A plain vector search over the same data returns in milliseconds. The agent’s answer was richer, with a price filter it built itself, but the user waited far longer.

Three controls keep cost in check. First, cap the number of steps or tool calls per question. Second, use smaller models for routing and grading, as Weaviate’s Elysia does by sending simple decisions to lightweight models. Third, expose effort as a setting. The Query Agent offers medium, high, and ultrahigh effort, trading quality for latency and cost. For testing, its free tier covers up to 250 Ask queries or 1,000 Search queries per month, enough to measure the loop on real questions before paying for scale. The cost question leads directly to when the loop is worth it.

When should you use agentic retrieval instead of a fixed pipeline?

You should use agentic retrieval when questions are multi-part, span several sources, or need filters and reasoning a fixed query cannot express; for simple lookups, a fixed pipeline is faster and cheaper. Weaviate warns that agents add unnecessary complexity to simple tasks.

Weaviate’s article on agentic workflows suggests a few questions to ask first. Is the task complex enough to need adaptive decisions, or would a deterministic approach work? Would RAG without an agent achieve the same result? What risks come with giving the agent autonomy, and can they be reduced?

A middle path often works well. Keep a fixed pipeline for most traffic and send only complex questions to an agent. A lightweight classifier, as in query intent classification, can make that call. Whatever the design, it needs to be measured.

How do you evaluate an agentic retrieval system?

You evaluate an agentic retrieval system by checking each decision in its trajectory, the plan, the routes, the retrieved evidence, and the final answer, rather than only the final answer. An answer can look right while resting on a wrong step.

Weaviate’s RAG evaluation guide suggests evaluating each stage separately. For decomposition, ask whether the sub-questions make sense for the original question. Then evaluate retrieval for each sub-question, and finally check whether the answers were combined correctly. For routing, ask whether the router picked the right source for the query.

Multi-agent systems need extra care. Weaviate’s article on retrieval quality describes how one agent’s bad retrieval propagates: a summary agent compresses the flawed content, and a reasoning agent treats it as fact. The final output gives no sign of where the error began. Weaviate recommends that every agent that retrieves context validates it before passing it on.

A practical evaluation set mixes question types: single-fact questions, multi-part questions, questions needing filters, and questions with no answer in the data. Track answer quality, but also the number of steps, tokens, and seconds per question. Standard retrieval metrics still apply to each search inside the loop, as how to evaluate semantic search describes. Related retrieval topics include retrieval models, dense retrieval, sparse retrieval, bi-encoders versus cross-encoders, ColBERT and late interaction, and multi-vector search.

Agentic retrieval puts a language model agent in charge of search: it plans, routes, retrieves, grades the evidence, and searches again when needed. Its techniques, decomposition, routing, rewriting, filter building, and grading, address the questions that defeat a single search. Research such as IRCoT, Self-RAG, and CRAG shows real gains on multi-step questions, and agentic search products apply the same loop for end users. The cost is several times the tokens and seconds of a single search. The practical step is to keep a fixed pipeline for simple lookups, route complex questions to an agent with a step limit, and evaluate every decision in its trajectory.