Home / Search evaluation / Relevance vs Recall in Vector Search

Relevance vs Recall in Vector Search

Recall in vector search measures whether the expected items came back, while relevance measures whether what came back is actually useful for the query. The two are easy to confuse because “recall” has two meanings. ANN recall compares an approximate index’s results with an exact nearest-neighbour search, so it measures the index. Retrieval recall compares results with human relevance judgments, so it measures the whole system. Relevance goes further: it asks whether the results answer the need, in a sensible order, without duplicates or noise. A system can score perfect ANN recall and still return irrelevant results, because the embedding model put the wrong documents nearest. It can also show highly relevant top results while missing documents a complete answer needs. Diagnosing poor search starts by asking which layer failed: the index, the model and retrieval, or the ranking. Each has its own metric and its own fix.

What is the difference between relevance and recall in vector search?

The difference between relevance and recall in vector search is that recall counts whether expected items were returned, while relevance judges whether returned items serve the user’s need. Recall is a coverage measure; relevance is a quality judgment.

Recall always needs a reference list of what should have come back. The question is where that list comes from. If it comes from an exact search over the same vectors, recall measures how faithful the approximate index is. If it comes from people judging documents, recall measures how well the system finds relevant material. These are different questions with the same name.

Relevance is the user’s view. A result is relevant if it helps with the query, and a result list is good if the most helpful items come first. Relevance depends on the query, on the user’s purpose and on the other results shown. Two near-identical relevant documents add less together than two different ones.

Search teams often report one number and assume it covers everything. A dashboard showing 0.98 recall can sit alongside users complaining that results are off. Untangling the meanings explains why. The metric family is introduced in information retrieval metrics. The clearest way to see the difference is to separate the layers each metric looks at.

Which three layers do recall and relevance measure?

Recall and relevance measure three layers: the index, which ANN recall checks; retrieval, which recall against judgments checks; and ranking and usefulness, which relevance metrics check. A failure in any layer shows up as poor results.

The index layer is the vector index itself. Approximate indexes such as HNSW trade a little accuracy for speed, so they may miss some of the true nearest vectors. ANN recall measures how many they miss. It says nothing about whether those nearest vectors are good answers.

The retrieval layer combines the embedding model, any keyword search and the filters. It decides which documents become candidates. Recall against judgments measures whether relevant documents are among them. The ranking layer orders the candidates, perhaps with a reranker, and decides what the user sees first. Relevance metrics such as nDCG and precision measure it.

The layers stack. A perfect index cannot rescue a model that embeds the wrong things, and a perfect model cannot rescue an index that drops its nearest neighbours. Measuring each layer separately is the only way to tell which one is at fault. The index layer is the easiest to measure, so it comes first.

How is ANN recall measured?

ANN recall is measured by running each query through both the approximate index and an exact brute-force search, then counting how many of the exact top k the approximate search also returned. No human judgments are involved.

Weaviate’s ANN benchmark describes the method. For each parameter setting it runs 10,000 requests and measures recall at 10 and recall at 100 by comparing Weaviate’s results with the ground truth given in each dataset. The ground truth is the exact nearest-neighbour list, so the benchmark measures how close the index comes to a perfect search over the same vectors.

Recall here rises with search effort. Weaviate’s HNSW documentation explains that a larger ef makes search more accurate but slower, and that values above 512 show diminishing improvements in recall. Larger graphs built with higher efConstruction and maxConnections also raise it. The tuning trade-off is covered in the recall vs latency trade-off.

A high ANN recall proves only that the index works. If exact search over the same embeddings returns irrelevant documents, the approximate search will faithfully return the same irrelevant documents. The measurement in practice is covered in ANN recall. Retrieval recall asks a harder question.

How is retrieval recall against judgments measured?

Retrieval recall against judgments is measured by dividing the relevant documents found in the top k by the total relevant documents judged for the query. It needs a test set of queries with human relevance labels.

Weaviate’s evaluation metrics article defines recall at k this way: relevant items in the top k divided by all relevant items in the dataset. Its pastry example has three relevant items, of which two appear in the top four, for a recall at 4 of about 0.667. The judgments, not the vectors, define what counts as relevant.

This recall depends on the whole retrieval stack. The embedding model decides which documents sit near the query. Keyword search can add documents with exact matching terms. Filters remove candidates. A change to any of them can move retrieval recall while ANN recall stays the same. The metric is covered in depth in recall at k.

Judgments have limits. They are expensive, so they rarely cover every relevant document, and unjudged documents usually count as irrelevant. A system that finds good documents nobody judged is penalized. Building better judgments is covered in search test sets and relevance judgments. Even complete recall does not guarantee that users are satisfied.

Why can a search system have high recall but feel irrelevant?

A search system can have high recall but feel irrelevant because recall ignores order, degree of usefulness, redundancy and noise among the results. Finding every relevant item does not mean showing it well.

Order is the most common cause. Recall at 10 counts a relevant document at rank 10 the same as one at rank 1. If the useful results sit at the bottom of the page, users see the irrelevant ones first and give up. Rank-aware metrics such as nDCG catch this, as described in nDCG explained.

Degree and redundancy matter too. Binary judgments treat a perfect answer and a marginal mention alike. Five chunks of the same paragraph can all be relevant and still waste the page. Diversity methods such as maximal marginal relevance address that. Noise is the last factor: a high recall at a large k often comes with low precision, so relevant items are buried among irrelevant ones, as covered in precision at k.

Granularity can also mislead. A long document may be judged relevant because one sentence answers the query, while the rest is off topic. Returning the whole document scores well on recall but reads poorly. Smaller chunks help. The reverse failure, relevant results that are incomplete, is just as common.

Why can highly relevant results still miss what users need?

Highly relevant results can still miss what users need because a short, precise list may leave out documents that a complete answer requires. The top results look good, so the gap goes unnoticed.

Questions that need several facts expose this. “What are the refund rules for digital and physical goods” may need two policy pages. A system that returns three excellent passages about physical goods scores high precision and still fails the user. Recall at the number of passages used catches the miss.

RAG makes the problem sharper. A language model writes from the passages it receives. If a needed passage is missing, the answer is incomplete or wrong, and nothing in the answer signals the gap. That is why recall at the context size is the first retrieval check for RAG, as discussed in reranking in RAG.

Vocabulary gaps often cause these misses. A relevant document using different words can sit far from the query in embedding space, or outside the keyword index’s reach. Hybrid search combines both signals to close the gap. Knowing whether the failure is coverage or quality points to the fix, so diagnosis comes next.

How do you diagnose whether recall or relevance is failing?

You diagnose the failure by comparing approximate results with exact search, then exact search with judgments, then the ranked list with what users found useful. The first layer that breaks is the one to fix.

Start with the index. Run a sample of queries with an exact, brute-force search over the same vectors, which a flat index provides, and compare with the HNSW results. If they mostly agree, the index is fine. If they differ, raise ef or the graph settings and measure again.

Next, check exact results against judgments. If relevant documents are missing even from exact search, the embedding model or the candidate set is the problem. Common fixes are hybrid search, a better or fine-tuned embedding model, and different chunking. The causes are listed in why vector search returns irrelevant results.

Finally, look at order and usefulness. If relevant results are present but low or repeated, ranking is the problem. A cross-encoder reranker or diversity selection usually helps. The evaluation workflow that ties these steps together is in how to evaluate semantic search. Each layer then needs a number in regular reports.

Which metrics should you report for recall and relevance?

You should report ANN recall for the index, recall at k against judgments for retrieval, and a rank-aware relevance metric such as nDCG for ranking, plus online signals from real users. Each answers a different question, so none replaces another.

ANN recall belongs in engineering dashboards, tracked when index settings, quantization or data size change. Recall and nDCG on a fixed judged test set belong in model and pipeline comparisons, run before every change to the embedding model, chunking or fusion. Precision at the visible count complements them for result pages. MRR suits single-answer interfaces, as covered in mean reciprocal rank.

Offline numbers need checking against behaviour. Click-through, reformulation and abandonment show whether users find what they need, as described in online search metrics and analytics. Ongoing tracking is covered in monitoring search quality in production.

Public benchmarks are a reference point, not a target. They measure retrieval on fixed datasets, which may not resemble your queries, as explained in retrieval benchmarks. Your own judged queries remain the test that predicts your results.

Recall and relevance answer different questions. ANN recall checks whether the index returns the true nearest vectors, retrieval recall checks whether relevant documents are found, and relevance checks whether the results help, in the right order and without redundancy. A system can pass one and fail another, so measure each layer: approximate against exact, exact against judgments, and ranked lists against usefulness. Fix the first layer that breaks, and report ANN recall, recall at k and nDCG together so no single number hides the problem.