Home / Search evaluation / ANN Recall: Measuring What Approximate Search Misses

ANN Recall: Measuring What Approximate Search Misses

ANN recall is the share of the true nearest neighbors that an approximate nearest neighbor (ANN) index actually returns, measured by comparing its results with an exact, brute-force search. If an index returns 9 of the 10 true nearest vectors, its recall@10 is 0.9. Recall measures how faithful the fast index is to exact search, not whether results are relevant to the user. It trades off against speed: in a small experiment with a simple clustered index, checking 0.3% of the vectors gave 0.807 recall, and checking 20% gave 0.980. In Weaviate, the HNSW ef setting controls this trade-off at query time, and published benchmarks show around 96% to 98% recall@10 at millisecond latencies. Recall depends heavily on the data, so measure it on your own vectors.

What is ANN recall?

ANN recall is a measure of how many of the true nearest neighbors an approximate index returns for a query, compared with what an exact search would return. It answers one question: how much does the shortcut miss?

Approximate nearest neighbor indexes, such as HNSW and IVF, are fast because they skip most of the data. Weaviate’s documentation calls this an explicit trade-off between recall and query speed. It gives a concrete reading: a recall of 98% means 2% of returned results are not the true nearest neighbors.

The “true” neighbors come from exact search, which compares the query with every vector. That makes ANN recall a property of the index and its settings, not of the embedding model. A perfect index with recall 1.0 returns exactly what brute force would, good or bad. Approximate nearest neighbor search explains why indexes approximate at all. The calculation itself is simple.

How do you calculate ANN recall?

You calculate ANN recall by running the same queries through exact search and through the ANN index, then dividing the number of shared results by k, averaged over queries. The exact results are the ground truth.

recall@k = | ANN top-k  ∩  exact top-k |  /  k
mean recall@k = average of recall@k over a set of test queries

In Python, with vectors normalized so that a dot product equals cosine similarity:

import numpy as np

def recall_at_k(ann_ids, exact_ids, k=10):
    return len(set(ann_ids[:k]) & set(exact_ids[:k])) / k

def exact_top_k(X, q, k=10):            # brute force: the ground truth
    return np.argpartition(-(X @ q), k)[:k]

# For each test query: ann_ids from your index, exact_ids from exact_top_k
# mean_recall = np.mean([recall_at_k(a, e) for a, e in zip(ann_results, exact_results)])

Order does not matter for recall@k; only membership does. Benchmarks usually report recall@10 and sometimes recall@100. Weaviate’s ANN benchmark, for example, reports both, measured over 10,000 requests against each dataset’s ground truth. The word “recall” causes a common confusion, though.

What is the difference between relevance and recall in vector search?

The difference between relevance and recall in vector search is that ANN recall measures whether the index found the mathematically nearest vectors, while relevance measures whether the results actually help the user. High recall does not guarantee relevant results.

The two can disagree in both directions. If the embedding model places the wrong documents nearest, a perfect ANN index returns them perfectly: recall 1.0, poor relevance. If the index misses a nearest neighbor that was irrelevant anyway, recall drops while users notice nothing. That gap is the motivation for semantic recall, covered further on.

Retrieval recall, as in recall at k, is the relevance-based cousin: the share of known relevant documents found. Relevance versus recall separates the terms further. ANN recall’s own behavior is best understood as a curve.

How does ANN recall trade off against speed?

ANN recall trades off against speed because finding more of the true neighbors requires examining more of the index, which takes more time. Every ANN setting is a point on that curve.

To show the curve, we built a simple IVF index in numpy: 100,000 synthetic 128-dimension vectors grouped around 2,000 centers, split into 316 clusters. A query checks only the nprobe clusters nearest to it. Recall@10 was computed against exact search over 200 queries:

Two lessons stand out. The first percent of work buys most of the recall; the last few points are expensive. And the simple Python version at nprobe 64 was slower than one exact matrix multiplication, because of per-cluster overhead. Real engines are optimized far beyond this sketch, but the shape of the curve is general. Weaviate’s ANN benchmark presents the same trade-off for HNSW. The shape of the curve, though, depends on the data.

Why does ANN recall depend on the data?

ANN recall depends on the data because indexes exploit structure, such as clusters or graph neighborhoods, and different data has more or less of that structure. The same setting can give very different recall on different collections.

We ran the same IVF script on three versions of the synthetic data, changing only how tightly vectors clustered. With tight clusters, searching a single cluster gave recall@10 of 0.989. With medium spread, the version in the table above, it gave 0.807. With wide spread, it gave just 0.297, and even 64 clusters reached only 0.805.

Weaviate’s documentation makes the same point about HNSW: which build parameters lead to which recall depends on the dataset used. That is why its benchmark page reports several datasets and suggests picking the one closest to your production data. It is also why another team’s recall figure, or a vendor’s, rarely transfers. In Weaviate, the main levers are a handful of HNSW settings.

How do HNSW settings change recall in Weaviate?

HNSW settings change recall in Weaviate through three parameters: ef at query time, and efConstruction and maxConnections at build time. Higher values raise recall at the cost of speed, memory, or import time.

The default ef of -1 means dynamic: Weaviate sets ef to the query limit times 8, kept between 100 and 500. A limit of 20 gives an ef of 160. The reference notes that values above 512 show diminishing recall gains. Weaviate lowered the default maxConnections from 64 to 32 in version 1.26, reporting better recall and throughput curves on modern high-dimensional datasets.

Weaviate’s benchmark shows what tuned settings achieve. On SIFT1M, its recommended configuration reaches 98.35% recall@10 at 10,940 queries per second and 1.44 milliseconds mean latency. On DBPedia vectors from OpenAI’s ada-002 model, it reaches 97.24%. These numbers include network time and object retrieval, unlike library-only benchmarks. For a general starting point, Weaviate’s configuration tips suggest an ef of 64, an efConstruction of 128, and a maxConnections of 32. Index settings are not the only thing that moves recall.

What can lower ANN recall in production?

ANN recall in production can be lowered by filters, compression, data drift, and a query limit that is too small, even when benchmark recall looked high. Each needs its own check.

• Filters: restricting results to a subset can strand a graph search. Weaviate uses the ACORN filter strategy by default from version 1.34. When fewer than 40,000 objects match a filter, it switches to flat search instead. Measure recall with your real filters.
• Compression: quantization stores approximate vectors. Weaviate recommends testing product quantization on your data before production, and its rescoring options recover much of the loss.
• Data drift: Weaviate’s RAG evaluation post notes that a shift in data distribution can lower recall for compressed indexes trained on older data.
• Small limits: with dynamic ef, a small limit searches less of the graph. A Weaviate support answer explains how a larger limit can surface a better top result for this reason.

The common fix is the same: keep a recall test running against exact search on a sample of real queries. Monitoring search quality in production covers the habit. Many teams first meet recall through public benchmarks.

Does any search engine have better recall for ANN search?

No search engine has better ANN recall in general, because recall depends on the index settings and the data; most engines can reach high recall if you give up some speed. Benchmarks compare the trade-off curve, not a single number.

ANN-Benchmarks, by Aumüller, Bernhardsson, and Faithfull, is the best-known comparison of ANN libraries. It uses public datasets such as glove-100-angular, sift-128-euclidean, and gist-960-euclidean, and plots recall against queries per second. Its README notes results as of April 2025 on an AWS r6i.16xlarge machine, and that the project is no longer actively maintained. The Big-ANN benchmarks extended the idea to billion-scale datasets.

Database benchmarks differ from library benchmarks. Weaviate’s ANN benchmark measures end-to-end requests, including network overhead and retrieving objects from disk, and its tooling is open source so results can be reproduced. When comparing engines, compare at the same recall level, on data like yours, with your filters. A recall figure alone hides whether it was bought with latency or memory. A newer line of work questions whether plain recall is the right target at all.

Does semantic recall replace ANN recall?

Semantic recall does not replace ANN recall; it is a variant that counts only the true nearest neighbors that are actually relevant to the query, so an index is not penalized for missing irrelevant ones. It complements traditional recall, and the two answer different questions about the same index.

Kuffo and colleagues introduced semantic recall in a SIGIR 2026 paper, “Semantic Recall for Vector Search.” Their argument is that traditional recall punishes an index for skipping exact neighbors that would not have helped anyone. They define semantic neighbors as the subset of brute-force neighbors that are semantically relevant. They also propose tolerant recall, a proxy for when relevance labels are unavailable, and evaluated both on MS MARCO with 8.83 million documents.

For most teams, traditional ANN recall remains the practical index metric, because it needs no labels. Semantic recall is a useful reminder of the relevance gap: once recall is high enough, further index tuning often helps less than a better embedding model or a reranker. The last step is measuring recall yourself.

How do you measure ANN recall on your own data?

You measure ANN recall on your own data by sampling real queries, computing exact results for them once, and comparing your index’s results at the settings you plan to use. A few hundred queries usually show the trend.

1. Sample a few hundred representative queries, including filtered ones.
2. Compute exact top-k for each with brute force, or a flat index, and store them.
3. Query the ANN index with the same queries, filters, and limits as production.
4. Average recall@k and record latency at each setting you test.
5. Choose the cheapest setting that meets your recall target, then re-run after big data changes.

In Weaviate, a collection with a flat index gives exact results for the ground truth, and the same data in an HNSW collection gives the approximate results. Adjust ef on the HNSW collection at runtime to trace the curve. Related evaluation pages include how to evaluate semantic search, information retrieval metrics, retrieval benchmarks, and the recall versus latency trade-off.

ANN recall is the share of exact nearest neighbors an approximate index returns, calculated against brute-force results. It measures the index’s faithfulness, not relevance, and it rises with the work the index does: in our illustrative IVF experiment, from 0.807 at 0.3% scanned to 0.980 at 20%. The same setting gave 0.989 or 0.297 depending on the data, which is why benchmark numbers rarely transfer. In Weaviate, ef, efConstruction, and maxConnections set the trade-off. The practical step is to measure recall on a sample of your own queries, with your filters, and pick the cheapest setting that meets your target.