Home / Search evaluation / Retrieval Benchmarks: BEIR, MS MARCO and MTEB Retrieval

Retrieval Benchmarks: BEIR, MS MARCO and MTEB Retrieval

Retrieval benchmarks are shared datasets of queries, documents and relevance labels used to compare search systems and embedding models on the same task. The best known are MS MARCO, built from real Bing queries, and BEIR, a set of 18 datasets that test models on unfamiliar domains. The retrieval part of MTEB ranks embedding models on a public leaderboard. They score ranking quality with metrics such as nDCG at 10 and MRR at 10. A separate family, ANN-Benchmarks, measures how fast vector indexes search and how faithfully, as recall against queries per second. Newer benchmarks such as LoCoMo and LongMemEval test whether AI agents can recall facts from long conversations. Benchmark scores are a starting point for a shortlist. Your own data and queries decide the final choice.

What is a retrieval benchmark?

A retrieval benchmark is a fixed collection of documents, a set of queries and labels saying which documents are relevant to each query, plus an agreed metric for scoring rankings. It lets different systems be compared fairly.

Every system runs the same queries over the same documents, and its rankings are scored against the same labels. That turns “model A seems better” into a number anyone can reproduce. The metrics are those covered in information retrieval metrics, especially nDCG, mean reciprocal rank and recall at k.

Benchmarks differ in what they test. Some measure ranking quality, some measure index speed, and some measure a whole question-answering system. Mixing their scores leads to wrong conclusions, so the first step is knowing which family a benchmark belongs to.

What kinds of retrieval benchmarks are there?

There are three main kinds of retrieval benchmarks: ranking quality benchmarks, index speed benchmarks, and end-to-end memory and RAG benchmarks. Each answers a different question.

Ranking quality benchmarks compare models and retrieval methods: does BM25, a dense model or a hybrid rank relevant documents first? Index speed benchmarks compare vector indexes on queries per second at a given recall, where recall means agreement with exact search. End-to-end benchmarks score the final answer of a system that retrieves and then generates, which mixes retrieval quality with the language model’s skill.

A model that tops a ranking leaderboard can be slow or memory-hungry, and a fast index can search weak embeddings very quickly. Choose benchmarks from the family that matches the decision at hand, then confirm on your own data.

What is MS MARCO?

MS MARCO is a large-scale dataset from Microsoft, built from real Bing search queries and web passages, and it became the standard training and test set for neural passage ranking. It was released in 2016 by Nguyen and colleagues.

The passage ranking task asks a system to rank about 8.8 million passages for each query, with MRR at 10 as the usual metric. Most queries have only one or two labelled relevant passages, so some unlabelled passages are actually relevant, and scores can understate good systems. The TREC Deep Learning track added deeper judgments on MS MARCO queries to reduce that problem.

Because so many models were trained on MS MARCO, a high score on it shows in-domain skill rather than general ability. That limitation motivated BEIR. The same corpus also appears in index benchmarks: Weaviate’s ANN benchmark includes an 8.8-million-vector MS MARCO set embedded with Snowflake’s Arctic Embed model.

What does BEIR measure?

BEIR measures zero-shot retrieval: how well a model ranks documents in domains and tasks it was not trained on, across 18 datasets. Thakur and colleagues introduced it in 2021.

BEIR’s datasets span fact checking, biomedical questions, financial questions, argument retrieval, duplicate question detection, entity search and more. Each has its own documents, queries and labels, and nDCG at 10 is the headline metric. The paper found BM25 a strong baseline out of domain, and that many dense models trained on MS MARCO did not beat it everywhere. That result helped make hybrid search and better-generalising models a priority.

Some BEIR datasets have licence restrictions, so public results often use a subset. Average scores hide large differences between datasets, so read the per-dataset numbers closest to your domain. A biomedical team should weigh the biomedical sets far more than the average.

How does MTEB relate to retrieval benchmarks?

MTEB, the Massive Text Embedding Benchmark, tests embedding models on many tasks, and its retrieval task is built largely from BEIR datasets. It hosts the most-used public leaderboard for embedding models.

The original MTEB paper, by Muennighoff and colleagues, covered 8 task types, 58 datasets and 112 languages. Retrieval sits alongside classification, clustering, reranking, semantic textual similarity and others. A model’s overall MTEB average blends all of them, so a model can rank highly overall while trailing on retrieval. Check the retrieval column for search work.

Leaderboard positions shift often as new models arrive, and some models are trained on data close to the test sets. The MTEB leaderboard explains how to read it, and choosing an embedding model covers turning a shortlist into a decision.

What do ANN benchmarks measure?

ANN benchmarks measure how fast a vector index answers queries and how many of the true nearest neighbours it returns, plotting recall against queries per second. They test the index, not the embeddings.

ANN-Benchmarks, maintained by Aumüller, Bernhardsson and Faithfull, compares index libraries on standard vector sets such as SIFT and GloVe. Recall here means overlap with exact search results, a different idea from relevance recall, as explained in ANN recall. Weaviate publishes its own ANN benchmark modeled on it. Each request includes network time and fetching objects from disk, while ANN-Benchmarks libraries return only IDs. Its reported metrics are recall at 10 and 100, queries per second, mean and p99 latency, and import time.

Speed numbers depend heavily on hardware, concurrency, client language and index settings. Weaviate’s benchmark scripts are open source, so a team can run them on its own hardware and data. That is more informative than comparing vendors’ published charts, which rarely share a setup.

What are LoCoMo and LongMemEval?

LoCoMo and LongMemEval are benchmarks for long-term conversational memory, testing whether AI assistants can recall and reason over facts from long multi-session chats. Memory systems often rely on retrieval, so they are retrieval benchmarks in a broader sense.

LoCoMo, from Maharana and colleagues in 2024, contains very long conversations of around 300 turns spread over many sessions. Its questions are single-hop, multi-hop, temporal and open-domain, plus adversarial questions that have no answer in the conversation. LongMemEval, from Wu and colleagues in 2024, poses 500 questions testing five abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates and abstention.

Scores on these benchmarks reflect the whole memory pipeline: what is stored, how it is chunked and indexed, how it is retrieved and how the model uses it. They are useful for comparing agent memory designs, less so for picking an embedding model alone. General RAG benchmarks follow the same end-to-end logic.

How well do benchmark scores transfer to your own data?

Benchmark scores transfer only partly to your own data, because your documents, queries and idea of relevance differ from any public test set. Use benchmarks to shortlist and your own test set to decide.

Public benchmarks use public text, often Wikipedia, news, web pages or scientific abstracts. Company documents, product catalogues and support tickets have their own vocabulary and query styles. A model ranked slightly lower on a leaderboard can win on your data, and the gap between two leading models is often smaller than the gap between domains. Some public test sets have also leaked into training data, which inflates scores.

Build a small benchmark of your own: a few hundred real queries with judged results, as described in search test sets and relevance judgments. Score candidate models and settings on it with the same metrics. How to evaluate semantic search covers the full process.

How should you use retrieval benchmarks in practice?

Use retrieval benchmarks to narrow the field, then run your own offline test, then confirm with live metrics. Each step filters out more options at lower risk.

Start with MTEB retrieval scores and the BEIR datasets nearest your domain to pick three or four candidate models. Test them on your own judged queries using precision at k, recall and nDCG. Check index settings with ANN recall at your target latency. Weigh relevance against recall for your use case.

After launch, watch real behaviour. Online search metrics and analytics and monitoring search quality in production show whether offline gains hold up. When results disappoint, why vector search returns irrelevant results helps trace the cause.

Retrieval benchmarks compare search systems on shared queries, documents and labels. MS MARCO tests in-domain passage ranking with real Bing queries. BEIR tests zero-shot retrieval across 18 datasets, and MTEB’s retrieval task, built largely on BEIR, ranks embedding models. ANN benchmarks such as ANN-Benchmarks and Weaviate’s measure index speed against recall, not relevance. LoCoMo and LongMemEval test long-term memory in AI agents. Use benchmarks to build a shortlist, your own judged queries to decide, and live metrics to confirm.