Home / Search evaluation / Mean Reciprocal Rank (MRR) Explained

Mean Reciprocal Rank (MRR) Explained

Mean reciprocal rank (MRR) is a search metric that scores how high the first relevant result appears, averaged over many queries. For each query you take 1 divided by the rank of the first correct result: 1 if it is first, 0.5 if second, 0.333 if third, and 0 if it never appears. MRR is the average of those values, from 0 to 1. It suits tasks with one right answer, such as question answering, navigational search, and finding the right passage for a RAG system. In a 12-query test from this site, keyword search scored an MRR of 0.533 while semantic search scored 1.000. MRR ignores every relevant result after the first, so pair it with recall or nDCG when several results matter.

What is mean reciprocal rank?

Mean reciprocal rank is a rank-aware evaluation metric that measures how early a system returns its first relevant result, averaged across a set of queries. It is widely used in information retrieval, NLP, machine learning, and AI evaluation.

Weaviate’s guide to retrieval metrics describes MRR as measuring how well a system finds a relevant result as the top result. It is rank-aware, like MAP and nDCG, meaning position matters. Precision and recall, by contrast, only count how many relevant items appear.

The metric has roots in question answering. Ellen Voorhees used it to score systems in the TREC-8 Question Answering track in 1999, where each question had a correct answer and systems returned ranked guesses. That origin explains its design: it cares about the first correct answer, because a person usually stops reading there.

The name helps. A “rank” is a position, the “reciprocal” is 1 divided by it, and the “mean” is the average over queries. A mean rank, by contrast, would average positions directly and treat a drop from 1 to 2 the same as from 50 to 51. The reciprocal makes the top positions count most, as the formula shows.

What is the mean reciprocal rank formula?

The mean reciprocal rank formula averages, over all queries, 1 divided by the rank of the first relevant result for each query. Each query contributes one number between 0 and 1, and the average of those numbers is the score. Written in plain notation, the formula has one term per query:

MRR = (1 / |Q|) × sum over queries q of  1 / rank_q

rank_q = position of the first relevant result for query q (1 = top)
if no relevant result is returned, the term for q is 0

The math has three useful properties. The score falls fast at the top: moving the first hit from rank 1 to 2 halves its contribution. It falls slowly lower down: rank 9 versus rank 10 barely differs. And MRR stays between 0 and 1, reaching 1 only when every query has a relevant result in first place.

Numbers from a real test make the formula concrete.

How do you calculate mean reciprocal rank by hand?

You calculate mean reciprocal rank by finding the first relevant result’s position for each query, taking 1 over that position, and averaging across queries. We used the 12-query test from this site’s semantic search examples page.

In that test, each query had one known relevant document. Keyword search with BM25 put it first for six queries, fourth for one, seventh for one, and missed it entirely for four, mostly paraphrased questions. Semantic search put it first every time.

Semantic search scores 12 / 12 = 1.000. The gap is driven mostly by the four misses, each worth zero. That illustrates MRR’s behavior: it is harsh on misses and on results that slip a few places, and it says nothing about what came after the first hit. Cutting the list at a fixed depth changes the score again.

What is MRR at k?

MRR at k, written MRR@k, is mean reciprocal rank computed only over the top k results, so a first relevant result ranked past position k counts as zero. It matches what users actually see.

In the example, BM25’s MRR@3 drops to 0.500, because the hits at ranks 4 and 7 now count as misses. Its MRR@10 stays at 0.533, since both fall within the first ten. Choosing k is a product decision: a search page that shows ten results suggests MRR@10, while a RAG system that passes three chunks to a model suggests MRR@3.

MRR@10 is a common standard. The MS MARCO passage ranking benchmark, one of the most widely used retrieval benchmarks, uses MRR@10 as its official metric. Always report k, because MRR and MRR@k are not interchangeable. The next question is what score counts as good.

What is considered a good MRR score?

A good MRR score depends on the dataset and task, so judge it against a baseline on the same queries rather than a fixed number. An MRR of 0.5 can be strong on a hard benchmark and weak on an easy help center.

Interpretation is simple at the extremes. An MRR of 1.0 means the first result was always relevant. An MRR around 0.5 means, roughly, the first relevant result tends to sit near position two, though a mix of first places and misses can produce the same number. That ambiguity is a reason to look at the distribution of ranks, not just the average.

The useful comparison is relative. Measure a baseline, such as BM25 or your current system, then measure the change you are testing. In our example, moving from BM25 to semantic search raised MRR from 0.533 to 1.000 on the same 12 queries. A small, well-chosen test set like that tells you more than any general threshold. MRR also needs to be read next to other metrics.

How is MRR different from recall and nDCG?

MRR is different from recall and nDCG because it scores only the position of the first relevant result. Recall counts how many relevant results appear, and nDCG scores the order of all of them, with graded relevance. Each answers a different question.

Weaviate’s own embedding comparison shows why the choice matters. Its guide compares OpenAI’s text-embedding-3-small and text-embedding-3-large on three queries and finds identical average precision, 0.5, and recall, 0.58. Computing MRR from the same result tables tells a different story. The small model placed a relevant item first, first, and second across the three queries; the large model second, first, and second.

That gives the small model an MRR of 0.833 and the large one 0.667, though precision and recall tie. Weaviate’s guidance notes that nDCG is often ideal when relevance is graded and MTEB uses it for retrieval, as nDCG explains. Recall at k covers the coverage view. MRR has a natural home in RAG.

What is MRR in RAG evaluation?

In RAG evaluation, MRR measures how high the first passage that answers the question appears in the retrieved list, which matters because the language model leans most on the top passages. It is one retrieval metric among several.

Weaviate’s overview of retrieval quality in RAG lists MRR alongside context precision, context recall, and faithfulness, metrics that frameworks such as RAGAS compute. MRR fits questions with one key passage, such as “what is the refund window?” Questions that need several passages are better served by recall, because a model cannot combine facts it never received.

A practical RAG check uses both. MRR@k with k equal to the number of chunks you pass to the model shows whether the key passage tends to lead. Recall@k shows whether all needed passages make it in. Semantic search in RAG covers the rest of the evaluation. Computing MRR takes a few lines.

How do you compute MRR in Python?

You compute MRR in Python by finding the first relevant position in each ranked list and averaging the reciprocals. The function needs no library, only a loop that stops at the first relevant ID in each list and adds 1 over its position. Pass the ranked IDs from your search engine and the set of IDs judged relevant for each query.

def mrr_at_k(ranked_lists, relevant_sets, k=10):
    """ranked_lists: list of result-ID lists; relevant_sets: list of sets of relevant IDs."""
    total = 0.0
    for results, relevant in zip(ranked_lists, relevant_sets):
        for position, doc_id in enumerate(results[:k], start=1):
            if doc_id in relevant:
                total += 1.0 / position
                break                     # only the first relevant result counts
    return total / len(ranked_lists)

# Weaviate's "sweet pastry" query, text-embedding-3-small results
print(mrr_at_k([["donut", "croissant", "roll", "bagel"]],
               [{"donut", "muffin", "scone"}], k=4))   # 1.0

Libraries cover it too. The sentence-transformers InformationRetrievalEvaluator, which Weaviate’s fine-tuning guide mentions, reports MRR@k alongside recall and nDCG. Scikit-learn has no dedicated MRR function, but label_ranking_average_precision_score equals MRR when each query has exactly one relevant item. With Weaviate, collect the ranked IDs from each query’s results and pass them in. The metric has blind spots worth knowing.

What are the limits of mean reciprocal rank?

The limits of mean reciprocal rank are that it ignores every relevant result after the first and treats relevance as yes or no. It also hides the spread of ranks behind one average and depends on how ties are broken. Each limit points to a companion metric.

• Only the first hit counts: a list with one relevant result scores the same as a list full of them. Use recall or nDCG when several results matter.
• Binary relevance: a perfect answer and a passable one score the same. nDCG handles graded relevance.
• Averages hide shape: many first places plus many misses can match steady second places. Look at the rank distribution.
• Ties and missing results: decide how to order tied scores and count empty result lists as zero, consistently.

MRR is also easy to confuse with reciprocal rank fusion, which uses the same 1-over-rank idea for a different purpose. RRF combines rankings from several retrievers into one list; MRR evaluates a ranking. Reciprocal rank fusion covers the former. Related evaluation pages include information retrieval metrics, precision at k, and how to evaluate semantic search.

Mean reciprocal rank averages 1 over the position of the first relevant result across queries, so it rewards systems that put a right answer at the top. In this site’s 12-query test it separated keyword search (0.533) from semantic search (1.000), and in Weaviate’s embedding example it separated two models that tied on precision and recall. It ignores everything after the first hit and treats relevance as binary, so it works best for single-answer tasks and alongside recall and nDCG. The practical step is to report MRR@k with k matched to what users or your RAG prompt actually see, against a baseline on the same queries.