Home / Embeddings / MTEB Leaderboard Explained: How to Read Embedding Benchmarks

MTEB Leaderboard Explained: How to Read Embedding Benchmarks

The MTEB leaderboard is a public ranking of text embedding models, hosted on Hugging Face, that scores each model across many tasks from the Massive Text Embedding Benchmark. MTEB was introduced in 2022 by Muennighoff and colleagues with 8 task types, 58 datasets, and 112 languages. Its central finding still holds: no single embedding method dominates across all tasks. Weaviate’s guidance is to treat the leaderboard as a starting point, not a verdict. Pick the task tab that matches your use case, which for semantic search is Retrieval, and shortlist a few models. Then test those models on your own data, because scores are self-reported and public test sets can leak into training data.

What is the MTEB leaderboard?

The MTEB leaderboard is a Hugging Face Space that ranks embedding models by their results on the Massive Text Embedding Benchmark, a large collection of evaluation tasks. MTEB stands for Massive Text Embedding Benchmark.

The benchmark came first. Muennighoff, Tazi, Magne, and Reimers published it in 2022, and it appeared at EACL 2023. Their motivation was that embeddings were usually tested on a small set of datasets from one task. A model that did well on semantic similarity might not do well on clustering or reranking, and nobody could tell. MTEB put many tasks in one place so models could be compared fairly.

The leaderboard is the public face of that benchmark. Weaviate’s model-selection guide describes what it shows for each model: its size, memory use, embedding dimensions, maximum input tokens, and scores on tasks such as retrieval and classification. Tabs let you switch between an overall view and individual tasks, languages, or domains.

The leaderboard is open source and maintained by the community around the mteb project on GitHub. That openness is its strength, and also why its results need careful reading. The first step is knowing what each task measures.

What tasks does MTEB measure, and with which metrics?

MTEB measures 8 task types in its original form: bitext mining, classification, clustering, pair classification, reranking, retrieval, semantic textual similarity, and summarization. Each task type has its own main metric, which the original paper defines.

The table shows why one number cannot describe a model. A model can cluster well but retrieve poorly. Weaviate Academy lists retrieval, classification, clustering, reranking, and STS as the groups most users look at.

The retrieval tasks borrow from an earlier benchmark. The MTEB paper reuses 15 datasets and their evaluation from BEIR, including MS MARCO, Natural Questions, HotpotQA, FiQA, SciFact, and TREC-COVID. The metric, nDCG@10, rewards a model for placing relevant documents near the top of the first ten results. nDCG explains the formula. With the tasks clear, the scores become easier to read.

How do you read a score on the MTEB leaderboard?

You read an MTEB score as an average of a model’s results across a set of tasks, so the tab you choose decides what the number means. Higher is better, and most metrics fall between 0 and 1, often shown as percentages.

The overall score deserves the most caution. The MTEB paper itself notes that averages computed over all datasets are biased toward task types with more datasets. A model that is strong on a large group of classification datasets can rank highly overall while being average at retrieval. Weaviate Academy makes the same point about the retrieval score alone: it averages many separate retrieval benchmarks, and the average hides real differences between them.

A few habits make the leaderboard more useful:

• Choose the task tab first. Weaviate’s guide says a RAG team should look closely at Retrieval rather than the overall rank.
• Choose the language. Benchmarks exist for specific languages and regions, such as Chinese, French, Polish, and the Scandinavian languages. A mean task score on a Scandinavian benchmark says more about Danish or Norwegian search than an English average does.
• Read the cost columns. Model size, memory, and embedding dimensions decide what the model costs to run.
• Check max tokens. Weaviate notes that 512 tokens is usually enough for paragraph-sized chunks, but long documents need more.
• Open the per-task view. Ranks can move sharply from one dataset to the next.

Weaviate Academy gives a concrete example from a dated snapshot. One Snowflake Arctic model sat mid-table on general tasks yet led clearly on a long-input passkey retrieval task. For semantic search, one task tab matters more than the rest.

Which MTEB score should you use for semantic search?

For semantic search you should use the Retrieval score, because retrieval tasks test exactly what search does: given a query, rank the relevant documents first. Reranking scores matter too if the model will reorder candidates.

The retrieval score is an nDCG@10 average over many query-and-document datasets. Each dataset has queries with human relevance judgments, so the score reflects how often the model puts known relevant documents near the top. That is close to how teams judge their own search, as the evaluation guide describes.

STS scores are tempting but measure something different. STS asks whether two sentences mean the same thing. Search asks whether a document answers a query, and queries are usually shorter than the documents they match. That difference is the subject of asymmetric versus symmetric search. A model can be excellent at STS and only average at retrieval.

Even within retrieval, look at the datasets closest to your content. A legal or medical search tool should weight datasets from those fields above general web question answering. Weaviate’s guide adds that domain-specific models can outperform general ones even with lower general MTEB scores. The reasons rankings can mislead go further than task choice.

Why can MTEB rankings mislead?

MTEB rankings can mislead because results are self-reported, public test data can leak into training, benchmark data may not resemble yours, and averages hide weak tasks. Weaviate’s guidance lists all four.

• Self-reported results: Weaviate notes that leaderboard results are self-reported, so they reflect each submitter’s setup.
• Data leakage: MTEB datasets are public. Weaviate warns that some models may include them in training data, which inflates scores.
• Domain mismatch: the benchmark datasets may not represent your documents, queries, or vocabulary.
• Averaging: a strong average can hide a weak score on the one task you need.
• Size and speed: larger models tend to score higher, but Weaviate cautions that they add latency and can overfit, underperforming in production.

Rankings also move. New models arrive often, and the task list itself changes. Weaviate Academy advises checking the leaderboard on the day you evaluate rather than trusting a number quoted in a course, a blog, or a video. Knowing how to read the leaderboard lasts longer than any list of current leaders. The benchmark has also grown well beyond its first version.

How has MTEB changed since the original benchmark?

MTEB has grown from 58 datasets in 2022 into MMTEB, a multilingual extension with more than 500 quality-controlled tasks across more than 250 languages. The leaderboard now offers many benchmarks, not one.

MMTEB, the Massive Multilingual Text Embedding Benchmark, was published in 2025 by a large community team led by Enevoldsen. It adds harder task families such as instruction following, long-document retrieval, and code retrieval. It also introduced a zero-shot English benchmark that keeps a ranking order close to the full version at a fraction of the computing cost.

One MMTEB result is worth remembering when reading any leaderboard. Billion-parameter models built on large language models reached top results on some languages and task types. Yet the best-performing publicly available model overall was multilingual-e5-large-instruct, with only 560 million parameters. Size alone did not decide the ranking.

For readers, the change means two things. Always check which benchmark a score refers to, since an English score and a multilingual score are not comparable. And prefer the zero-shot or newest version when it exists, because it reduces the effect of training on test data. The leaderboard is most useful as one step in a selection process.

How should you use MTEB to choose an embedding model?

You should use MTEB to build a shortlist, then choose among the shortlisted models by testing them on your own queries and documents. Weaviate Academy describes this as a four-stage workflow.

1. Define your needs: language, domain, document length, latency, budget, and hosting.
2. Shortlist three to five models from the relevant MTEB tab, filtered by size and license.
3. Test each model on your own data, with retrieval metrics such as nDCG or recall.
4. Re-evaluate periodically, because better models and new data keep arriving.

The test does not need to be large. Weaviate’s guide suggests a small hand-labeled set of 50 to 100 objects as a first check. The mteb package can also run the official tasks on any model, which helps when you want the leaderboard numbers for a model you plan to fine-tune.

import mteb

# Run official MTEB tasks on a candidate model
model = mteb.get_model("sentence-transformers/all-MiniLM-L6-v2")
tasks = mteb.get_tasks(tasks=["SciFact", "NFCorpus"])  # two retrieval tasks
results = mteb.evaluate(model, tasks=tasks)

Once a model is chosen, Weaviate makes it easy to try in place. A vectorizer is set per collection, so two candidates can be loaded into two collections and queried side by side. Changing models later means re-embedding everything, as re-embedding when the model changes explains. Choosing an embedding model, domain-specific embeddings, and fine-tuning embedding models cover the next steps. One last confusion comes from a different leaderboard entirely.

Is LoCoMo an embedding leaderboard like MTEB?

LoCoMo is not an embedding leaderboard; it is a benchmark for the long-term conversational memory of LLM agents, so its scores cannot be compared with MTEB scores. The two are often searched together because both appear in discussions of AI memory and retrieval.

Maharana and colleagues introduced LoCoMo in 2024 in “Evaluating Very Long-Term Conversational Memory of LLM Agents.” Its conversations average about 300 turns and 9,000 tokens, spread over up to 35 sessions. It tests question answering, event summarization, and multimodal dialogue generation over that history.

MTEB scores the embedding model alone. LoCoMo scores a whole system, which may include an LLM, a memory store, and a retriever. A memory system could use an embedding model chosen from MTEB as one part, but a LoCoMo result says nothing direct about that model. The same caution applies to general LLM leaderboards. They rank chat or reasoning ability, not the quality of vectors for search. Related pages include what embeddings are, Matryoshka embeddings, and multimodal embeddings.

The MTEB leaderboard ranks embedding models across many tasks, from retrieval and reranking to clustering and classification, each with its own metric. For semantic search, the Retrieval tab and its nDCG@10 score matter most, read in the right language and domain. Averages, self-reported results, and possible test-data leakage mean a high rank is a reason to test a model, not proof that it suits your data. The practical step is to shortlist three to five models from the relevant tab and test them on a small labeled set of your own queries.