Home / Search evaluation / Building a Test Set for Search Evaluation

Building a Test Set for Search Evaluation

A search test set, often called a golden dataset, is a fixed list of realistic queries paired with relevance judgments that say which documents are good answers to each query, and how good. It lets a team score any search system or change with metrics such as nDCG and recall, and compare the results fairly. Building one takes five steps. Collect real queries, pool candidate results from several systems, judge their relevance on a graded scale, store the labels, and evaluate. Judging is the expensive part. Language models can now label at scale, but their labels should be checked against a sample of human judgments. A few hundred well-chosen queries is a common starting size. The exact number depends on how small a difference you need to detect, and it can be estimated from a pilot.

What is a golden dataset for search evaluation?

A golden dataset for search evaluation is a trusted set of queries with known correct results, used as the reference against which every version of a search system is scored. In information retrieval it is called a test collection.

The idea comes from decades of retrieval research, where shared test collections let researchers compare methods on equal terms. A company’s own golden dataset does the same job for one product. It answers the question “is this change better?” with a number instead of an opinion. Retrieval benchmarks are public test sets; a golden dataset is yours, built from your documents and your users’ queries.

“Golden” signals that the labels are trusted, not that they are perfect. Every test set has gaps and judgment errors. What matters is that it is stable, realistic and large enough that small errors do not change conclusions. The same idea applies beyond search: an AI agent’s tool-use evaluation set pairs tasks with the expected tool calls, and a RAG evaluation set pairs questions with reference answers.

How do you build a search test set?

You build a search test set by collecting queries, pooling candidate results, judging their relevance, storing the labels and running evaluations. Each step has choices that decide how useful the set will be.

A good set covers how people really search. Draw queries from search logs, support tickets and subject experts. Include the frequent head queries, which carry most traffic, and a sample of rare tail queries, where search often fails. Keep the original wording, including typos and vague phrasing.

Store the result as plain data: a list of queries with IDs, and a labels file with one row per query, document and grade. This is the format standard tools read, such as pytrec_eval, which computes the metrics described in information retrieval metrics. Version the set like code, so results from different months stay comparable.

How do you choose which documents to judge?

You choose which documents to judge by pooling: taking the top results from several different retrieval systems for each query and judging only that combined pool. Documents outside the pool are treated as not relevant.

Judging every document for every query is impossible beyond tiny collections; Weaviate’s guide to RAG evaluation notes that one query can need a hundred documents annotated. Pooling, used in TREC evaluations for decades, focuses effort where systems disagree and where relevant documents are likely. Pool from varied systems, such as BM25, a dense model and hybrid search, so one method’s blind spots do not shape the labels.

Pooling has a known bias. A new system that finds relevant documents no pooled system returned gets no credit for them, because they were never judged. When a new method is tested, judge its unjudged top results too before drawing conclusions. This matters most for recall at k, which depends on knowing every relevant document.

How should relevance be judged?

Relevance should be judged on a graded scale with written guidelines, such as 0 for not relevant up to 3 for a perfect answer, by judges who understand the domain. Graded labels support nDCG; binary labels support precision and recall.

Write guidelines with examples for each grade, and include hard cases: partly relevant pages, outdated content, near duplicates. Judges should read the query as a user would, not hunt for keyword matches. Weaviate’s evaluation guidance makes the same distinction: graded labels capture that a page about vitamin B12 is closer to a vitamin D query than a sports article is, even if neither is perfect.

Check consistency. Have two people judge a shared sample and measure agreement, for example with Cohen’s kappa. Low agreement means the guidelines are unclear or the task is genuinely ambiguous. Fix the guidelines before judging thousands of pairs. nDCG uses the grades directly, while precision at k and mean reciprocal rank need a cut-off that turns grades into relevant or not.

Can LLMs create relevance judgments?

LLMs can create relevance judgments at a fraction of the cost of human judging, and research from Bing found their labels agreed well with searchers. They still need validation against human labels in each new domain. Treat them as fast judges that need supervision.

Thomas and colleagues at Microsoft reported in 2023 that carefully prompted language models could label relevance with agreement comparable to human labellers. Frameworks such as Ragas use zero-shot LLM prompts to score context relevance for RAG. The ARES framework, from Saad-Falcon and colleagues, trains small judge models instead. It starts from a human-annotated validation set of at least 150 datapoints and a few in-domain example queries.

LLM judges have biases. They can favour longer passages, passages that share the query’s wording, or text written in their own style. Label a few hundred pairs with both people and the model, measure agreement, and adjust the prompt until it is acceptable. Keep a human-labelled subset as the anchor, and recheck when the judging model changes.

Can you generate test queries synthetically?

You can generate test queries synthetically by asking a language model to write questions that a given passage answers, which gives labelled pairs without any search logs. They are useful for new products but differ from real queries.

Methods such as InPars and Promptagator generate queries for documents, and round-trip filtering keeps only queries whose source passage is retrieved when searching with them. Weaviate’s Transformation Agent recipe uses this approach, and Weaviate’s search mode benchmark reports results on EnronQA, a benchmark built from synthetic queries over emails.

Synthetic queries tend to copy the passage’s vocabulary, which flatters keyword search and makes the task easier than real use. They also have exactly one known relevant passage, while real queries often have several. Mix synthetic and real queries where possible, and replace synthetic ones with real ones as logs accumulate.

How many queries does a test set need?

A test set needs enough queries that the difference you care about is larger than the noise between queries, which you can estimate from the spread of per-query scores. A few hundred queries is common; small differences need more.

The formula is the standard one for a confidence interval on a mean. The number of queries equals 1.96 times the standard deviation of per-query scores, divided by the half-width you want, all squared. With a spread of 0.3, a half-width of 0.05 needs 139 queries and 0.02 needs 865. The spread of 0.3 is an assumption for illustration; compute yours from a pilot of 50 or so queries.

Comparisons between two systems on the same queries are usually more efficient, because both systems find the same queries easy or hard. A paired test, such as a paired t-test or a randomisation test on per-query differences, uses that link. Report per-query results as well as averages, since a change can help most queries while badly hurting a few.

How do you keep a test set useful over time?

You keep a test set useful by adding new queries from production failures, re-judging when documents change, and keeping a frozen core for long-term comparison. A stale test set slowly stops reflecting users.

Feed it from real failures. Queries with no clicks or quick reformulations, found through online search metrics and analytics, make good additions once judged. Cases traced in why vector search returns irrelevant results belong there too. Remove or re-judge items whose documents were deleted or rewritten.

Keep a frozen core subset that never changes, so scores from last year and this year stay comparable, and a rolling subset that tracks current behaviour. Use the set for every change, from models to ANN index settings, as described in how to evaluate semantic search. In production, pair it with monitoring search quality. Remember the gap between relevance and recall when reading results.

A search test set, or golden dataset, pairs realistic queries with graded relevance labels so every search change can be scored the same way. Collect queries from real use, pool candidates from several systems, judge them with clear guidelines, store the labels and evaluate with standard metrics. LLM judges and synthetic queries cut cost but need checking against human labels. Size the set from the spread of per-query scores: 139 queries give about plus or minus 0.05 on mean nDCG when that spread is 0.3. Keep a frozen core and feed in new failures over time.