Home / Foundations / What Is Information Retrieval?

What Is Information Retrieval?

Information retrieval is the field that studies how to find documents in a large collection that satisfy a person’s information need, and how to rank them by relevance. It is the science behind web search engines, library catalogues, site search, enterprise search and the retrieval step of RAG. An IR system has three parts: an index built ahead of time, a way to turn a query into something the index can match, and a ranking model that orders the matches. Ranking models have moved from Boolean matching to the vector space model, probabilistic scoring such as BM25, learning to rank, and now neural embeddings. Evaluation is as central as ranking. Since the Cranfield experiments of the 1960s, systems have been judged on test collections of queries and relevance judgments, using measures such as precision, recall and nDCG. Semantic search is the latest chapter of this field, not a separate one.

What is information retrieval?

Information retrieval is the discipline of finding relevant material, usually text documents, within large unstructured collections in response to a query. Christopher Manning, Prabhakar Raghavan and Hinrich Schütze give the standard definition in Introduction to Information Retrieval (2008).

Their wording is that IR “is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers)”. Each part matters. The material is unstructured, so there are no neat columns to query. The goal is an information need, not a literal string. The collection is large, so scanning every document for every query is too slow.

The output is a ranked list, not a yes-or-no answer. A database returns every row that matches a condition. An IR system returns the documents most likely to be useful, best first, and accepts that some will be imperfect. That shift from exact matching to ranking by estimated relevance defines the field.

IR covers more than web search. Library catalogues, email search, legal discovery, product search, question answering and the retrieval step that feeds a language model are all IR problems. The modern form that ranks by meaning is described in what is semantic search. Every IR problem starts from the gap between what a person wants and what that person types.

How does an information need differ from a keyword-based query?

An information need is what a person actually wants to know, while a keyword-based query is the few words that person types to express it. IR systems only see the query, so they must infer the need from it.

Someone who wants to know whether a laptop battery can be replaced might type “laptop battery swap”. The words are a lossy summary. The need includes the model, the context and the reason for asking. Relevance in IR is judged against the need, which is why a document can be relevant without containing the query words.

A keyword-based query in information retrieval is a set of terms matched against an index of the terms in each document. In the Boolean model the terms combine with AND, OR and NOT, and a document either matches or does not. In ranked models such as BM25 every document containing some of the terms gets a score. Weaviate’s documentation describes this for its keyword search: the query is tokenized, stopwords are removed, and BM25F scores are computed against selected properties.

Keyword queries work well when the user and the document share words, such as names, codes and exact phrases. They fail when the same idea uses different words, the vocabulary mismatch problem covered in the vocabulary mismatch problem. The basics are in what is keyword search. Whatever the query style, an IR system answers it with the same three components.

How does an information retrieval system work?

An information retrieval system works by indexing documents ahead of time, processing each query into the index’s terms or vectors, and ranking the matching documents with a scoring model. Indexing is slow and done once; querying must be fast.

The classic index is the inverted index. It maps each term to the list of documents that contain it, often with counts and positions. Weaviate’s documentation describes the same structure: inverted indexes map values such as words to the objects that contain them. A query for two terms reads two lists instead of every document. The data structure is explained in the inverted index.

Query processing applies the same steps used at indexing, such as lowercasing, tokenization and sometimes stemming, so query terms match document terms. The ranking model then scores each candidate. Semantic systems add a second index type, a vector index of embeddings, and rank by vector similarity. The structure of the system stays the same. What has changed most over the decades is the ranking model.

What are the main retrieval models in information retrieval?

The main retrieval models are the Boolean model, the vector space model, probabilistic models such as BM25, language models, learning to rank and neural models. Each one is a different answer to the question of how to score a document for a query.

The Boolean model returns exactly the documents that satisfy a logical expression, with no ranking. The vector space model, developed by Gerard Salton and colleagues, represents documents and queries as weighted term vectors and ranks by cosine similarity, as covered in the vector space model. Term weighting by TF-IDF came from this line of work.

Probabilistic models estimate the chance that a document is relevant. BM25, which came out of Stephen Robertson’s Okapi work at City University London, became the standard keyword ranking function. It combines term frequency, inverse document frequency and document length normalization, as Weaviate’s keyword search documentation explains. Its details are in the BM25 algorithm. Language models for retrieval, introduced by Ponte and Croft in 1998, rank documents by how likely they are to generate the query.

Learning to rank combines many signals, such as BM25 scores, freshness and clicks, in a trained model. Latent semantic indexing tried to capture meaning with matrix factorization, described in latent semantic indexing. Neural models arrived next.

How did neural models change information retrieval?

Neural models changed information retrieval by learning representations of meaning from data, so documents can match a query that shares none of its words. This is the shift that semantic search names.

Word embeddings such as word2vec and GloVe placed related words close together, as covered in word2vec and GloVe. Transformer models such as BERT then produced contextual representations of whole passages, described in transformers and BERT in search. Sentence-BERT made it practical to embed sentences for fast similarity search, as explained in Sentence-BERT.

Three neural patterns now dominate. Dense retrieval embeds queries and documents separately and finds nearest vectors. Learned sparse retrieval keeps an inverted index but learns term weights. Cross-encoders read the query and document together to rerank a shortlist. The families are compared in retrieval models explained.

Embeddings bring their own geometry. Vectors with hundreds of dimensions behave in ways that defy intuition, as covered in high-dimensional vector space and the curse of dimensionality. Closeness also mixes similarity with mere relatedness, a distinction covered in semantic similarity vs relatedness. Neural models did not remove the need to measure results; they made measurement more important.

How is information retrieval evaluated?

Information retrieval is evaluated with test collections: a set of documents, a set of queries and human judgments of which documents are relevant to each query. A system’s ranked results are compared with the judgments using metrics such as precision, recall, MRR and nDCG.

The method traces back to the Cranfield experiments led by Cyril Cleverdon in the 1960s, which compared indexing methods on a fixed collection with known relevant documents. The Text REtrieval Conference, TREC, run by the US National Institute of Standards and Technology since 1992, scaled the approach up and made shared test collections the norm. Benchmarks such as BEIR and MTEB continue the tradition for neural models, as covered in retrieval benchmarks.

Precision is the share of returned documents that are relevant. Recall is the share of relevant documents that were returned. Ranking metrics reward putting relevant documents near the top, and nDCG also handles graded relevance. The definitions and formulas are in information retrieval metrics.

Offline test collections have limits. Judgments are expensive, often incomplete, and may not match what real users want. Production systems add online signals such as clicks and reformulations. A practical approach is in how to evaluate semantic search. Evaluation also clarifies how IR differs from neighbouring problems.

How does information retrieval differ from database search and question answering?

Information retrieval differs from database search because it ranks unstructured documents by estimated relevance, while a database returns exact matches from structured records. It differs from question answering because it returns documents, not answers.

A SQL query such as “orders over 100 dollars placed this week” has one correct result set. Every row either qualifies or not, and order is set by a sort clause. An IR query such as “late delivery complaints” has no single correct set. Documents are more or less relevant, and the ranking is the result. Many systems combine both: structured filters narrow the set, and IR ranks what remains.

Question answering and RAG sit on top of IR. A retrieval step finds candidate passages, and a reader model or language model writes the answer from them. If retrieval misses the right passage, no generator can recover it. The relationship is covered in semantic search vs RAG.

Knowledge graphs are another neighbour. They store facts as entities and relations, and search engines use them to answer factual queries directly, as described in knowledge graphs in search. Web search engines blend all of these, as the history in semantic search at Google shows. In a modern stack, these ideas meet in a single retrieval engine.

How do modern systems such as Weaviate implement information retrieval?

Modern systems such as Weaviate implement information retrieval with both classic and neural components: an inverted index scored by BM25F, a vector index scored by embedding similarity, and hybrid fusion of the two. The old and new models run side by side.

Weaviate’s keyword search ranks by BM25F, which extends BM25 to weight several fields, such as a title more than a body. Its academy course gives the BM25 parameters’ defaults as 1.2 for k1, which controls term frequency saturation, and 0.75 for b, which controls length normalization. Its vector search ranks by distance between embeddings. Its hybrid search runs both and fuses the scores.

Weaviate’s search documentation summarizes the choice. Vector search matches meaning, for queries whose wording differs from the data. Keyword search matches exact terms, for names, codes and error strings. Hybrid search handles both and is described as a good default for text search. Rerankers and boosts then reorder the shortlist, without adding new results.

Caching sits in front of all of this in busy systems, so repeated queries skip retrieval, which brings its own vocabulary covered in cache hits and misses. The hybrid approach is explained in hybrid search, and dense retrieval in dense retrieval.

Information retrieval is the field of finding and ranking documents that satisfy an information need in large unstructured collections. Every IR system builds an index offline, processes queries online, and ranks candidates with a model, which has evolved from Boolean matching through the vector space model and BM25 to neural embeddings. Test collections with relevance judgments, from Cranfield to TREC and BEIR, keep the field honest. Semantic search is IR with learned representations of meaning, and modern engines such as Weaviate run keyword and vector retrieval together, with results measured on real queries.