Home / The History of Semantic Search: From LSI to BERT and LLMs

The History of Semantic Search: From LSI to BERT and LLMs

The history of semantic search is the story of retrieval moving from matching words to matching meaning. It passes through four broad stages: sparse term vectors in the 1970s, latent topics and probabilistic scoring in the 1990s, dense word embeddings in the 2010s, and contextual transformer models from 2018 onward. Gerard Salton’s vector space model, set out in 1975, turned documents into vectors and relevance into similarity. Latent semantic indexing in 1990 compressed terms into hidden topics, and BM25 refined keyword scoring in the 1990s. Word2vec in 2013 and GloVe in 2014 learned dense word vectors from context, and HNSW in 2016 made nearest-neighbour search over such vectors fast. The transformer in 2017 and BERT in 2018 brought context-aware reading, and 2019 put it into production search and sentence embeddings. Since late 2022, retrieval also serves as the memory of language models. Weaviate’s own history runs through the middle of this story, from GloVe in 2015 to a vector database.

What does the history of semantic search cover?

The history of semantic search covers how retrieval systems learned to match the meaning of a query rather than only its words, and the representations and algorithms that made each step possible. It spans about half a century.

The problem has stayed the same. People describe the same idea in different words, a difficulty known as the vocabulary mismatch problem, covered in the vocabulary mismatch problem. Keyword search can only find documents that share the query’s words. Every stage in this history is an attempt to close that gap.

What changed is the representation. Each era found a new way to turn text into numbers that captured more meaning, and a new way to search those numbers quickly. A modern overview of the result is in how semantic search works; the history explains how it came to work that way.

Which milestones are most important in semantic search?

The milestones that shaped semantic search are the vector space model, latent semantic indexing and BM25, word2vec and GloVe, approximate nearest neighbour graphs, transformers and BERT, sentence embeddings, and retrieval for language models. Each built on the last.

The timeline shows a steady thread. From the start, documents and queries were vectors and relevance was similarity between them. What improved was how much meaning each vector could hold and how fast large collections could be searched.

The eras that follow are described in order, each with links to the foundations guides that cover it in depth.

What role does the vector space model play in the story?

The vector space model starts the story by representing documents and queries as vectors of weighted terms and ranking documents by cosine similarity, replacing unranked Boolean matching. Gerard Salton and colleagues developed it around the SMART system.

Salton, Wong and Yang set out the model in 1975, building on years of experiments at Cornell. Karen Spärck Jones had shown in 1972 that rare terms discriminate better than common ones, which gave inverse document frequency and the TF-IDF weighting. Results could now be ranked by graded relevance. The model is described in the vector space model.

Its limit was built in: each term was its own dimension, so documents only matched when they shared words. Semantic search, in the modern sense, would need representations in which different words could still be close.

What does the 1990s era add to semantic search?

The 1990s era adds latent semantic indexing, which compressed terms into hidden topics so related words could match, while probabilistic models led to BM25, which became the standard for keyword scoring. Shared evaluation campaigns sped up progress.

Latent semantic indexing, published by Deerwester, Dumais, Furnas, Landauer and Harshman in 1990, applied singular value decomposition to the term-document matrix. Words that appeared in similar documents ended up on the same latent dimensions, so “car” could match “automobile”. It is covered in latent semantic indexing.

On the keyword side, Robertson and colleagues developed the Okapi BM25 scoring function, presented at the TREC evaluations in 1994. Weaviate’s hybrid search article describes BM25 as building on TF-IDF with a normalisation for document length. BM25 remains the keyword half of hybrid search today, as described in what is keyword search.

How do word embeddings change search in the 2010s?

Word embeddings change search in the 2010s by learning dense vectors from context, so similar words got similar vectors, while new nearest-neighbour indexes made searching millions of such vectors fast. Meaning became learnable from raw text.

Word2vec, from Mikolov and colleagues at Google in 2013, and GloVe, from Pennington, Socher and Manning at Stanford in 2014, produced vectors in which “cat” sat near “kitty”. Google announced the Knowledge Graph in 2012 and RankBrain in 2015, bringing entities and machine learning into web search. The models are covered in word2vec and GloVe and Google’s path in semantic search at Google.

Search over vectors needed new indexes. Libraries such as Spotify’s Annoy, Facebook’s faiss and Google’s ScaNN, named in Weaviate’s article on vector databases, trade a little accuracy for large speed gains. Malkov and Yashunin’s HNSW graph, published in 2016, became the default index in many systems, including Weaviate. The method is explained in approximate nearest neighbour search.

Where does Weaviate fit into the history of semantic search?

Weaviate fits into this history as one of the projects that turned embeddings and nearest-neighbour search into a database, combining vector indexes with storage, updates and filters. Its own account starts with GloVe.

The founder’s history of Weaviate describes a first encounter with word embeddings through GloVe in early 2015. Early experiments showed that moving through the embedding space could disambiguate words such as “apple”, and that centroids of word vectors could represent short texts. The data model drew inspiration from the RDF structure used by schema.org. By the end of 2018 the project entered a startup accelerator in the Netherlands as SeMI Technologies, short for Semantic Machine Insights, later renamed Weaviate.

Weaviate’s article on vector databases names the gap it addressed: approximate nearest neighbour libraries were fast but often read-only, lacking the create, read, update and delete operations expected of a database. Weaviate’s custom HNSW implementation added those, along with filters and, from version 1.17, hybrid search. Its article on enterprise use cases notes that ChatGPT’s release in November 2022 brought vector databases to much wider attention.

Why is 2019 a turning point for semantic search?

2019 is a turning point because transformers moved into practice: Google applied BERT to Search, BERT rerankers beat earlier methods, and Sentence-BERT made fast transformer embeddings possible. The groundwork was laid in 2017 and 2018.

The transformer architecture, introduced by Google researchers in 2017, and BERT, released in 2018, gave each occurrence of a word its own context-aware vector. In October 2019 Google said BERT affected about one in ten English searches in the U.S. Reimers and Gurevych’s Sentence-BERT, published that year, made sentence vectors comparable by cosine. These steps are covered in transformers and BERT in search and Sentence-BERT.

2020 extended the shift. Dense passage retrieval trained BERT encoders for question answering, ColBERT introduced late interaction, and Lewis and colleagues named retrieval-augmented generation. The models behind these steps are compared in retrieval models explained.

How have language models changed semantic search since 2022?

Language models have changed semantic search since 2022 by turning retrieval into the memory of generative systems, so search results now often feed an answer rather than a list of links. Retrieval quality matters more as a result.

ChatGPT’s release in November 2022 made large language models widely used, and retrieval-augmented generation became the standard way to ground them in private or current information. Search engines followed: Google announced MUM in 2021 and launched AI Overviews broadly in the U.S. in 2024. The relationship is covered in semantic search vs RAG.

Benchmarks grew alongside. BEIR, from 2021, tested retrieval across many domains, and MTEB, from 2022, compared embedding models on many tasks, as covered in the MTEB leaderboard. Meanwhile hybrid search, reranking and semantic caching became standard parts of production systems, covered in what is hybrid search, what is reranking and what is semantic caching.

What does the history teach about building semantic search?

The history teaches that no representation replaces the ones before it: exact terms, learned meaning and structured knowledge each solve different problems, so modern systems combine them. Evaluation decides the mix.

BM25 survives because exact matches still matter for names, codes and rare terms. Dense embeddings matter because people phrase things differently. Filters and knowledge graphs matter because some facts are structured. Rerankers and language models add deeper reading at the top of the list. Putting these together is the subject of how to build semantic search, and measuring the result is covered in how to evaluate semantic search.

The foundations series goes deeper into each era, from what is information retrieval to what are embeddings. Practical overviews include query understanding, metadata filtering, scaling semantic search and semantic search tools compared.

Semantic search grew from Salton’s vector space model of 1975, through latent semantic indexing and BM25 in the 1990s and word2vec, GloVe and HNSW in the 2010s. Transformers, BERT and Sentence-BERT followed from 2017 to 2019, and retrieval for language models after 2022. Throughout, documents and queries were vectors ranked by similarity; what changed was how much meaning the vectors held and how fast they could be searched. Weaviate’s path, from a first encounter with GloVe in 2015 to a vector database with hybrid search, mirrors that arc. The lasting lesson is to combine exact terms, learned meaning and structure rather than rely on one.