Home / Foundations / Transformers and BERT: Why Search Changed in 2019

Transformers and BERT: Why Search Changed in 2019

Transformers are neural networks that read text with self-attention, letting every word draw on every other word, and BERT is the 2018 transformer model from Google that made this context-aware reading practical for search. Earlier word embeddings gave each word one fixed meaning; BERT gives each occurrence of a word a vector shaped by the words around it, in both directions. That is why 2019 became a turning point: Google applied BERT to Search, researchers showed BERT could rerank passages far better than earlier methods, and Sentence-BERT turned it into a fast embedding model. Today transformers enter search in three ways: as bi-encoders that embed queries and documents separately, as cross-encoders that rerank by reading both together, and as late-interaction models that compare token vectors. Their main limits are input length and compute, because attention compares every token with every other. Weaviate runs transformer models as vectorizers and rerankers.

What are transformers and BERT?

A transformer is a neural network architecture built around self-attention, and BERT, short for Bidirectional Encoder Representations from Transformers, is an encoder-only transformer pre-trained on large amounts of text. BERT’s outputs are contextual token vectors.

Before transformers, language models processed text largely in sequence, word by word, which made long-range connections hard to learn and training slow to parallelise. The transformer, introduced by Vaswani and colleagues at Google in 2017, replaced that with attention: at every layer, each token looks at all other tokens and mixes in information from the most relevant ones.

BERT, published by Devlin and colleagues at Google in 2018, applied the transformer’s encoder half to language understanding. It produces one vector per input token, and each vector reflects the whole input. Weaviate’s article on vector embeddings explains the difference from word2vec: transformer embeddings take the entire input text into account, so each occurrence of a word has its own embedding, modified by the surrounding text.

That property is what search needed. Earlier static vectors, described in word2vec and GloVe, could not tell “bank” by a river from “bank” with a loan. Contextual vectors can.

How does self-attention read context?

Self-attention reads context by letting each token compute a relevance weight for every other token, then building its new representation as a weighted mix of theirs. Stacking many such layers builds rich, context-aware vectors.

Each token is projected into three vectors, often called query, key and value. A token’s query is compared with every token’s key to produce attention weights, and the token’s output is the weighted sum of the values. Several attention heads run in parallel, each free to focus on different relationships, such as which noun a pronoun refers to or which verb a preposition belongs to.

BERT’s attention is bidirectional: every token can attend to tokens on both its left and its right. Generative models such as GPT attend only to earlier tokens, because they predict the next word. For understanding a query or a passage, seeing the whole input at once is an advantage, which is why encoder models like BERT became the basis for search and embedding models.

Attention is also the source of the main cost. Comparing every token with every other means the work grows with the square of the input length, a limit discussed later.

How is BERT pre-trained?

BERT is pre-trained by hiding some words in its input and learning to predict them from the context on both sides, plus a second task of predicting whether one sentence follows another. The pre-trained model is then fine-tuned for specific tasks.

The masked language modelling task hides around 15% of the input tokens and asks the model to recover them. Because the model cannot see the hidden word, it must use everything around it, which forces it to learn how words relate in context. The next sentence prediction task paired sentences and asked whether the second really followed the first. Text is split into subword pieces with a WordPiece vocabulary, so rare words are built from smaller units, as described in tokenization vs embedding.

BERT came in two main sizes. Weaviate’s overview of late interaction models gives BERT-base as 110 million parameters with 768-dimensional outputs and BERT-large as 340 million parameters with 1,024 dimensions. Both accept inputs of up to 512 tokens. After pre-training, a small amount of task-specific fine-tuning produced state-of-the-art results across many benchmarks when the paper appeared.

Why did search change in 2019?

Search changed in 2019 because BERT moved from research into practice: Google applied it to Search, BERT-based rerankers beat earlier ranking methods, and Sentence-BERT made BERT usable for fast retrieval. Three separate advances arrived within months.

Google announced BERT in Search in October 2019 and said it affected about one in ten English searches in the U.S. Its example query about a Brazilian traveller needing a visa for the USA showed how a small word, “to”, could now change the results. That history is covered in semantic search at Google.

In research, Nogueira and Cho showed early in 2019 that BERT could work as a cross-encoder, reading the query and a passage together. Used that way, it reranked candidate passages on the MS MARCO benchmark far better than earlier approaches. Later that year, Reimers and Gurevych published Sentence-BERT, which fine-tuned BERT to produce sentence vectors comparable by cosine similarity, as described in Sentence-BERT. Dense passage retrieval with BERT encoders followed in 2020. Together these made transformers the core of semantic search.

How do transformers fit into a search system?

Transformers fit into a search system in three roles: bi-encoders that embed queries and documents separately for fast retrieval, cross-encoders that rerank by reading both together, and late-interaction models that compare token vectors. Most systems combine at least two.

Weaviate’s overview of late interaction models describes these as no-interaction, full-interaction and late-interaction models. Bi-encoders precompute one vector per document, so a query needs one encoding and a nearest-neighbour lookup; they suit first-stage retrieval, as covered in bi-encoders. Cross-encoders process query and document together with full attention, which is accurate but too expensive for a whole collection, so they rerank a short list, as covered in cross-encoders.

Late interaction sits between them. ColBERT keeps a vector for every token of the query and document and scores them with a maximum-similarity sum, keeping some token-level detail while still allowing precomputed document vectors. It is described in ColBERT and late interaction.

What are the limits of transformers in search?

The limits of transformers in search are input length, compute cost and the compression of a whole passage into one vector, all of which shape how documents are chunked and indexed. Long-context models ease, but do not remove, these limits.

Attention cost grows with the square of the input. At 512 tokens, each head in each layer computes 512 × 512 = 262,144 scores; at 8,192 tokens it is 67,108,864, which is 256 times as many. Weaviate’s article on vector embeddings lists the resulting memory demands as a reason for limited input lengths. Its ref2vec article calls long documents beyond the 512-token limit one of the outstanding problems in search technology.

Chunking is the standard response, splitting documents into pieces that fit the model, as covered in chunking for semantic search. Newer embedding models accept far longer inputs: Weaviate’s documentation lists Snowflake Arctic Embed L v2.0, served by Weaviate Embeddings, with a context of up to 8,192 tokens. One vector for a long passage still blurs detail, which is why chunking remains common even with long-context models.

How do you use transformer models in Weaviate?

You use transformer models in Weaviate by configuring a transformer vectorizer to embed objects and queries, and optionally a transformer reranker to reorder the top results. Both can run in local inference containers.

from weaviate.classes.config import Configure, Property, DataType
from weaviate.classes.query import Rerank

client.collections.create(
    "Article",
    properties=[Property(name="body", data_type=DataType.TEXT)],
    vector_config=Configure.Vectors.text2vec_transformers(
        pooling_strategy="masked_mean"),            # bi-encoder for retrieval
    reranker_config=Configure.Reranker.transformers(),  # cross-encoder for reranking
)

articles = client.collections.use("Article")
res = articles.query.hybrid(
    query="visa rules for Brazilian travellers to the US",
    limit=10,
    rerank=Rerank(prop="body", query="visa rules for Brazilian travellers to the US"),
)

The vectorizer and reranker configurations were built with the Weaviate Python client 4.18.3 as an API check; running them requires the transformer inference containers. The bi-encoder retrieves candidates through hybrid search, and the cross-encoder reorders them, the same two-stage pattern described in bi-encoder vs cross-encoder.

Hosted embedding models follow the same transformer principles, as discussed in choosing an embedding model.

How do transformers connect to earlier retrieval ideas?

Transformers connect to earlier retrieval ideas by producing dense vectors compared by cosine similarity, the same representation that the vector space model and latent semantic indexing introduced decades earlier. What changed is how much meaning the vectors carry.

The vector space model represented documents as term vectors, and latent semantic indexing compressed them into topics, as described in the vector space model and latent semantic indexing. Transformers learn their vectors from context at enormous scale, but the retrieval step, comparing vectors, is recognisably the same.

The broader picture is covered in what is information retrieval, high-dimensional vector space, the curse of dimensionality, semantic similarity vs relatedness, knowledge graphs in search and cache hits and misses.

Transformers read text with self-attention, letting every token draw on every other, and BERT applied that to language understanding with bidirectional context and masked-word pre-training. In 2019 BERT reached Google Search, transformed passage reranking and, through Sentence-BERT, became a fast embedding model, which is why search changed that year. Transformers now serve as bi-encoders for retrieval, cross-encoders for reranking and late-interaction models in between. Their quadratic attention cost limits input length, so documents are chunked, even as newer models accept up to 8,192 tokens.