Home / Embeddings / Word Embeddings vs Sentence Embeddings for Search

Word Embeddings vs Sentence Embeddings for Search

Word embeddings give each word one fixed vector, while sentence embeddings encode a whole sentence or passage into one vector that reflects its words in context. Early semantic search averaged word vectors from models such as word2vec, GloVe and fastText to represent a query or document. That approach captured topic but lost word order, gave every sense of a word the same vector and blurred long texts. Sentence embedding models, built on transformers such as BERT and trained on pairs of related sentences, read words in context and produce one vector per passage, which made them the standard for semantic search. Weaviate’s own early vectorizer, Contextionary, used a weighted mean of 300-dimensional fastText word vectors; Weaviate now marks it deprecated and recommends transformer-based models for production, with the static Model2Vec module for lightweight local testing.

What is the difference between word embeddings and sentence embeddings?

The difference is the unit being represented: a word embedding represents one word out of context, while a sentence embedding represents a whole piece of text with each word read in context. Search compares queries and documents, so it needs text-level vectors.

A word embedding model is a lookup table. Each word in its vocabulary maps to one vector, learned so that words appearing in similar contexts get similar vectors. “Run” and “ran” end up close; “run” and “coffee” end up far apart. But “bank” has one vector, whether the text is about rivers or loans.

A sentence embedding model is an encoder. It takes the whole input, lets every word influence the representation of every other word, and outputs one vector for the text. “Bank” in “river bank” and “bank loan” contributes differently, because the surrounding words change it.

Semantic search needs one vector per query and one per document or chunk. Word models must be combined into text vectors by some rule, usually averaging, while sentence models produce text vectors directly. The general idea of embeddings is covered in what are embeddings.

How do word embeddings work?

Word embeddings work by training a model to predict words from their neighbours, or to match word co-occurrence statistics, so that words used in similar contexts end up with similar vectors. The result is one dense vector per vocabulary word.

Word2vec, introduced by Mikolov and colleagues at Google in 2013, popularised the idea. Weaviate’s article on vector embeddings notes that it was initially trained on about 100 billion words and typically produces 300-dimensional vectors. GloVe, from Stanford in 2014, learned vectors from global co-occurrence counts. FastText, from Facebook research, added subword pieces so that rare and misspelled words still get vectors. The history of these models is covered in word2vec and GloVe.

These vectors captured striking regularities. Words cluster by topic, and some relationships appear as consistent directions in the space. For their time, they were a large step beyond sparse term vectors, because similar words finally had similar representations.

Their limits are built in. Each word has exactly one vector, so meaning that depends on context cannot be represented. Weaviate’s article points out that words such as “run”, “set” and “take” each have many meanings yet one vector. And a word model says nothing about sentences on its own.

How does search use averaged word embeddings?

Search uses word embeddings by combining the vectors of the words in a text, usually as a plain or weighted average, to get one vector for each query and document. Weighting gave rare, informative words more influence.

A plain average is the simplest rule. Add up the vectors of all words in the text and divide by their count. The result sits near the centre of the text’s vocabulary, which reflects its topic reasonably well. Common words such as “the” and “has” pull every average towards the same point, so better methods down-weight them.

Weaviate’s Contextionary followed this pattern. Its documentation describes a weighted mean of fastText word vectors trained on Wikipedia and Common Crawl. Each word is weighted by how often it occurred in the training corpus, so that “has” counts for less than “apples”. The output is a 300-dimensional vector per object.

Researchers refined averaging further. Arora, Liang and Ma’s 2017 method, often called SIF, weighted words by frequency and removed a common direction, and they described it as a simple but tough-to-beat baseline. Even so, every averaging method shares the same structural limits.

Why do averaged word vectors fall short for search?

Averaged word vectors fall short for search because an average ignores word order, cannot separate the senses of a word, barely registers negation and dilutes each word as texts get longer. These limits follow from the arithmetic.

Order disappears. An average does not depend on the order of its inputs, so “dog bites man” and “man bites dog” produce the same vector exactly. Search rarely hinges on this example, but many questions do depend on who did what to whom, or on which item a property belongs to.

Senses merge and small words vanish. “Bank” contributes the same vector to a text about rivers and one about loans. Negation is a single word among many, so “good” and “not good” end up close. In long passages, each word is one of hundreds, so specific details have little influence on the final vector, and a short query struggles to match the one passage that answers it.

Queries and documents also differ in kind. A question and the passage that answers it share a topic but rarely share wording or length. Averaging has no way to learn that relationship, which later models learned directly, as described in asymmetric vs symmetric semantic search.

How do sentence embeddings work?

Sentence embeddings work by passing the whole text through a transformer encoder, pooling its contextual token vectors into one vector, and training that vector so related texts land close together. Training on pairs is what makes them suited to search.

Transformers changed the representation. BERT, released by Google in 2018, gives each token a vector that depends on every other token in the input, so “bank” near “river” differs from “bank” near “loan”. The architecture and its use in search are covered in transformers and BERT in search.

BERT alone did not produce good sentence vectors for similarity. Sentence-BERT, published by Reimers and Gurevych in 2019, fine-tuned BERT in a siamese setup on sentence pairs and pooled the token vectors, so that two sentences could be compared by cosine similarity. That made fast semantic search with transformers practical. The approach is explained in Sentence-BERT.

Modern embedding models extend the recipe. They train on very large collections of query and passage pairs with contrastive objectives, often with separate prompts for queries and documents. Many also support truncation to shorter lengths, as described in Matryoshka embeddings, or compression, as in embedding quantization.

How do word and sentence embeddings compare for search?

Sentence embeddings give clearly better search relevance, while word embeddings are far cheaper and faster to compute, because they need only a table lookup and an average. The choice is a trade between quality and cost.

Quality favours sentence models. They handle context, word sense and the gap between questions and answers, which is why current retrieval models are contextual encoders. Benchmarks are discussed in the MTEB leaderboard, and choosing among models in choosing an embedding model.

Cost favours word models. A word-vector lookup involves no neural network at query time, so it runs on modest hardware with tiny latency. Weaviate’s article on vector embeddings notes the downsides of transformers: higher compute and memory needs, and limited input lengths. Batch processing reduces the cost of embedding large collections, as covered in batch embedding.

Switching from one to the other is not incremental. Vectors from different models live in different spaces, so moving from a word model to a sentence model, or between sentence models, means re-embedding everything, as described in re-embedding when the model changes.

When are word-level or static embeddings still useful?

Word-level and static embeddings are still useful when speed and resources matter more than top relevance, such as local testing, edge devices, or quick candidate generation before a stronger model. Modern static models narrow the quality gap.

Static embedding models compute a text vector without attention, much like averaging word vectors, but learn their token vectors from a strong transformer. Weaviate added the Model2Vec integration in version 1.31, describing it as a static embedding model that does not use attention and is therefore much faster to run. Weaviate’s documentation suggests it as the replacement for Contextionary when a local, lightweight model is needed for testing or development.

Word vectors also remain useful for analysis tasks such as finding related terms for query expansion, and as features in classic machine learning. For production search where relevance matters, Weaviate’s documentation recommends transformer-based models. Domain fit still matters, as discussed in domain-specific embeddings and fine-tuning embedding models.

How do you configure static and contextual embeddings in Weaviate?

You configure static and contextual embeddings in Weaviate as named vectors on one collection, so you can compare a fast static model with a transformer model on the same data. Each named vector has its own index.

from weaviate.classes.config import Configure, Property, DataType

client.collections.create(
    "Article",
    properties=[Property(name="body", data_type=DataType.TEXT)],
    vector_config=[
        Configure.Vectors.text2vec_model2vec(name="static"),           # fast, no attention
        Configure.Vectors.text2vec_transformers(name="contextual"),    # transformer encoder
    ],
)

articles = client.collections.use("Article")
fast = articles.query.near_text("refund for a late delivery", target_vector="static", limit=5)
best = articles.query.near_text("refund for a late delivery", target_vector="contextual", limit=5)

Both vectorizer configurations were built with the Weaviate Python client 4.18.3 as an API check; running them requires the corresponding inference containers. Comparing results from the two named vectors on your own queries shows how much relevance the contextual model adds for your data. Vector size also differs between models, as covered in embedding dimensions.

Images and other media need different encoders again, as described in multimodal embeddings. Tokens, the units both kinds of model read, are explained in tokenization vs embedding.

Word embeddings give each word one fixed vector, and early semantic search averaged them into text vectors, as Weaviate’s deprecated Contextionary did with 300-dimensional fastText vectors. Averaging captures topic but loses word order, merges word senses, mutes negation and dilutes long texts. Sentence embeddings, built on transformers such as BERT and trained on sentence pairs since Sentence-BERT in 2019, read words in context and produce one vector per passage, which is why they dominate semantic search today. Static models such as Model2Vec remain useful where speed matters more than top relevance, and Weaviate’s named vectors let you compare both on your own data.