Home / Foundations / Word2vec and GloVe: The First Useful Word Embeddings

Word2vec and GloVe: The First Useful Word Embeddings

Word2vec and GloVe are the two models that made dense word vectors practical, learning one vector per word from the contexts it appears in so that words with similar usage get similar vectors. Word2vec, from Mikolov and colleagues at Google in 2013, trains a shallow network to predict a word from its neighbours or its neighbours from the word. GloVe, from Pennington, Socher and Manning at Stanford in 2014, fits vectors directly to global counts of how often words occur together. Both produce vectors of a few hundred dimensions in which “cat” sits near “kitty” and relationships such as king minus man plus woman land near queen. For search they were a big step beyond keyword matching, but not enough: each word gets a single vector regardless of context, and they do not represent sentences directly. Weaviate’s own story begins here; its founder was introduced to word embeddings through GloVe in 2015.

What are word2vec and GloVe?

Word2vec and GloVe are unsupervised methods that learn a dense vector for every word in a vocabulary from large amounts of plain text, based on the words that appear around it. They are the best-known static word embedding models.

Both rest on the distributional hypothesis: words that occur in similar contexts tend to have similar meanings. If “coffee” and “tea” both appear near “cup”, “drink” and “morning”, their vectors should be close. Neither model is told what words mean; meaning emerges from co-occurrence across billions of words.

The result is a lookup table. Each word maps to one vector, typically 50 to 300 numbers, and those vectors can be compared with cosine similarity. Weaviate’s article on vector embeddings explains the effect: “run” and “ran” come out far more similar than “run” and “coffee”, although each word still has only one vector.

These models replaced sparse representations in which each word was its own dimension, as in the vector space model. Dense vectors let similar words share information, which earlier methods such as latent semantic indexing had approached through matrix factorisation.

How does word2vec learn word vectors?

Word2vec learns word vectors by training a shallow neural network on a prediction task: given a word, predict its neighbours (skip-gram), or given the neighbours, predict the word (CBOW). The network’s weights become the vectors.

Mikolov and colleagues introduced the two architectures in 2013. Skip-gram takes each word in turn and tries to predict the words within a window around it. CBOW, continuous bag of words, does the reverse, predicting a word from the average of its neighbours. A follow-up paper the same year added negative sampling, which trains the model to tell real neighbour pairs from random ones, making training on huge corpora fast.

The task is only a means to an end. Nobody needs the predictions; what matters is that, to predict neighbours well, the model must give words with similar neighbours similar internal vectors. Weaviate’s article notes that word2vec was initially trained by Google with 100 billion words and typically produces 300-dimensional vectors.

What does word2vec train on?

Word2vec trains on pairs of words that occur near each other, generated by sliding a window across the text, so every sentence becomes many small training examples. The window size sets how much context counts.

With a window of two words, the centre word “sat” in “the cat sat on the mat” pairs with “the”, “cat”, “on” and “the”. Words near the ends of the sentence have fewer neighbours. Across the sentence, that gives 2 + 3 + 4 + 4 + 3 + 2 = 18 pairs. A corpus of billions of words produces billions of such pairs.

Real implementations add refinements, such as down-sampling very frequent words like “the” and drawing random negative words in proportion to their frequency, but the pairs remain the raw material.

Small windows emphasise words that can substitute for each other, because they share immediate neighbours. Larger windows capture broader topical association. That choice shapes whether the space reflects similarity or relatedness, a distinction covered in semantic similarity vs relatedness.

How does GloVe differ from word2vec?

GloVe differs from word2vec by working from global statistics: it first counts how often every pair of words co-occurs across the whole corpus, then fits vectors so their dot products match those counts. Word2vec learns from local windows one at a time.

GloVe, short for Global Vectors, was published by Pennington, Socher and Manning in 2014. Its authors argued that ratios of co-occurrence counts carry meaning: words related to “ice” co-occur more with “solid” than with “gas”, and the reverse holds for “steam”. Fitting vectors to the logarithm of the counts, with a weighting that tempers very common pairs, captures those ratios in the geometry. Stanford released pre-trained vectors, including sets of 50 to 300 dimensions.

In practice the two methods give similar results, and both became standard tools. GloVe has a special place in Weaviate’s history. The founder’s account of the company’s origins describes a first introduction to word embeddings through GloVe in early 2015, and Weaviate’s glossary describes its early Contextionary as inspired by the GloVe concept. That account uses a supermarket metaphor: products sit near similar products, and words with similar meanings sit near each other in a space of around 300 dimensions.

What do word analogies like king minus man plus woman show?

Word analogies like king minus man plus woman landing near queen show that some relationships appear as consistent directions in the vector space, so vector arithmetic can capture them. They are illustrations, not a general rule.

Weaviate’s article on vector embeddings calls “king − man + woman ≈ queen” perhaps the most famous result: the difference between “king” and “man” behaves like a direction for royalty that also separates “queen” from “woman”. Similar patterns appear for countries and capitals or verb tenses. They suggested that word vectors encode structure, not just similarity.

Analogies should be read with care. The nearest vector to the arithmetic result is often one of the input words, so standard tests exclude them, and many analogies fail. What the examples show reliably is that the space has meaningful geometry. The general behaviour of such spaces is covered in high-dimensional vector space.

How does search use word vectors?

Search uses word vectors in two main ways: to expand queries with similar words, and to build document and query vectors by averaging the vectors of their words. Both helped with vocabulary mismatch.

Word vectors also supported classic tasks around search, such as clustering documents, tagging named entities and classifying text, usually as input features to other models. For retrieval itself, two uses stood out.

Query expansion adds neighbours of query terms, such as “automobile” for “car”, so keyword search can find documents using different words. This addresses part of the vocabulary mismatch problem, and the technique is covered in query expansion.

Averaging builds a vector for a whole text. Weaviate’s Contextionary did this with a weighted mean of fastText word vectors, fastText being a later model that adds subword pieces. Averaging captures topic but loses order and context, as explained in word vs sentence embeddings.

Why are word2vec and GloVe not enough for search?

Word2vec and GloVe are not enough for search because each word has a single vector regardless of context, they do not represent sentences directly, and their averaged text vectors lose word order and meaning. Contextual models fixed these gaps.

Weaviate’s article lists the core problem: words such as “run”, “set”, “go” or “take” have many meanings, yet each gets one vector. The vector for “bank” sits somewhere between rivers and loans. Words outside the training vocabulary get no vector at all in word2vec and GloVe, a gap fastText reduced with subword pieces.

Search compares queries with passages, not single words. Transformers solved this by giving every occurrence of a word its own vector shaped by context. The approach is described in transformers and BERT in search, and sentence models turned those into passage vectors, as described in Sentence-BERT.

How do you generate skip-gram training pairs?

You generate skip-gram training pairs by pairing each word with every word within the window on either side, skipping the word itself. A short Python function reproduces the example.

def skipgram_pairs(words, window=2):
    pairs = []
    for i, centre in enumerate(words):
        for j in range(max(0, i - window), min(len(words), i + window + 1)):
            if j != i:
                pairs.append((centre, words[j]))
    return pairs

pairs = skipgram_pairs("the cat sat on the mat".split())
print(len(pairs))                                  # 18
print([p for p in pairs if p[0] == "sat"])         # sat with the, cat, on, the

The function was run to produce the example in the figure. Libraries such as gensim implement full word2vec training, including negative sampling, on large corpora.

The rest of the foundations series covers what came after: semantic search at Google, the curse of dimensionality, knowledge graphs in search, what is information retrieval and cache hits and misses.

Word2vec and GloVe made dense word vectors practical by learning from context: word2vec by predicting neighbours in sliding windows, GloVe by fitting global co-occurrence counts. Their vectors place words with similar usage close together and even encode some relationships as directions, as in king minus man plus woman near queen. For search they enabled query expansion and averaged text vectors, but one vector per word and no sentence representation limited them. Contextual transformers and sentence embeddings took over, while the core idea of meaning as position in a learned vector space carried through to modern vector databases such as Weaviate.