Home / Embeddings / Tokenization vs Embedding: What Is the Difference?

Tokenization vs Embedding: What Is the Difference?

Tokenization splits text into tokens, small pieces such as words or parts of words, each mapped to an ID; embedding turns those tokens into vectors of numbers that represent meaning. They are two consecutive steps, not alternatives. A tokenizer is a fixed set of rules and a vocabulary, with nothing learned about meaning. An embedding model is a trained neural network that starts by looking up a vector for each token ID, then mixes in context. Pooling then combines the token vectors into one vector for the whole text. In all-MiniLM-L6-v2, the sentence “Tokenization splits unbelievable words into pieces.” becomes 10 tokens, then 10 token vectors of 384 numbers, then one 384-number sentence embedding. Tokens decide what the model sees and how long a text can be; embeddings decide what search can compare.

What is the difference between tokenization and embedding?

The difference between tokenization and embedding is that tokenization cuts text into discrete, countable units, while embedding converts those units into continuous vectors that place similar meanings close together. Tokenization comes first and feeds embedding.

Think of tokenization as spelling out the input in the model’s own alphabet. The tokenizer reads “Tokenization” and outputs two pieces, “token” and “##ization”, each with a fixed number. Nothing about meaning is involved; the same rules apply to any text. Embedding is where meaning appears. The model maps each token ID to a vector, adjusts every vector using the tokens around it, and produces numbers whose geometry reflects meaning.

The two steps also differ in what they produce. A tokenized text is a sequence of integers whose length varies with the text. A sentence embedding is a fixed-length vector, 384 numbers for all-MiniLM-L6-v2 whether the input is three words or three hundred. That fixed length is what lets a vector database compare any two texts. What are embeddings covers the vector side in depth.

How does text become an embedding step by step?

Text becomes an embedding in six steps: tokenize, convert tokens to IDs, look up token embeddings, run the transformer, pool the token vectors, and output one sentence vector. The first two steps are tokenization; the rest are embedding.

For the example sentence, the tokenizer adds a [CLS] marker at the start and [SEP] at the end, giving 10 tokens. Each token ID selects one row of the token embedding matrix, a learned table of 30,522 rows and 384 columns. The six transformer layers then let every token vector absorb information from the others. This is how “bank” ends up with different vectors in “river bank” and “bank loan”.

The model outputs 10 contextual vectors of 384 numbers each. all-MiniLM-L6-v2 averages them, called mean pooling, into one sentence embedding. Other models take the [CLS] vector instead, or keep every token vector for late-interaction search. Transformers and BERT in search explains the architecture behind steps 3 and 4.

What is a token in an embedding model?

A token in an embedding model is a unit of text from the model’s fixed vocabulary, usually a whole common word or a fragment of a rarer word, identified by an integer ID. Most models use subword tokenizers.

Subword tokenization keeps the vocabulary small while still handling any word. all-MiniLM-L6-v2 uses a WordPiece tokenizer inherited from BERT, where “##” marks a piece that continues the previous one. “Unbelievable” is common enough to be one token, but “Weaviate” splits into “we”, “##avia” and “##te”. Other model families use byte-pair encoding or SentencePiece, which follow the same idea with different rules.

Token counts matter in practice. Hosted embedding APIs charge and limit by tokens, and every model has a maximum input length in tokens. Sentence-transformers truncates all-MiniLM-L6-v2 inputs at 256 tokens, so anything after that point does not affect the vector. That limit is a main reason for chunking for semantic search, and the chunk size calculator helps size chunks against it.

What is a token embedding?

A token embedding is the vector a model assigns to a single token, either the static vector looked up from the embedding matrix or the contextual vector after the transformer layers. The term covers both, which causes confusion.

The static token embedding is one row of the token embedding matrix. Every occurrence of a token ID gets the same starting vector, much like a word2vec or GloVe word vector. The contextual token embedding is the output of the last transformer layer, which differs each time the token appears in a new sentence.

Sentence-transformers exposes the contextual version through output_value="token_embeddings". For the example sentence it returns a 10 by 384 array, one row per token, before pooling. Most semantic search stores only the pooled sentence vector. Late-interaction models such as ColBERT store the token vectors instead, trading memory for finer matching. Word vs sentence embeddings compares the two levels.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
text = "Tokenization splits unbelievable words into pieces."
print(model.tokenizer.tokenize(text))                       # 8 subword tokens
print(model.encode(text, output_value="token_embeddings").shape)  # (10, 384)
print(model.encode(text).shape)                             # (384,)

How do tokens, vectors and embeddings differ?

Tokens, vectors and embeddings differ in kind: a token is a piece of text, a vector is any ordered list of numbers, and an embedding is a vector learned so that its position reflects meaning. Every embedding is a vector, but not every vector is an embedding.

A token ID is one integer, not a vector. A one-hot vector or a bag-of-words count vector is a vector, but its numbers are counts, not learned meaning. These are sparse vectors, mostly zeros, the basis of the vector space model. An embedding is a dense learned vector, where every number carries part of the meaning.

“Vectorization” adds a further layer of vocabulary. In machine learning it means turning any data into vectors, by counting or by embedding. Weaviate uses “vectorizer” for the model integration that creates embeddings at import. So “tokenization vs vectorization vs embedding” usually means splitting, converting to numbers, and the learned kind of conversion.

“Token vector” is a loose phrase that usually means a token embedding, the vector for one token. In a contextual model that vector changes with the surrounding words, so the same token can have many token vectors. When a page or tool uses these words, check whether it means a single token’s vector, a whole text’s vector, or a sparse count vector. The answer changes what can be compared and how much storage it needs.

Why does tokenization matter for semantic search?

Tokenization matters for semantic search because it sets the maximum text length, decides how rare words and names are represented, and determines cost on token-priced APIs. Problems that look like embedding failures often start at the tokenizer.

Product codes, chemical names and new brand names often split into many fragments. The model may have seen those fragments in other contexts, so their vectors can be noisy. That is one reason exact identifiers are better served by keyword search, and why domain-specific embeddings or fine-tuning help in specialised fields. Languages that the tokenizer handles poorly also produce more tokens per word, which shortens the effective input length.

Token counts also drive cost and throughput. Embedding APIs bill per token, and batch embedding limits are usually set in tokens per request or per minute. When comparing models, check each one’s tokenizer and maximum length alongside its benchmark scores, as part of choosing an embedding model.

How does keyword tokenization differ from model tokenization?

Keyword tokenization differs from model tokenization because it produces index terms for exact matching, while model tokenization produces subword units for a neural network. A hybrid search system runs both on the same text.

Weaviate’s documentation makes this point directly: tokenization for keyword search, vectorization and text generation use different tokenizers and produce different tokens from the same text. For BM25, Weaviate’s default word tokenization splits on non-alphanumeric characters and lowercases. Other options include lowercase, whitespace, field for exact values, and trigram for typo-tolerant matching.

The keyword setting is chosen per property and changes what keyword search and filters can match, without affecting the vector. A SKU property might use whitespace to keep case, while a description uses word. BM25 vs vector search covers how the two kinds of retrieval compare.

Which embedding choices does tokenization affect?

The embedding choices that tokenization affects include vector size, compression, query and document handling, and the move to images and audio. Each starts from the token sequence.

The width of the token embedding matrix sets the vector length, covered in embedding dimensions. Shorter vectors from Matryoshka embeddings and compressed ones from embedding quantization change the output, not the tokens. Some models add instruction prefixes to queries, which costs tokens, as explained in asymmetric vs symmetric semantic search.

Images and audio have tokens too. Vision models cut an image into patches, and each patch plays the role of a token, as described in multimodal embeddings. Changing the model changes the tokenizer as well, so stored vectors must be rebuilt, the job described in re-embedding when the model changes. The MTEB leaderboard lists maximum token lengths alongside scores.

Tokenization and embedding are consecutive steps. Tokenization splits text into tokens from a fixed vocabulary and maps them to integer IDs. Embedding looks up a vector for each token, adds context through the transformer, and pools the result into one fixed-length vector that search can compare. Tokens set length limits and costs; embeddings carry meaning. Keyword search uses its own tokenizer for the inverted index, so a hybrid system tokenizes the same text twice, for different purposes.