Home / Embeddings / What Are Vector Embeddings? Embeddings for Semantic Search Explained

What Are Vector Embeddings? Embeddings for Semantic Search Explained

A vector embedding is a fixed-length list of numbers that a trained model assigns to text, images or other data so that items with related meaning sit close together. OpenAI’s embeddings guide defines an embedding as a vector of floating-point numbers whose distance from another vector measures relatedness. Its text-embedding-3-small model returns 1,536 of those numbers per input by default. Search systems compare these vectors instead of words, so a query can reach a passage that shares its meaning but not its vocabulary. The model encodes every document once, a vector database such as Weaviate stores the results, and each query is encoded the same way at search time. The nearest stored vectors under a metric such as cosine become the results. Search quality then rests on the model, its dimensions, the metric, and keeping queries and documents in one model’s space.

What is a vector embedding?

A vector embedding is a fixed-length list of floating-point numbers produced by a trained model, where the distance between two lists measures how related their inputs are. OpenAI’s embeddings guide, read October 8, 2026, states the rule plainly: small distances suggest high relatedness and large distances suggest low relatedness. Weaviate’s vector search documentation describes the same object as a set of numbers that captures an object’s semantic meaning. The model that produces it is called a vectorizer or an embedding model.

Machine learning models compute with numbers, not with words or pixels. An embedding is the translation step that makes text, images and audio usable by those computations. A shortened, illustrative embedding looks like [0.021, -0.113, 0.087, ...], with one number per position. Each position is a dimension. The model fixes the number of positions, whatever the input length. All-MiniLM-L6-v2 maps every sentence or paragraph to 384 dimensions, according to its Hugging Face model card. OpenAI documents 1,536 dimensions for text-embedding-3-small and 3,072 for text-embedding-3-large.

Individual dimensions carry no human label. No position means “colour” or “price” on its own. Meaning lives in the pattern across all positions and in the distances that pattern creates. The count of positions is the model’s embedding dimensions setting, which OpenAI’s guide ties to compute, memory and storage cost. Most text models used in semantic search output dense vectors, where most positions hold nonzero values. A list of numbers alone is still not an embedding, and the difference decides whether distance means anything.

How does an embedding differ from an ordinary vector?

Every embedding is a vector, but a vector becomes an embedding only when a trained model places it so that distance carries meaning. A vector is any ordered list of numbers. A row of daily temperatures is a vector. It describes something, yet the distance between two such rows says nothing about related ideas. An embedding is a vector whose position was learned so that related inputs land near each other.

One-hot encoding shows the difference. It gives every vocabulary word its own position, sets that position to 1 and every other position to 0. The result is long and mostly zeros, and every word is equally distant from every other word. Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean described this limit in their 2013 Google paper on word vectors. Words treated as vocabulary indices carry “no notion of similarity between words,” they wrote. A learned embedding replaces those isolated slots with real numbers, so “coffee” can sit nearer to “tea” than to “truck.”

Sparse and dense describe how many positions hold values. Sparse vectors are mostly zeros, and their nonzero positions often map to vocabulary terms. pgvector, the Postgres extension, stores the two kinds in separate vector and sparsevec types. Learned sparse models are covered in sparse vectors, and dense versus sparse vectors compares where each wins. Once distance carries meaning, a search engine can rank by it, and that ranking is the reason retrieval adopted embeddings.

Why do search systems use vector embeddings?

Search systems use vector embeddings so a query can match text that shares its meaning but not its words. A keyword index matches tokens. A query for “hot drink” misses a document that only says “espresso” unless someone wrote a synonym rule. An embedding model trained on enough related text places both phrases close together. The distance between them stays small even though they share no word.

Weaviate’s vector search documentation describes the retrieval step. The system compares stored object vectors against the query vector, finds the closest matches and returns the top results. Weaviate returns 10 results when no limit is set. That nearest-neighbor ranking is the core of semantic search and of vector search more broadly. Weaviate also notes that vector search works on images, video and audio, not only on text.

Retrieval is one use among several. OpenAI’s guide lists six uses for text embeddings: search, clustering, recommendations, anomaly detection, diversity measurement and classification. Each one turns a question about meaning into a question about distance. Clustering groups nearby vectors. Classification assigns the label whose example vectors sit closest. Anomaly detection flags vectors with no close neighbors.

The map in the figure draws that idea in two dimensions. Drink terms form one neighborhood, vehicle terms form another, and the query lands inside the drink neighborhood. Every neighborhood on such a map starts with a model turning a piece of text into a single vector.

How does an embedding model turn text into a vector embedding?

An embedding model turns text into a vector by splitting it into tokens, encoding each token in context, pooling the token vectors into one vector and usually normalizing the result. The Hugging Face model card for all-MiniLM-L6-v2 shows each step in code. A tokenizer splits the sentence. A Transformer produces one vector per token. Mean pooling averages those token vectors and uses the attention mask to skip padding. A final L2 normalization scales the result to length 1.

Tokens and embeddings are different objects. A token is an integer ID for a word piece. An embedding is the learned vector the model builds from many tokens together. Ashish Vaswani and seven coauthors described the first step in “Attention Is All You Need” at NeurIPS 2017. Learned embeddings convert input tokens to vectors, 512 numbers wide in their base model. Tokenization versus embedding separates the two when debugging input limits and vocabulary.

Input length is capped in tokens, not characters. The all-MiniLM-L6-v2 card states that input longer than 256 word pieces is truncated by default. OpenAI lists a maximum input of 8,192 tokens for its text-embedding-3 models and recommends the cl100k_base tokenizer for counting them. Text past a truncating model’s cap never reaches the vector. Long documents are therefore split before encoding, and chunking for semantic search covers sizes and overlap.

Queries pass through the same model as documents, because only vectors from one model share one space. Queries and documents are still different kinds of text, such as a six-word question against a 200-word passage. Whether a model should treat both the same way is the subject of asymmetric and symmetric semantic search. The pipeline figure traces the full path from text to one stored vector. That path only produces useful vectors after training, and the training objective decides what “close” means.

How are embedding models trained to place similar meanings together?

Embedding models are trained on a prediction task whose loss rewards nearby vectors for related inputs and distant vectors for unrelated ones. Nobody writes the numbers by hand. Hand-engineered features, where experts choose what each number measures, still exist in some fields. Modern text embeddings instead learn their numbers from examples. The training objective decides what “related” means.

Mikolov and colleagues at Google introduced two word2vec objectives in 2013. Continuous bag-of-words predicts a word from its neighbors, while Skip-gram predicts the neighbors from the word. Their paper reports learning high-quality vectors from a 1.6 billion word data set in under a day. It also reports that vector(“King”) minus vector(“Man”) plus vector(“Woman”) lands closest to the vector for “Queen.”

Three objectives show the range of training signals behind common embeddings.

Sentence embedding models such as all-MiniLM-L6-v2 add that contrastive stage after pre-training. The card says the model computes the cosine similarity of every sentence pair in a batch and is penalized when the true pair does not win. Its training data totals 1,170,060,424 pairs from sources such as Reddit comments, citation pairs and question-answer sets. CLIP applied the same idea across modalities. Alec Radford and coauthors trained an image encoder and a text encoder jointly on 400 million image and text pairs collected from the internet.

The objective explains a model’s blind spots. A model trained on web question-answer pairs learns what those pairs treat as related. A legal or clinical corpus may rely on relations that never appeared in training. Continued training on in-domain pairs is one remedy, covered in fine-tuning embedding models. Training fixes where vectors land, and search then needs a rule for measuring how far apart two of them are.

How do search systems measure similarity between two vector embeddings?

Search systems measure similarity between two vector embeddings with cosine similarity, dot product or Euclidean distance, and the right metric is the one the model was trained with. Weaviate’s documentation states that each vectorizer model is trained with a specific distance metric and that search needs the same one. Weaviate uses cosine distance by default because it is the typical metric for vectorizer models.

Cosine similarity divides the dot product of two vectors by the product of their lengths: cos(a, b) = (a · b) / (|a| |b|). It measures the angle and ignores length. The value is 1 for the same direction, 0 at a right angle and -1 for opposite directions. Weaviate reports cosine distance, defined as 1 minus cosine similarity, so its values run from 0 to 2 and smaller means closer. Dot product keeps the length effect. Euclidean distance measures the straight line between the two points.

Length is where the metrics disagree. In the calculator’s default example, the query scores a dot product of 1.00000 with both “coffee” and “tea.” Cosine still ranks coffee first, at 0.99753 against 0.99263 for tea. Tea’s vector is longer, which inflates its dot product without bringing it closer in direction. Normalization removes that gap. OpenAI states that its embeddings are normalized to length 1, so cosine similarity and Euclidean distance produce identical rankings. The guide to cosine similarity, dot product and Euclidean distance compares the three in depth.

The calculator applies the three formulas to a three-number query and three illustrative documents. Moving a slider changes the query and reorders the cosine bars. The same arithmetic works on any embedding, whatever kind of input produced it.

Query value 1:

0.75

Query value 2:

0.65

Query value 3:

0.10

Which types of vector embeddings do search systems use?

Search systems use embeddings that differ by input, by structure and by numeric element type. By input, a model may encode a word, a sentence, a passage, an image or audio. By structure, an object may get one dense vector, one sparse vector or a set of vectors. By element type, each number may be stored as a 32-bit float, a smaller float, an 8-bit integer or a single bit.

Word embeddings give each vocabulary word one vector, so “bank” gets the same vector beside a river and beside a loan. Sentence and passage embeddings encode the whole input, so the same word contributes differently in different contexts. Nils Reimers and Iryna Gurevych built Sentence-BERT in 2019 to produce sentence vectors that can be compared with cosine similarity. The trade-offs are set out in word versus sentence embeddings.

Multimodal models place different input types in one shared space. CLIP trained its two encoders to maximize the cosine similarity between each image and its own caption. A text query can then retrieve images. Weaviate exposes this kind of search through nearImage and nearVideo queries alongside nearText. Models, inputs and failure cases are covered in multimodal embeddings.

The main embedding types differ in what one stored item holds.

Element type sets storage cost and precision. Weaviate’s quantization documentation states that embeddings are typically stored as 32-bit floats, 4 bytes per number. Weaviate’s scalar quantization maps each value to an 8-bit integer, a 4x reduction, and binary quantization keeps 1 bit per dimension, a 32x reduction. Both methods are lossy. pgvector offers a half-precision halfvec type and a binary bit type beside its standard vector type. Whatever the type, every vector needs a home that can store it and search it quickly.

Where are vector embeddings stored and how much space does each embedding take?

Vector embeddings are stored next to their source objects in a vector database or a vector-capable database, and each float32 dimension costs 4 bytes before index overhead. Weaviate stores each object together with its vector, produced by the collection’s vectorizer model. pgvector adds a vector column to Postgres tables, so embeddings sit beside the rest of the row.

A plain list of vectors is searchable, but a scan compares the query against every row. Vector indexes avoid that cost. HNSW, a layered graph, is Weaviate’s default index type, and Weaviate also offers flat, dynamic and HFresh indexes. Weaviate notes that the HNSW index is held in memory, which makes vector size a memory question as well as a disk question. The speed and recall trade is explained in approximate nearest neighbor search.

Size follows from simple arithmetic. A float32 vector of 1,536 dimensions needs 1,536 × 4 = 6,144 bytes. pgvector documents 4 × dimensions + 8 bytes per vector, so the same embedding takes 6,152 bytes in Postgres. One million such vectors need about 6.1 GB before any index. The table applies the 4-byte rule to three documented model sizes.

Doubling the dimensions doubles raw storage, and the same multiple carries into memory for an in-memory index. Embedding quantization trades precision for space. The vector storage and memory calculator runs the numbers for a given corpus size and dimension count.

The phrase “embedded vector database” means something else. It describes a database that runs inside the application process instead of as a separate server. Embedded Weaviate is one example. Weaviate’s documentation describes it as an experimental deployment model started from application code. It writes to a permanent datastore at persistence_data_path and exits with the client, while the data persists. An “embedding database” simply means any store that holds embeddings.

Persisting embeddings matters because generating them costs time and money. Weaviate’s documentation notes that reusing vectors from the same model saves both. Large corpora are encoded in bulk, a process covered in batch embedding. Picking the store itself is covered in choosing a vector database. Storage cost depends on dimension count, one of several embedding decisions with a direct effect on quality.

Which embedding decisions change semantic search quality?

Model choice, dimension count, compression, domain fit and query handling are the embedding decisions that change semantic search quality. Each one trades relevance against cost or speed. None has a universal answer. The deciding test set is the system’s own queries.

Model choice comes first because every other setting depends on it. The Massive Text Embedding Benchmark (MTEB), published in 2022 by Niklas Muennighoff, Nouamane Tazi, Loïc Magne and Nils Reimers, spans 8 task types, 58 datasets and 112 languages. Its authors found that no single text embedding method dominated across all tasks. A leaderboard rank is therefore a shortlist, as the guide to the MTEB leaderboard explains. The full selection process is set out in choosing an embedding model.

Dimension count trades detail for storage. OpenAI’s guide states that larger embeddings generally cost more compute, memory and storage. Its text-embedding-3 models accept a dimensions parameter that shortens the vector. The guide reports that text-embedding-3-large shortened to 256 dimensions still outperforms the 1,536-dimension text-embedding-ada-002 on MTEB. Aditya Kusupati and colleagues published a training method with the same goal in 2022, called Matryoshka embeddings, in which one vector encodes information at several granularities.

Dimensions also have to fit the store. Text-embedding-3-large returns 3,072 dimensions by default. pgvector’s HNSW index accepts standard vector columns of up to 2,000 dimensions and halfvec columns of up to 4,000. OpenAI’s guide gives the matching remedy: request 1,024 dimensions for a store limited to 1,024. Quantization is the other size lever, at a recall cost that depends on the method.

Domain fit and query handling complete the list. A general model trained on web text may place specialist terms poorly, which is the case for domain-specific embeddings or fine-tuning. Short questions matched against long passages raise the asymmetric-search choice. The table pairs each decision with the evidence that settles it.

Every row ends in measurement because published scores describe someone else’s data. A wrong call on any of these decisions produces failures that look alike from the outside: relevant documents missing, and confident neighbors that do not answer the query.

When do vector embeddings fail in semantic search?

Vector embeddings fail when models are mixed, when exact strings matter more than meaning, when text exceeds the input limit, or when training data is biased or off-domain. Mixed models are the silent failure. Weaviate’s documentation describes each named vector as its own vector space, independent of the others. Weaviate also makes a collection’s vectorizer immutable, so changing it requires a new collection and a migration. Re-embedding when the model changes plans that work.

Exact identifiers expose a different weakness. A part number, an error code or a new product name has little learned meaning. Its nearest neighbors may be similar-looking strings rather than the exact match. Weaviate’s documentation adds a related warning: every vector search returns its closest objects, even when none is relevant. Its example query for “SkyBlue” against “Red”, “Crimson” and “LightCoral” still returns “Red.” Limits, thresholds and filters guard against that, and hybrid search adds a keyword route for exact strings.

Input limits fail quietly. A model that truncates at 256 word pieces, as all-MiniLM-L6-v2 does by default, ignores the rest of a long passage. Knowledge also stops at training time. OpenAI states that its text-embedding-3 models lack knowledge of events after September 2021. OpenAI calls this a smaller limit for embeddings than for text generation, though it can reduce performance in edge cases.

Bias travels with the training data. Tolga Bolukbasi and four coauthors showed in 2016 that word embeddings trained on Google News articles carry gender stereotypes. The bias forms a measurable direction in the vector space. Search ranks by those distances, so a biased space can reorder results along the same lines. Domain mismatch works the same way: a clinical term placed by general web text may sit near the wrong neighbors. Those limits apply to every embedding, including the vectors inside large language models.

Do LLMs and ChatGPT use vector embeddings?

Large language models use vector embeddings internally, but those token vectors are not the sentence embeddings that retrieval systems store. A Transformer converts each input token into a learned embedding before attention runs, as Vaswani and coauthors described in 2017. OpenAI’s GPT-4 Technical Report (2023) calls GPT-4 a Transformer-style model pre-trained to predict the next token. The same report compares GPT-4 with earlier ChatGPT versions based on GPT-3.5, so the models behind ChatGPT embed tokens this way.

Internal token vectors make weak retrieval vectors on their own. Reimers and Gurevych tested this with BERT in their 2019 Sentence-BERT paper. Averaging BERT’s output vectors reached an average correlation of 54.81 across semantic textual similarity tasks. The CLS-token vector reached 29.19. Both trailed simple averaged GloVe word vectors, which reached 61.32. Retrieval therefore uses models trained for the job, such as sentence-transformers models or OpenAI’s text-embedding-3 models.

Chat answers meet retrieval embeddings through retrieval-augmented generation (RAG). An application embeds its documents, embeds each question with the same model, retrieves the nearest passages and passes them to the language model as context. OpenAI’s embeddings guide frames embeddings-based search as the alternative to placing all information in the context window, which raises token costs. The comparison of semantic search and RAG separates the retrieval step from the generation step.

A vector embedding is a fixed-length list of numbers that places related meaning close together, and semantic search ranks by that closeness. The list is only as useful as the model that produced it, the metric that compares it and the store that holds it. A working system keeps queries and documents in one model’s space, matches the metric to the model and sizes dimensions to its index. The decision that follows is the model itself, tested on real queries before any corpus is embedded.