Home / Foundations / High-Dimensional Vector Space Explained

High-Dimensional Vector Space Explained

A high-dimensional vector space is a space whose points are described by many coordinates, often hundreds or thousands, instead of the two or three we can picture. Embedding models place each text, image or sound at such a point, for example a list of 384 or 1,536 numbers, so that similar meanings sit near each other. Distances and angles work exactly as they do in two or three dimensions; the arithmetic just runs over more numbers. What changes is intuition. In high dimensions, randomly chosen directions are almost always nearly perpendicular, most of a shape’s volume sits near its surface, and the gap between the nearest and farthest neighbour can shrink. These effects, grouped under the curse of dimensionality, explain why semantic search uses cosine similarity, why exact search gets expensive, and why approximate indexes such as HNSW exist.

What is a high-dimensional space?

A high-dimensional space is a mathematical space in which each point needs many coordinates to describe it, one per dimension. “High” has no fixed threshold, but in machine learning it usually means tens to thousands of dimensions.

A point on a map needs two numbers, east and north. A point in a room needs three. Nothing in the mathematics stops at three: a point can have four, ten or 1,536 coordinates, and distances and angles are computed the same way, by summing over every coordinate. The space cannot be drawn, but it can be calculated with.

Four-dimensional, five-dimensional or ten-dimensional spaces are simply spaces whose points have four, five or ten coordinates. In physics a fourth dimension often means time; in data, extra dimensions are just more measured or learned features. In semantic search, the dimensions come from an embedding model, which is where high-dimensional spaces earn their place. The vector space model covers the older idea that started it.

Why do embeddings live in high-dimensional vector spaces?

Embeddings live in high-dimensional vector spaces because meaning has many independent aspects, and each extra dimension gives the model room to separate texts that differ in some way. A few dimensions could not keep millions of meanings apart.

An embedding model learns to place texts so that related ones are close and unrelated ones are far apart. Topic, tone, language, entity, tense and countless subtler features all affect what counts as related. With only three dimensions, many different meanings would be forced into the same neighbourhood. With hundreds, the model can encode many distinctions at once. No single dimension usually means one human concept; meaning is spread across them.

The history of word and sentence vectors shows the trend. Classic term vectors had one dimension per vocabulary word, tens of thousands of sparse dimensions; latent semantic indexing compressed them to a few hundred. Word2vec and GloVe learned dense word vectors of a few hundred dimensions, and transformers and BERT and Sentence-BERT brought dense sentence vectors. More dimensions are not free, though, and the space behaves in surprising ways.

How do distances behave in high-dimensional space?

Distances in high-dimensional space behave unintuitively: random points tend to be roughly equally far apart, and random directions are almost always close to perpendicular. Real embeddings are not random, which is why search still works.

A simple calculation shows the effect. For two random unit vectors in d dimensions, the cosine similarity has a mean of 0 and a standard deviation of 1 divided by the square root of d. In 3 dimensions that spread is about 0.58, so random directions can easily point the same way. In 384 dimensions it is about 0.05, and in 1,536 about 0.03, so two unrelated random vectors almost always have a cosine near 0.

Distances show a related pattern. Beyer and colleagues’ 1999 paper “When Is Nearest Neighbor Meaningful?” proved that for many data distributions, as dimensions grow, the distance to the nearest point and to the farthest point become nearly equal. If everything is about equally far away, “nearest” stops meaning much. These effects are the core of the curse of dimensionality, covered in the curse of dimensionality. Embeddings escape the worst of it because their structure is far from random.

Volume behaves strangely too. In a high-dimensional ball, almost all the volume lies in a thin shell near the surface, because the fraction of volume within radius r shrinks as r raised to the power of the dimension count. At 384 dimensions, a ball shrunk to 99 percent of the radius holds only about 2 percent of the original volume, since 0.99 to the power 384 is roughly 0.02. Random points therefore cluster near the edge, which is another way of seeing why they all sit at similar distances.

Why does semantic search still work in high dimensions?

Semantic search still works in high dimensions because embeddings are not scattered randomly; training packs related meanings into small, structured regions, so nearest neighbours remain meaningful. The useful structure has far fewer effective dimensions than the vector length.

Real data tends to lie near lower-dimensional structures inside the full space, often described as a manifold. Two paraphrases of the same question sit much closer to each other than the random baseline suggests, while unrelated texts sit near the typical distance. The gap between those two is what retrieval relies on. Studies of language model representations have also found them anisotropic, occupying a narrow cone rather than the whole space, which is why raw similarity scores can look high even between unrelated texts.

That last point has a practical consequence. Absolute similarity values depend on the model, so a threshold that works for one model fails for another; only the ranking within one model’s space is reliable. Semantic similarity vs relatedness covers what closeness does and does not mean. The geometry also explains the choice of distance measure.

Which distance measures work in high-dimensional spaces?

Cosine similarity, dot product and Euclidean distance all work in high-dimensional spaces, and for vectors normalized to unit length they produce the same ranking. Cosine is the most common choice for text embeddings.

Cosine similarity measures the angle between two vectors, ignoring their length. That suits embeddings, where direction carries the meaning and length often reflects incidental factors. Dot product equals cosine when vectors have unit length and is cheaper to compute. Euclidean distance on unit vectors is a fixed function of cosine, so it ranks results identically. Weaviate’s default distance metric for its HNSW index is cosine, according to its documentation.

Use the measure the embedding model was trained with, and normalize vectors when the model expects it. Mixing measures, or comparing unnormalized vectors with cosine-tuned thresholds, produces confusing results. The full comparison is in cosine vs dot product vs Euclidean distance. Computing any of them over millions of high-dimensional vectors is the next problem.

How do you search a high-dimensional vector space efficiently?

You search a high-dimensional vector space efficiently with approximate nearest neighbour indexes, such as HNSW graphs, which find very close neighbours without comparing the query with every vector. Exact search becomes too slow at scale.

Exact search computes the distance from the query to every stored vector: millions of vectors times hundreds of dimensions per query. Classic space-partitioning trees that work well in two or three dimensions lose their advantage in high dimensions, because almost every region must be checked. Approximate methods trade a little accuracy for large speed gains. HNSW, which Weaviate uses by default, builds a layered graph that a search walks towards the query in a small number of steps.

Memory follows dimensions directly: each float32 dimension takes 4 bytes, so a 1,536-dimension vector takes 6,144 bytes before compression. That is why quantization and smaller models matter for large collections. Approximate nearest neighbour search covers the algorithms. Choosing how many dimensions to use is the last practical question.

How many dimensions should embeddings have?

Embeddings should have as many dimensions as the chosen model produces, and the choice of model should weigh retrieval quality against the memory and speed cost of more dimensions. More dimensions are not automatically better.

Larger embeddings can encode finer distinctions, but beyond a point the gain is small while storage and search costs keep rising in proportion. Some models are trained so their vectors can be truncated to fewer dimensions with modest quality loss, known as Matryoshka embeddings. Testing a smaller and a larger model on your own queries usually settles the question better than general advice.

The same high-dimensional ideas reach well beyond embeddings. They shaped information retrieval from the start, sit behind semantic search at Google, and complement structured approaches such as knowledge graphs in search. Caching systems that match similar queries rely on the same distances, as described in cache hits and misses.

A high-dimensional vector space is a space whose points have many coordinates, such as the 384 or 1,536 numbers of an embedding. The arithmetic of distance and angle is the same as in two or three dimensions, but intuition breaks. Random directions become nearly perpendicular, with cosine spread of about 1 divided by the square root of the dimension count, and nearest and farthest distances can converge. Semantic search works because embeddings are structured, not random, packing related meanings close together. Use the model’s own similarity measure, rely on approximate indexes such as HNSW for speed, and weigh extra dimensions against their memory cost.