Home / How it works / Multilingual and Cross-Lingual Semantic Search Explained

Multilingual and Cross-Lingual Semantic Search Explained

Multilingual semantic search finds documents by meaning in many languages, and cross-lingual search goes further, letting a query in one language retrieve documents written in another. Both rely on multilingual embedding models, which place sentences with the same meaning close together whatever their language. A German question about cancelling an order can then match an English help page. The alternatives are translating documents into one language, translating queries at search time, or keeping a separate index per language. Multilingual embeddings are usually the simplest, but they can be weaker for rare languages. The keyword half of hybrid search still needs language-aware tokenization. Weaviate, for example, offers tokenizers for Chinese, Japanese and Korean, plus per-property stopword lists. Evaluate with real queries in every language you serve, including mixed-language queries.

What is multilingual and cross-lingual semantic search?

Multilingual semantic search handles queries and documents in several languages, and cross-lingual semantic search matches a query in one language to documents in another. The second is a stronger form of the first.

A multilingual site might serve French queries over French pages and English queries over English pages. That is multilingual but not cross-lingual. Cross-lingual search lets the French visitor find an English page when no French version exists. It matters for global help centres, research archives, legal and patent collections, and any company whose documents are spread across languages.

Both build on semantic search, covered in how semantic search works, but they need models trained on many languages. The choice of model and approach depends on which languages matter, how much content each has, and whether visitors expect results in their own language only.

How does cross-lingual semantic search work?

Cross-lingual semantic search works by embedding queries and documents with a multilingual model that maps the same meaning in different languages to nearby points in one vector space. Search then proceeds as usual.

Multilingual models learn this alignment from parallel text, such as translated sentence pairs, alongside large amounts of single-language text. LaBSE, from Feng and colleagues at Google, was trained for sentence alignment across 109 languages. Reimers and Gurevych showed in 2020 that an English sentence model can be made multilingual by distillation. A student model learns to place translations at the same point as the English teacher’s embedding.

Once texts share a space, nothing else changes. Documents are embedded and indexed once, queries are embedded at search time, and nearest neighbours are returned whatever their language. Dense vectors and vector search cover the mechanics.

What are the ways to search content in many languages?

The ways to search content in many languages are multilingual embeddings, translating documents into one language, translating queries at search time, and keeping a separate index per language. Each has a different cost profile.

Multilingual embeddings need one model and one index, and they support cross-lingual results directly. Translating every document into one pivot language lets you use a strong single-language model, but it costs translation at import and can introduce errors. Translating queries is cheaper to set up but adds latency and can lose nuance in short queries. A separate index per language with a specialist model can give the best quality per language, but it cannot return results across languages.

Many teams combine approaches: a multilingual model for retrieval, a language filter when users want results in their own language only, and translation of retrieved snippets for display. Language is usually stored as a property, so filtering by it is exact, as covered in metadata filtering.

The right mix depends on the content. A help centre with full translations of every page can filter to the visitor’s language and rarely needs cross-lingual matching. A research archive with papers in many languages benefits most from cross-lingual retrieval, because the best source may exist in only one language. A store with product data in one language and visitors in many can embed queries multilingually and still show product pages unchanged. Decide which case you are in before choosing an approach.

How does hybrid search work across languages?

Hybrid search across languages combines multilingual vector search with keyword search that uses the right tokenizer for each language. The keyword side is language-specific even when the vectors are not.

BM25 needs text split into words, and languages differ. Chinese and Japanese do not separate words with spaces, so whitespace tokenization fails for them. Weaviate offers language-aware tokenizers for these cases, the gse tokenizers for Japanese and Chinese and the kagome tokenizers for Japanese and Korean. It also offers trigram tokenization as a fallback. Per-property stopword presets and accent folding help with European languages, where “café” and “cafe” should match.

A common design stores text in a property per language, or tags each object with its language, so each can use the right tokenizer. The vector side handles cross-lingual matching, and the keyword side handles exact names and terms within each language. What is hybrid search covers how the two are fused, and sparse vectors covers the keyword representation.

Which embedding models support multilingual search?

Embedding models that support multilingual search include open models such as LaBSE, multilingual E5 and multilingual Sentence-Transformers models, and many commercial embedding APIs. Coverage and quality vary by language.

Check each candidate model’s list of training languages, and its scores on multilingual retrieval benchmarks such as MIRACL, from Zhang and colleagues, which covers 18 languages. Weaviate’s integrations include hosted multilingual models from providers such as Cohere and Google, and local models through its Transformers integration. Its configuration examples include a multilingual MiniLM model.

Benchmarks give a shortlist, not a decision. Performance on a language with little training data can be much weaker than on English, and domain vocabulary may not transfer. Test the top candidates on your own queries in each language, as described in choosing an embedding model. Tokenization also differs by language, as covered in tokenization vs embedding.

What problems does multilingual semantic search have?

Multilingual semantic search has problems with uneven quality across languages, same-language bias in rankings, mixed-language text, and missing keyword support for some scripts. Most problems show up only when every language is tested.

Some multilingual models rank documents in the query’s own language above better answers in another language, because shared surface features add similarity. If users want the best answer regardless of language, check for this bias. Mixed-language queries, common in technical fields, may be embedded unpredictably. Names and product codes behave the same in every language and are better served by keyword search.

Length limits also differ in practice: some scripts produce more tokens per word, so long documents in those languages are truncated sooner. Chunk by meaning rather than by raw characters, as covered in chunking for semantic search. Synonyms and related terms vary by language too, covered in synonyms and related terms.

How do you handle multilingual embeddings for global content?

You handle multilingual embeddings for global content by choosing one multilingual model, storing language as a property, using language-aware keyword tokenization, and evaluating each language separately. Add translation only where quality demands it.

Start with a single multilingual model and one collection. Store a language property on every object and let users filter by it when they want local results. Configure keyword tokenization per language where your database allows it. Build a test set of real queries in each major language, judge results, and compare languages on the same metrics. If one language lags, try a specialist model or translation for that language only.

Keep the rest of the pipeline consistent across languages. Relevance scores can differ in scale between languages, so avoid fixed thresholds. Reranking with a multilingual cross-encoder can lift precision, and semantic matching explains what the scores capture. Semantic search architecture shows where each piece sits, and top-k retrieval covers how many results to pass on.

How do index and storage choices change with many languages?

Index and storage choices change little with many languages when one multilingual model is used, because all documents share one vector index. Per-language designs multiply indexes.

One multilingual collection means one HNSW index, one set of settings and one memory budget, covered in semantic indexing and vector indexes and approximate nearest neighbor search. Per-language collections add an index each, which is simpler to tune per language but costlier to run. Exact search for small languages, covered in exact vs approximate search, can keep costs down.

Distance measures behave the same across languages, as covered in cosine vs dot product vs Euclidean distance. Similarity search and dense vs sparse vectors describe the retrieval side in general.

Multilingual semantic search serves queries and documents in many languages, and cross-lingual search matches across them using a multilingual embedding model that maps the same meaning to nearby vectors. Alternatives are translating documents, translating queries or keeping an index per language. Hybrid search still needs language-aware keyword tokenization, such as Weaviate’s gse and kagome tokenizers for Chinese, Japanese and Korean. Store language as a property, use one model where possible, and evaluate each language separately, since quality varies by language.