Home / Embeddings / Domain-Specific Embeddings for Semantic Search

Domain-Specific Embeddings for Semantic Search

Domain-specific embeddings are vector representations from a model trained or adapted on text from one field, such as medicine, law, finance or source code, so that similarity reflects how that field uses its words. General embedding models learn from broad web text and can blur distinctions that matter to specialists, or split rare technical terms into meaningless pieces. Before reaching for a specialist model, measure the gap. Run a general model on your own queries with relevance judgments, then try hybrid search and a reranker, which keep the existing index. If a measured gap remains, switch to a specialist model, adapt a general one without labels, or fine-tune it on query and passage pairs. Each of those changes the vectors and means re-embedding the collection. Weaviate integrates specialist models such as voyage-law-2, voyage-finance-2 and voyage-code-2 through its Voyage AI vectorizer.

What are domain-specific embeddings?

Domain-specific embeddings are embeddings produced by a model whose training data or training objective focuses on one field, so its vector space reflects that field’s vocabulary and notion of relevance. They contrast with general-purpose embeddings trained on mixed web, forum and question-answer data.

In a general model, “discharge” sits near meanings from electricity and employment as much as from hospitals. A clinical model learns that “discharge summary” and “patient release note” are close. In law, “consideration” means something exchanged under a contract, not thoughtfulness. Code models learn that “list comprehension” relates to a pattern of square brackets, not a reading test.

The term covers several things in practice: models pretrained on domain text, retrieval models trained on domain query logs, and general models fine-tuned on domain pairs. What they share is that the domain shaped the training. Embeddings in general are introduced in what are embeddings.

When do general embedding models fail a domain?

General embedding models fail a domain when its words carry meanings their training data rarely showed, or when key terms are rare or tokenized poorly. They also fail when relevance follows rules general data does not teach. The failure shows up as plausible but wrong neighbours.

The BEIR benchmark, published by Thakur and colleagues in 2021, measured this zero-shot. Dense retrievers trained on MS MARCO web questions were tested on biomedical, scientific, financial and argument retrieval sets. Many of them fell behind plain BM25 on these out-of-domain sets, even though they beat it in domain. Keyword matching transfers because it needs no learned notion of meaning.

Typical symptoms in a domain are easy to spot in a test run. Exact identifiers such as drug codes, case citations or function names are missed. Documents that share topic words but answer a different question rank high. Rare terms are split by the tokenizer into fragments, as described in tokenization vs embedding. Abbreviations with several domain meanings collapse together.

How do you measure whether you need a domain model?

You measure the need for a domain model by building a test set of real domain queries with relevance judgments and comparing retrieval quality across candidate models and setups. Leaderboard scores on general data do not predict domain performance.

A few hundred queries drawn from search logs or written by domain experts, each with judged relevant documents, are enough to compare options. Measure recall at the candidate depth you will use and a ranking metric such as nDCG at 10. The method is covered in search test sets and relevance judgments.

Include BM25 and hybrid search in the comparison, not only embedding models. If hybrid search with the general model closes most of the gap, the cheapest fix is already found. Public leaderboards such as MTEB report scores across many tasks, including some domain sets, but averages hide domain weaknesses, as explained in the MTEB leaderboard.

Can hybrid search and reranking replace a domain model?

Hybrid search and reranking often close much of the domain gap without changing the embedding model, because keyword matching catches exact domain terms and a reranker judges relevance more finely. Neither requires re-embedding the collection.

BM25 handles exactly the cases where general embeddings struggle: rare identifiers, codes and technical terms that appear verbatim in documents. Fusing BM25 with vector results lets each catch what the other misses, as explained in hybrid search. In Weaviate, this is a single hybrid query with an alpha weight that can be tuned on the test set.

A cross-encoder reranker on top re-scores the shortlist with full attention over query and document, which handles many domain distinctions the bi-encoder misses. Rerankers can also be specialist or fine-tuned models, which is often cheaper than changing the embeddings. If the measured gap remains after both, change the embedding model itself.

Which specialist embedding models are available?

Specialist embedding models exist for medicine and biology, law, finance, science and source code, offered both as open models and through embedding APIs. Check whether a model was trained for retrieval, not just for language understanding.

Many well-known domain models are language models rather than retrieval models. BioBERT, from Lee and colleagues in 2019, continued BERT pretraining on PubMed abstracts and PMC articles. SciBERT, from Beltagy and colleagues in 2019, trained on scientific papers with its own vocabulary. Legal-BERT, from Chalkidis and colleagues in 2020, trained on legal text. Used as-is, their pooled outputs are weak sentence embeddings; they need contrastive retrieval training first.

Retrieval-ready specialists exist too. MedCPT, from Jin and colleagues in 2023, trained biomedical query and article encoders on PubMed search logs. Commercial APIs offer domain models: Weaviate’s Voyage AI integration lists voyage-law-2, voyage-finance-2 and voyage-code-2, and its Jina AI integration lists a code model. How to compare candidates fairly is covered in choosing an embedding model.

How can you adapt a general model without labelled data?

You can adapt a general embedding model without labelled data by continuing training on unlabelled domain text, either with a denoising objective such as TSDAE or with synthetic queries as in GPL. Both use only the documents you already have.

GPL, Generative Pseudo Labeling, was published by Wang, Thakur, Reimers and Gurevych in 2022. A question generation model writes synthetic queries for domain passages. Negative passages are mined with an existing retriever, and a cross-encoder scores each pair. The embedding model is then trained to match those scores. The paper reported clear gains on several BEIR domains without any human labels.

TSDAE, from Wang, Reimers and Gurevych in 2021, trains a model to reconstruct sentences from corrupted versions, learning domain vocabulary from raw text. Domain-adaptive pretraining, studied by Gururangan and colleagues in 2020 in “Don’t Stop Pretraining”, continues language model pretraining on domain text before task training. These methods suit domains with plenty of documents but no query logs.

When should you fine-tune on labelled domain pairs?

Fine-tune on labelled domain pairs when you have real queries matched to relevant passages, from search logs, support tickets or expert annotation, and unsupervised options leave a gap. Real pairs teach the model your users’ notion of relevance directly.

Fine-tuning typically uses a contrastive loss with in-batch negatives and mined hard negatives, starting from a strong general model. Hard negatives from the domain matter most: passages that look relevant but are not. Hold out part of the test set, and keep the general model’s scores as the baseline to beat. The full process is in fine-tuning embedding models.

Watch for overfitting to the domain at the cost of everything else. A model tuned narrowly on contract clauses may get worse on the general questions users also ask. Testing on a mixed set guards against that. Some teams keep two named vectors, a general and a domain one, and query both.

How do you deploy domain-specific embeddings in Weaviate?

In Weaviate you deploy domain-specific embeddings either through a model integration that names the specialist model, or by importing vectors from your own fine-tuned model. Either way, all objects in one vector space must come from the same model.

from weaviate.classes.config import Configure, Property, DataType

client.collections.create(
    "Contract",
    vector_config=[
        Configure.Vectors.text2vec_voyageai(name="legal", model="voyage-law-2"),
    ],
    properties=[Property(name="clause", data_type=DataType.TEXT)],
)

For a custom fine-tuned model, use self_provided vectors and embed queries with the same model in the application, or serve the model through a local inference integration. Named vectors let one collection hold a general and a domain vector side by side, which makes comparison and gradual rollout easier. The dimensions and memory of each choice are covered in embedding dimensions.

Moving to a domain model means re-embedding every object, because vectors from different models are not comparable. Plan it like any model change, with a parallel index and a cutover, as described in re-embedding when the model changes. Large collections should use batch embedding to keep cost and time down.

How do domain models relate to other embedding choices?

Domain models interact with other embedding choices: query and document asymmetry, sentence versus word granularity, dimension size, quantization and modality all still apply. A domain model solves vocabulary and relevance, not every other problem.

Domain search is often asymmetric, with short questions against long technical passages, so models trained for asymmetric semantic search suit it. Sentence-level models, explained in word vs sentence embeddings, are needed for retrieval. Domain models can be compressed like any other, as covered in embedding quantization and Matryoshka embeddings.

Some domains are not text-only. Radiology images, engineering drawings and product photos need image or multimodal models, covered in multimodal embeddings. Whatever the modality, the same rule applies: measure on real domain queries before and after every change.

Domain-specific embeddings come from models trained or adapted on one field’s text, so similarity follows that field’s vocabulary and relevance. General models often lose ground in new domains, as the BEIR benchmark showed. Measure first on real domain queries, then try hybrid search and a reranker, which keep the existing index. If a gap remains, switch to a retrieval-trained specialist such as voyage-law-2 or MedCPT, adapt without labels using GPL or TSDAE, or fine-tune on real pairs. Any model change means re-embedding the collection, which Weaviate’s named vectors and model integrations make easier to stage.