Home / Keyword search / Stemming, Lemmatization and Text Analyzers in Search

Stemming, Lemmatization and Text Analyzers in Search

Stemming and lemmatization reduce words to a common base form before indexing, so a keyword search for “connection” can also match “connected” or “connecting”. Stemming chops suffixes by rule: the Porter and Snowball stemmers both turn “connected” into “connect” and “studies” into “studi”. Lemmatization looks words up in a dictionary with their part of speech, so “mice” becomes “mouse” and “ran” becomes “run”. Both are steps in a text analyzer, the pipeline that also splits text into tokens, lowercases it, folds accents and drops stopwords before terms reach the inverted index. Stemming raises recall but can merge unrelated words; lemmatization is more precise but slower and language-specific. Weaviate’s keyword search documentation states that stemming is not supported yet, so applications that need it store a stemmed copy of the text. Its 1.37 text analyzer preview adds accent folding and custom stopwords, and the vector leg of hybrid search handles many word forms without stemming.

What are stemming and lemmatization?

Stemming and lemmatization are text normalisation methods that map different forms of a word to one shared form, so a keyword index treats them as the same term. Stemming uses rules; lemmatization uses a dictionary.

Without normalisation, keyword search is literal. The inverted index, described in the inverted index, stores each distinct token separately, so “connect”, “connected”, “connecting” and “connection” are four unrelated terms. A query for one finds documents with exactly that form and misses the rest.

A stemmer strips endings by rule to produce a stem, which need not be a real word: “studies” becomes “studi”. A lemmatizer returns the dictionary form, or lemma, using knowledge of the word’s part of speech: “studies” becomes “study”, and “ran” as a verb becomes “run”. Both are applied the same way at index time and query time, so the forms meet in the middle.

They tackle one slice of the gap between query words and document words, the slice caused by grammar rather than by different vocabulary. The wider problem is covered in the vocabulary mismatch problem.

How does stemming work?

Stemming works by applying an ordered set of suffix-stripping rules, such as removing “-ing”, “-ed” or “-ation” when the remaining stem is long enough, without consulting a dictionary. It is fast and needs no linguistic resources.

The Porter stemmer, published by Martin Porter in 1980, is the classic English example. It applies several rounds of rules, each checking the shape of the remaining stem before removing a suffix. Snowball, a later framework also by Porter, refines the English rules and provides stemmers for many other languages. In NLTK’s implementations, both turn “connected”, “connection” and “running” into “connect”, “connect” and “run”.

Rules make mistakes in both directions. Overstemming merges words that should stay apart: both stemmers reduce “university” and “universe” to “univers”, and Porter reduces “generous” and “generation” to “gener”, where Snowball keeps “generous”. Understemming leaves related forms apart: irregular words such as “ran” or “mice” pass through unchanged, because no suffix rule connects them to “run” or “mouse”.

How does lemmatization work?

Lemmatization works by identifying each word’s part of speech and looking up its dictionary form, so irregular forms such as “mice”, “ran” and “better” map to “mouse”, “run” and “good”. It needs a lexicon and usually a tagger.

Part of speech matters. “Saw” as a noun is its own lemma, while “saw” as a verb becomes “see”. “Better” as an adjective becomes “good”, but as a verb (“to better oneself”) it stays “better”. A lemmatizer therefore runs a part-of-speech tagger first, or accepts the part of speech as input.

The result is cleaner than a stem: real words, no overstemming of unrelated terms, and correct handling of irregular forms. The cost is speed, memory for the lexicon and dependence on language resources, which are rich for English and thin for many other languages. Unknown words, such as product names, are usually left unchanged.

Which is better for search, stemming or lemmatization?

Stemming is usually the better default for keyword search because it is fast and raises recall, while lemmatization suits cases where precision matters and good language resources exist. Many systems use a light stemmer as a compromise.

Search tolerates stems that are not words, because users never see them. What matters is whether the right documents share a term with the query. Stemming’s broader merging helps recall, at the price of some false matches from overstemming. A light stemmer that only removes plurals and common endings limits that risk.

Lemmatization helps when false merges are costly, such as in legal or medical text, or when irregular forms are common. It also produces terms that can be shown to users, for example in facets or suggestions. Both affect how BM25 counts term frequency, as described in the BM25 algorithm and TF-IDF, since merged forms count as one term.

What does a text analyzer do besides stemming?

A text analyzer also splits text into tokens, lowercases them, folds accents, removes stopwords and sometimes adds synonyms, and the same analyzer must be applied at index time and query time. Each step changes what can match.

Tokenization decides where terms begin and end, which matters for punctuation, hyphens and identifiers. Weaviate offers word, lowercase, whitespace and field tokenization, plus trigram and language-specific tokenizers. Its documentation shows “variable_name” becoming “variable” and “name” under word tokenization but staying whole under lowercase. These options are covered in full-text search.

Weaviate 1.37 added a text analyzer preview with two more steps. Accent folding, through textAnalyzer.asciiFold, makes “Café Crème” searchable as “cafe creme”. Custom stopword presets let different properties use different lists, such as French and English. Weaviate notes that stopwords are still indexed and only filtered at query time, so changing them does not require reindexing. A tokenize endpoint shows exactly which tokens a configuration produces.

How do you handle stemming in Weaviate?

You handle stemming in Weaviate by stemming text in your application, storing the result in a separate property, stemming queries the same way, and searching that property with BM25 or hybrid search. Weaviate’s BM25 does not stem natively yet.

import re
from nltk.stem import SnowballStemmer
from weaviate.classes.config import Configure, Property, DataType

stem = SnowballStemmer("english")
def stemmed(text):
    return " ".join(stem.stem(w) for w in re.findall(r"[a-z0-9]+", text.lower()))

docs = client.collections.create("Doc", vector_config=Configure.Vectors.self_provided(),
    properties=[Property(name="body", data_type=DataType.TEXT),
                Property(name="body_stemmed", data_type=DataType.TEXT)])

text = "The router connected after a firmware update"
docs.data.insert({"body": text, "body_stemmed": stemmed(text)}, vector=doc_vector)

res = docs.query.bm25(query=stemmed("connection problems"), query_properties=["body_stemmed"])

The pattern was run on Embedded Weaviate 1.30.5 with three short documents as an API check. A BM25 query for “connection problems” on the plain property matched only the document containing “Connection”; on the stemmed property it also matched “The router connected after a firmware update”. Keeping the original property lets exact-form searches still work, and boosting it can rank exact matches higher.

The feature request is tracked as Weaviate GitHub issue 2439, linked from the BM25 documentation.

Do embeddings make stemming unnecessary?

Embeddings make stemming less necessary for the vector side of search, because embedding models map different forms of a word to nearby vectors, but the keyword side still matches literal tokens. Hybrid search combines both behaviours.

Embedding models break words into subword pieces and learn from context, so “connected” and “connection” end up close without any explicit rule. A vector search for one finds documents using the other. This is part of why semantic search reduces the vocabulary gap, as explained in what is keyword search and its sibling guides.

The keyword leg of hybrid search still matters for exact names, codes and rare terms, and there literal matching can miss inflected forms. A stemmed property closes that gap for languages where stemming helps. Filters on text properties also match tokens literally, so a stemmed copy can make filters more forgiving too.

What can go wrong with stemming and lemmatization?

Stemming and lemmatization can go wrong by merging unrelated words, mangling names and codes, applying the wrong language’s rules, or being applied inconsistently between indexing and querying. Each causes silent relevance errors.

Overstemming is the classic failure. NLTK’s Porter and Snowball stemmers both reduce “organization” to “organ”, so a search for organs could match organisations. Product names, brands and identifiers should usually be stored in an unstemmed property, since stemming them adds nothing and can create false matches.

Over-eager normalisation can also hide meaning. “News” is not the plural of “new”, and both stemmers rightly leave it alone, but simpler plural-stripping rules can turn it into “new”. Short words, abbreviations and terms in other languages need the same caution, which is why many teams keep a list of protected words that the analyzer never changes.

Language and consistency matter as much as the algorithm. An English stemmer applied to French text produces nonsense stems. Changing the stemmer means reindexing, because old and new stems will not match. Test changes on real queries, comparing recall and precision before and after, rather than assuming more normalisation is always better.

Stemming and lemmatization map word forms to a shared base so keyword search can match “connected” to “connection”. Stemmers such as Porter and Snowball strip suffixes by rule, quickly but with overstemming such as “university” and “universe” both becoming “univers”. Lemmatizers use dictionaries and part of speech to return real lemmas such as “mouse” for “mice”. Both sit inside a text analyzer with tokenization, lowercasing, accent folding and stopwords. Weaviate’s BM25 does not stem yet, so store a stemmed copy and stem queries the same way, and rely on the vector leg of hybrid search for word forms in meaning-based matching.