Home / Keyword search / What Is Keyword Search?

What Is Keyword Search?

Keyword search, also called lexical search or keyword-based search, retrieves and ranks documents by matching analyzed query tokens with indexed text tokens. Analysis turns both query and stored text into comparable terms. A term-to-document index then finds candidates, and a relevance function orders them. This direct token evidence makes names, codes, quoted phrases, and domain language useful search keys. Different wording can still hide a relevant document because literal matching does not establish that two phrases mean the same thing.

What is keyword search, and how is it different from keyword research?

Keyword search retrieves documents from query terms, while keyword research discovers and evaluates terms for advertising or content planning. Keyword search, lexical search, and keyword-based search all describe a retrieval method based on words or other text tokens. Weaviate documentation defines it as exact matching based on tokens rather than whole raw strings.

A keyword can be one word, a phrase, an identifier, or an error string. Illustrative examples include refund, refund policy, SKU-4821, and ERR_CONN_RESET. Entering one of these expressions to retrieve matching records is keyword searching. Tokenization and query rules determine whether the system treats punctuation, spacing, or the entire value as significant.

Keyword research addresses another question: which terms people use and how those terms might support advertising or content planning. Google describes Keyword Planner as a free tool for discovering keywords and viewing search-volume estimates and forecasts. Google also says access requires completed account setup and billing information. That research workflow does not retrieve documents from a corpus.

Google Search is free to use, but it is not a pure exact-term engine. Google Search Help says related terms may produce results even when exact search terms do not appear. The distinction matters because knowing what retrieval means comes before examining how text becomes searchable terms.

How does keyword search turn text into searchable terms?

Keyword search analyzes both stored text and the query into comparable tokens before matching begins. A full-text search system applies analysis to document fields and query text. Collection starts with raw strings. Tokenization splits those strings, normalization can change case, and optional stopword handling can remove selected common words. The resulting indexed terms are what matching uses.

Raw strings, tokens, and indexed terms are not interchangeable. For example, punctuation can divide one raw value into several tokens, while lowercasing can make differently capitalized tokens comparable. Manning, Raghavan, and Schütze explain that document and query text should use the same tokenization for Boolean or free-text retrieval. Language and hyphen rules can therefore change what matches.

Product labels vary. Weaviate documents word tokenization as splitting on non-alphanumeric characters and lowercasing. Its lowercase option splits on whitespace and lowercases, whitespace preserves case, and field treats a full property value as one token. These are Weaviate behaviors, not universal analyzer names.

Stemming and lemmatization can map selected word forms toward a shared indexed form. Manning, Raghavan, and Schütze distinguish heuristic affix removal from vocabulary-based morphological analysis. Once analysis produces terms, the system needs a structure that records where each term occurs.

How does an inverted index find matching documents?

An inverted index maps each analyzed term to postings that identify documents containing that term. The inverted index contains a term dictionary and a postings list for each term. Manning, Raghavan, and Schütze describe index construction as collecting documents, tokenizing text, applying linguistic preprocessing, and creating this dictionary-plus-postings structure.

At query time, the analyzer produces query tokens and looks up each one in the dictionary. Their postings identify candidate documents. An operator can combine those postings by requiring every token, accepting any token, or excluding a token. This lookup avoids reading every document merely to learn whether it contains a term.

Indexes may be field-specific, and stored positions may support phrase or proximity constraints. Weaviate documentation says each processed text token points to objects that contain it. Query text uses the same configured tokenization, after which BM25F ranks the matching objects. Candidate retrieval decides what can be scored; it does not establish final order.

A relevant document that never enters the candidate set cannot rise through better scoring. After lookup establishes candidates, the next problem is deciding which matching document deserves the highest rank.

How do TF-IDF and BM25 rank keyword matches?

TF-IDF and BM25 reward informative query terms, while BM25 also limits repeated-term gains and normalizes for field or document length. TF-IDF combines term frequency with inverse document frequency. A term repeated in one document supplies local evidence, while a term found throughout the collection usually supplies less distinguishing evidence.

That distinction explains why a rare matched product code often contributes more useful ranking evidence than a ubiquitous word. It does not make every rare term relevant. Ranking still depends on the query, corpus, fields, and chosen scoring model. Term matching must first place the document in the candidate set.

BM25 adds term-frequency saturation and length normalization to lexical scoring. Saturation means each repeated occurrence contributes less additional evidence than the previous one. Length normalization limits the advantage a long field might receive merely because it contains more words. Robertson and Zaragoza describe BM25 as a term-weighting and document-scoring function in the probabilistic relevance framework.

Weaviate uses BM25F, which extends scoring across multiple fields and permits query-time property boosts. Its documented collection defaults are k1 = 1.2 and b = 0.75. In Weaviate, k1 controls saturation and b controls length normalization. These values are product defaults, not universal BM25 constants.

Scoring orders available matches but cannot rescue an absent candidate. That boundary makes query controls important because some controls change inclusion, while others change a candidate’s contribution to rank.

Which controls change what a keyword search can match?

Analyzers, operators, phrase rules, fields, boosts, wildcards, fuzzy matching, and filters change lexical candidate sets or their ranking. Controls differ by product, so visible syntax should be treated as an interface contract rather than a universal search language. Weaviate documents its own operator and tokenization defaults, and Elastic documentation says full-text queries process query strings with the field’s index analyzer.

Common controls affect retrieval in distinct ways:

The table separates candidate controls from ranking influence. Elastic documents fuzzy, phrase, and proximity behavior for its query types, while its wildcard documentation warns that leading wildcards can be resource intensive. Academic databases and job-search systems can expose similar ideas, but their fields and syntax vary.

Weaviate’s documented default operator is or, with one minimum matching token. Its and rule requires every query token within one searched property. Property boosts change each property’s BM25F contribution. These behaviors belong to Weaviate, not every lexical engine.

Google Search Help documents quotation marks for an exact phrase, site: for a domain restriction, and a leading minus sign for exclusion. Those concise controls help users express literal evidence. The next question is when that evidence matches the retrieval task.

When is keyword search the right retrieval method?

Keyword search is strongest when literal terms are decisive evidence and users need explicit control over inclusion. Names, product codes, legal citations, error strings, quoted phrases, tags, and established domain terminology carry value in their exact wording. Weaviate documentation specifically identifies exact domain terms, precise categories, and tags as keyword-search use cases.

A support engineer may search an error string, while a catalog user may search a stock identifier. A recruiter may restrict terms to title, skills, or description fields. In each case, the system can surface matched terms and fields when its interface provides that metadata. Such evidence is inspectable, but no implementation makes every literal match relevant.

A useful selection check asks three questions:

• Does the query contain a name, code, quotation, or domain term whose spelling carries meaning?
• Does the corpus use terminology consistently enough for query and document tokens to overlap?
• Can the application tolerate missing synonyms or differently worded expressions?

These questions tie method choice to query shape and corpus language. Field boosts can emphasize titles or identifiers, while filters can enforce categories, dates, or permissions. Still, precision is not guaranteed: a document may contain the requested literal string in an irrelevant context.

Lexical retrieval fits best when shared terms are dependable evidence. Its limits become visible when two people express the same need with different vocabulary.

Why can keyword search miss relevant documents?

Keyword search can miss relevant documents when query and corpus use different tokens for the same need. This vocabulary mismatch problem includes synonymy, abbreviations, spelling variation, multilingual wording, and different professional terminology. Manning, Raghavan, and Schütze explain that synonymy lowers recall when query and relevant document express one concept with different words.

The fixed illustrative example uses lowercase word tokenization, configured English stopword removal, and no synonym expansion, fuzzy matching, or vector retrieval. Query reset password overlaps both meaningful tokens in Document A, Reset your account password. It overlaps one token in Document C, Reset your router, and zero tokens in Document B, Change your login credentials.

Document B may satisfy the same information need, yet literal matching supplies no candidate evidence. Document C contains a query word but discusses another subject. The example shows why token overlap and semantic relevance are different judgments, without assigning invented BM25 scores.

Remedies target different gaps. Stemming or lemmatization handles selected word forms, not every synonym or concept gap. Fuzzy or trigram matching can tolerate spelling variation. Query expansion adds synonyms, although Manning, Raghavan, and Schütze warn that ambiguous expansions can reduce precision. Vector retrieval can supply conceptual similarity.

Each remedy changes which evidence the system accepts. Choosing one retriever or combining several depends on whether literal, conceptual, or both kinds of relevance matter.

When should a system use keyword, vector, or hybrid search?

Choose keyword retrieval for literal evidence, vector retrieval for conceptual similarity, and hybrid retrieval when both signals matter. Vector retrieval can find related text without shared tokens. Hybrid search combines BM25 evidence with vector similarity when either source alone is incomplete.

The methods differ by evidence and characteristic failure:

The comparison has no universal winner. Semantic search is the broader contrast when meaning matters more than shared wording. Weaviate documentation says its hybrid search runs keyword and vector retrievers in parallel, combines their scores, and returns a fused ranking.

A decision sequence starts by inspecting real query types and corpus language. Teams then label relevance judgments, evaluate retrieval metrics, and choose or tune a method against those judgments. Hybrid retrieval still needs evaluation because fusion weights trade lexical influence against vector influence. A mixed method does not remove poor analysis, weak embeddings, or incomplete relevance labels.

Method choice therefore follows required evidence. Precise identifiers favor lexical matching, paraphrases favor conceptual matching, and mixed workloads justify measuring a fused approach.

Keyword search remains the direct choice when shared terms are reliable evidence. Lexical search analyzes query text and stored text into comparable terms, retrieves candidates where analyzed query tokens overlap indexed text tokens, and ranks those candidates with a relevance function. Its strength is controllable literal evidence; its central limitation is missed relevance across different wording. Vector retrieval addresses conceptual similarity, while hybrid retrieval combines both signals. Choose among them from query evidence and measured relevance, not fashion.