Home / How it works / Chunking Documents for Semantic Search: Size and Overlap

Chunking Documents for Semantic Search: Size and Overlap

Chunking for semantic search is splitting long documents into smaller passages, each embedded and retrieved on its own, so a search can return the exact part of a document that answers a question. It matters because one embedding cannot represent a long document well: the vector averages every topic in it, and a language model can only read a limited amount of retrieved text. The main choices are chunk size, overlap and strategy. Fixed-size chunking cuts at a word or token count, recursive chunking splits on paragraphs and sentences, and semantic chunking cuts where the topic changes. Weaviate’s guidance suggests starting around 100 to 200 words with about 20 percent overlap, or 512 tokens with 50 to 100 tokens of overlap, then testing on real queries. Good chunks also carry metadata, such as the source document and position, so results can be filtered and shown in context.

What is chunking in semantic search?

Chunking in semantic search is the step that divides each document into passages before embedding, so that every passage gets its own vector and can be retrieved independently. It is the most influential data preparation decision in a semantic search or RAG system.

A semantic search index stores one vector per object. If the object is a 30-page manual, its vector is a blur of every topic the manual covers, and a question about one setting will match it only weakly. Split into passages of a few paragraphs each, the same manual becomes dozens of focused vectors, one of which closely matches the question. The search then returns that passage instead of the whole manual.

Weaviate’s blog on chunking strategies puts the goal in two parts: chunks must be easy for vector search to find, and they must give the language model enough context to answer. A simple test it proposes is whether a chunk makes sense when read alone. The rest of the design follows from balancing those two needs, starting with why chunking is needed at all. How semantic search works covers the surrounding pipeline.

Why does semantic search need chunking?

Semantic search needs chunking because one embedding cannot capture every idea in a long document, embedding models have input limits, and language models read only a limited amount of retrieved text well. Chunking addresses all three.

The first reason is precision. An embedding is a fixed-length summary; the longer the text, the more topics it averages together. Weaviate’s blog compares embedding a large chunk to describing a book by averaging its chapters. Small, focused chunks produce precise vectors that match specific questions. The second reason is input length: embedding models truncate text beyond their maximum input, so anything past the limit is never represented. Dense vectors explains what the embedding captures.

The third reason is generation. Weaviate’s blog notes that language model accuracy degrades with long inputs, citing the “lost in the middle” effect where models handle the start and end of a long context better than the middle. Retrieving a few focused chunks gives the model what it needs without burying it. Not every collection needs chunking, though. The same post points out that FAQs, product descriptions and other short, complete items are already good units, and splitting them can hurt.

What are the main chunking strategies?

The main chunking strategies are fixed-size, recursive, document-based, semantic, LLM-based and late chunking, ranging from simple rules to model-driven splitting. Simple strategies are often good enough and easy to test.

• Fixed-size chunking cuts every N words or tokens, optionally with overlap; it is simple and predictable.
• Recursive chunking splits on paragraphs first, then sentences, then words, until each piece fits the size limit.
• Document-based chunking keeps short documents whole or splits only at natural boundaries such as headings.
• Semantic chunking embeds sentences and cuts where consecutive sentences become dissimilar, marking a topic change.
• LLM-based chunking asks a language model to split text into self-contained statements or sections.
• Late chunking embeds the whole document first and derives chunk vectors from it, so each chunk keeps document context.

Weaviate’s blog recommends recursive chunking as a solid default for unstructured text such as articles and reports, because it respects the document’s own organisation. Semantic and LLM-based chunking cost more compute and suit dense, high-value documents. Code calls for splitting by functions and classes, and structured documents such as manuals suit hierarchical chunking by section. Size is the next decision.

How do you decide chunk size and overlap?

You decide chunk size and overlap by starting from a published baseline, then testing a few sizes on real queries and keeping the one that retrieves the right passages most often. No single size is best for every collection.

Weaviate gives several baselines. Its academy course suggests 100 to 150 words, or 100 to 200 words with a 20 percent overlap. Its RAG starter guide suggests a 100 to 200 word sliding window with a 50-word overlap. Its blog suggests 512 tokens with 50 to 100 tokens of overlap as a production starting point, and overlap of 10 to 20 percent of the chunk size in general. These ranges agree in spirit: a few paragraphs per chunk, with a sentence or two repeated across boundaries.

Adjust from there based on the data and the queries. Weaviate’s academy notes that short queries tend to match short chunks better, while long queries suit longer chunks. Chunks must also fit the embedding model’s input limit, and the reranker’s if one is used. The chunk size calculator helps estimate counts and sizes, and the code is short.

How do you chunk documents in Python?

You chunk documents in Python with a function that splits text into windows of a fixed number of words or tokens, stepping forward by the size minus the overlap. Libraries such as LangChain and LlamaIndex offer recursive and semantic splitters as well.

import re

def chunk_words(text, size=150, overlap=30):
    words = re.sub(r"\s+", " ", text).strip().split(" ")
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, max(len(words) - overlap, 1), step)]

On a 500-word document, this produced four chunks of 150, 150, 150 and 140 words, each sharing 30 words with its neighbour. Counting words is a rough stand-in for tokens; for exact limits, count with the embedding model’s own tokenizer. For structure-aware splitting, split on blank lines first and fall back to this function for paragraphs that exceed the size. Semantic search over documents and PDFs covers extracting text from files before chunking.

Whatever the splitter, keep a record of the settings used. Changing chunk size or strategy later means re-chunking and re-embedding the whole collection, so the settings belong next to the embedding model name in your configuration. What gets stored with each chunk matters as much as the text.

What metadata should each chunk store?

Each chunk should store its source document ID, title, section heading, position in the document and any fields needed for filtering, such as date, language or access group. Metadata turns isolated passages back into navigable documents.

Weaviate’s RAG starter guide recommends storing source-level data with each chunk, such as the book title, chapter and chunk number. That lets you filter searches, order retrieved chunks as they appear in the source, and link back to the full document. Its academy course adds a useful pattern: return the matching chunk together with its neighbours when more context is needed.

chunks = client.collections.use("DocChunk")
with chunks.batch.fixed_size(batch_size=200) as batch:
    for i, text in enumerate(chunk_words(doc_text)):
        batch.add_object(properties={
            "doc_id": doc_id, "title": doc_title, "chunk_index": i, "body": text,
        })

Prepending the document title or section heading to the chunk text before embedding often helps, since a passage like “it applies after 30 days” means little without its subject. Metadata also enables per-document filters and permission checks, as in enterprise and internal document search. With chunks stored, the last step is checking that they work.

How do you evaluate chunk quality for RAG?

You evaluate chunk quality for RAG by running real questions against each chunking setup and measuring whether the passage containing the answer appears in the top results, then reading the retrieved chunks yourself. Metrics and inspection catch different problems.

Weaviate’s blog suggests the method: start from a baseline, vary size and overlap, run typical queries, and compare hit rate, precision and recall, with human review of both retrieved chunks and generated answers. Recall at k is the key metric for RAG, since the model can only use what reaches the prompt; recall at k explains it. Store each setup in a separate collection, or with a strategy property for filtering, so they can be compared side by side.

Reading results reveals what metrics hide: chunks that start mid-sentence, tables split from their headers, or headings stored as chunks of their own. Fix those first, then re-measure. Chunking also interacts with other choices, such as top-k retrieval depth, since smaller chunks usually need a larger k to cover the same content.

How does chunking fit with the rest of semantic search?

Chunking fits with the rest of semantic search as the step that decides what each vector represents, which shapes indexing, similarity, ranking and every later stage. A good index cannot fix bad chunks.

After chunking, each passage is embedded into a dense vector and often a keyword representation, covered in sparse vectors and dense vs sparse vectors. The vectors are indexed for vector search and similarity search, usually with approximate nearest neighbour search, whose trade-off with exact search is covered in exact vs approximate search and semantic indexing and vector indexes.

At query time, chunks are compared by a distance measure, explained in cosine vs dot product vs Euclidean distance, and returned with relevance scores. Matching by meaning, covered in semantic matching and synonyms and related terms, works only as well as the chunks allow, including across languages in multilingual and cross-lingual semantic search. The full architecture is in semantic search architecture.

Chunking for semantic search splits long documents into passages that are each embedded and retrieved on their own, so searches return the precise part that answers a question and language models get focused context. Choose a strategy that fits the documents, with recursive chunking as a sound default. Start from Weaviate’s baselines of about 100 to 200 words with roughly 20 percent overlap, or 512 tokens with 50 to 100 tokens of overlap. Store titles, positions and filter fields with every chunk. Then test a few sizes on real questions, measure recall at k, read the retrieved chunks, and keep the settings with the embedding model so changes are reproducible.