Document Chunking Strategies
Chunking decides how documents are split before embedding, and the boundary choices determine whether retrieval returns coherent, self-contained passages.
Definition
Chunking is the process of dividing source documents into passages small enough to embed and retrieve, with strategies that balance semantic coherence against index size and precision.
Context and scope
It is a design decision in every retrieval pipeline that ingests long or structured documents.
Common misconceptions
- a fixed character count is always the best boundary
- larger chunks give strictly better answers
Anti-patterns
- splitting on arbitrary lengths and severing sentences or tables
- ignoring document structure such as headings and code blocks
Related concepts
Authoritative source
Definition
Chunking is the process of dividing source documents into passages small enough to embed and retrieve, with strategies that balance semantic coherence against index size and precision.
It is a design decision in every retrieval pipeline that ingests long or structured documents. Compared to simpler alternatives, this concept fits best when the trade-off it represents is the one the system actually cares about.
When to use it
A common use case is applying it at the boundary where the cost of getting it wrong is highest, and the scenario usually involves a concrete guarantee the system must uphold. Unlike an ad hoc approach, it makes that guarantee explicit, reviewable, and testable.
How it works
Chunks too large dilute the embedding and waste context, while chunks too small lose the surrounding meaning needed to answer, so overlap and structure-aware splits help.
Respecting document hierarchy, such as headings and sections, produces passages that stand alone and match retrieval queries more reliably.
Trade-offs and alternatives
Metadata attached to chunks, including source, section, and position, enables filtering and lets the generator cite where an answer came from. The main drawback is the extra complexity it introduces, so the benefit should clearly outweigh that cost before a team adopts it.
Common misconceptions
A persistent misconception is that a fixed character count is always the best boundary. Another is that larger chunks give strictly better answers. For example, an engineer who assumes otherwise may design around a guarantee that does not actually exist, which only becomes obvious under load or failure.
Anti-patterns to avoid
Two recurring anti-patterns are to splitting on arbitrary lengths and severing sentences or tables and to ignoring document structure such as headings and code blocks. Both tend to surface as hard-to-debug production issues rather than obvious test failures. Related concepts include AI Guardrails and Output Validation, Retrieval-Augmented Generation, Vector Databases.