Chunking Strategies for Enhanced RAG Quality

Splits text into chunks of a fixed number of characters or tokens, usually with a fixed overlap. Simple, fast and fully deterministic, with no dependency on document structure — but naive fixed-size splitting with large chunks and heavy overlap is exactly what underperformed in controlled testing (see Section 3). Tuned carefully — smaller chunks, little or no overlap — it stays a reasonable baseline; used as a lazy default, it usually isn't.

Recursively splits on a prioritized list of separators — paragraph breaks first, then sentences, then words — until each chunk fits a target size, preserving more natural boundaries than raw character counting. This is what LangChain's RecursiveCharacterTextSplitter implements, and it's the de facto default for mixed or unknown document types because it degrades gracefully without needing document-specific logic.

Splits at sentence boundaries using NLP sentence segmentation, so each chunk is a self-contained thought. Well matched to FAQ pages and support articles, where each retrieval unit should map cleanly onto one question's answer. The trade-off is that very short chunks can lack surrounding context — a problem sliding-window or contextual chunking (below) exist to fix.

Splits at paragraph boundaries, preserving the natural unit of argument or explanation a human author already delineated. A good fit for reports, manuals and long-form prose where paragraphs are internally coherent — though resulting chunk sizes are less uniform and less predictable than fixed-size or recursive splitting, which complicates capacity planning for the vector index.

Chunks with deliberate overlap — a window that advances a fixed number of tokens at a time — so content near a boundary still appears in full within an adjacent chunk. Useful whenever meaning-carrying context crosses a natural split point, such as a clause continuing an argument from the previous paragraph, at the cost of a larger index and some retrieved redundancy.

Uses the document's own markup — headings, sections, list items, table boundaries — as chunk boundaries instead of raw character counts. Close to essential for PDFs, policies and technical documentation, where a chunk that silently crosses a section heading mixes two unrelated topics into one embedding. Document-parsing pipelines such as Unstructured's partitioning tools exist specifically to recover this structure from PDFs and HTML before chunking begins.

Instead of splitting on visible structure, semantic chunking embeds consecutive sentences and inserts a boundary wherever the similarity between neighboring sentence embeddings drops — wherever the topic actually shifts, regardless of paragraph or heading markers. The approach traces back to Greg Kamradt's widely cited "five levels of text splitting" work; a cluster-based variant that maximizes embedding similarity within each chunk globally, rather than only comparing adjacent sentences, achieved the highest precision in Chroma's controlled chunking evaluation (Section 3). Best for long-form articles, transcripts and meeting notes that don't map cleanly onto headings.

Prepends a short, LLM-generated explanation of what a chunk is about and where it sits in the source document, before that chunk is embedded or indexed — so a fragment reading "revenue grew 3% over the previous quarter" gets rewritten with the company name and fiscal period attached. This is Anthropic's Contextual Retrieval technique: contextual embeddings alone cut retrieval failures by 35%, adding contextual BM25 lexical search on top cut failures by 49%, and layering in reranking pushed the improvement to 67% (see Section 3). Reach for it when chunks are frequently ambiguous out of context — financial filings, legal contracts, dense technical specifications.

Indexes small, precise chunks for matching, but returns the larger "parent" chunk or section they belong to as the context actually sent to the LLM. LangChain's ParentDocumentRetriever and LlamaIndex's small-to-big retrieval pattern implement this directly: small chunks give the retriever precision — a query matches one specific sentence — while the larger parent gives the generation step the surrounding context it needs to actually answer well. It decouples what gets matched from what gets read.

Extends parent-child chunking into several nested levels — document, section, paragraph, sentence — letting retrieval operate at whichever level best matches a given query and letting the system walk up or down the hierarchy as needed. Suited to large repositories — documentation sites, knowledge bases spanning thousands of documents — where one flat chunk size can't serve both broad "which document covers this" queries and narrow "what exactly does this clause say" queries equally well.