Document Chunking & Vector Indexing Strategies for RAG Stacks

Introduction

In Retrieval-Augmented Generation (RAG) systems, search resolution depends on chunking.

If you feed raw, undivided documents directly into a vector embedding model, the semantic representations are diluted because embedding models compress entire sequences into a single fixed-dimension vector (e.g., 1536 dimensions for text-embedding-3-small). If you split text too aggressively, you sever critical context boundaries, leading to incomplete query matches.

To build production-grade retrieval pipelines, engineers must move beyond naive character-limit slicing and implement structure-aware splitting, context overlap buffers, and advanced multi-vector index topologies.

---

NAIVE VS. RECURSIVE CHUNKING STRATEGIES

1. Naive Fixed-Size Chunking

Fixed-size chunking slices text strictly by character or token limits (e.g., every 500 characters, overlap by 50). It is computationally trivial, but it introduces major semantic issues:

• Boundary Cuts: Sentences, code blocks, or markdown tables are cut in half, splitting subject/verb patterns or breaking JSON syntax.
• Context Loss: Information spanning across block boundaries is diluted, leading to lower cosine similarity scores during retrieval.

2. Recursive Character Splitting

The industry standard is Recursive Character Splitting (implemented by default in libraries like LangChain and LlamaIndex). Instead of slicing mechanically, the splitter evaluates a prioritized list of delimiter separators—typically:

1. Double line breaks (\n\n - Paragraphs)
2. Single line breaks (\n - Sentences/Lists)
3. Spaces ( - Words)
4. Empty strings ("" - Characters)

The splitter attempts to keep paragraphs intact. If a paragraph is larger than the target chunk size, it falls back to splitting by sentences, and then by words if necessary, keeping text coherent.

---

Sizing the Chunk Overlap

The overlap parameter ensures that information near boundaries is not lost.

For a target chunk size $C$ and an overlap $O$, each step moves forward by $C - O$ characters. If $O$ is too small, relationships between adjacent paragraphs are lost. If $O$ is too large, you inflate the total token count of your index, driving up storage costs in your vector database (e.g. Pinecone, Milvus, Qdrant) and increasing prefill token costs at query execution time.

A standard recommendation is to target 10% to 20% overlap relative to the target chunk size (e.g., 500-token chunks with 50-token to 100-token overlaps).

---

Advanced Retrieval Topologies: Parent-Child & Hierarchical Indexing

For complex documents (like financial filings or multi-page systems manuals), a flat list of chunks is insufficient. Two advanced architectures are commonly used:

graph TD
    Query[User Query] --> Embed[Embedding Model]
    Embed --> Retrieve[Search Child Index]
    Retrieve --> Match[Match Child Chunks - Small Context]
    Match --> Parent[Fetch Parent Chunks - Full Context]
    Parent --> LLM[Inject to LLM Prompt]

1. Parent-Child (Small-to-Large) Retrieval

In this architecture, documents are split into very small child chunks (e.g., 100 tokens) and larger parent chunks (e.g., 1000 tokens).

• The Index: Only the small child chunks are vectorized and stored in the vector database.
• The Retrieval: The search matches the highly precise child chunks. Once a match is found, the system queries the relational database or document store to fetch the corresponding larger parent chunk (which surrounds the child chunk).
• The Benefit: Embedding vectors are highly precise because they compress small, specific facts, while the LLM receives the full surrounding context needed to generate accurate answers.

2. Hierarchical Node Indexing

For long manuals, create a tree structure:

• Summary Node (Top-Level): Captures the semantic overview of a chapter.
• Section Nodes (Mid-Level): Details sub-headings.
• Leaf Nodes (Base-Level): Details individual paragraphs.

During search, the system first matches against top-level summaries, and then traverses down the specific matching branches, ignoring irrelevant sections. This reduces noise and prevents context window overflow.