RAG

Chunking Strategies for RAG: How to Split Documents So Retrieval Actually Works

Leave a Comment / By /

Chunking is the process of splitting source documents into smaller pieces before they are embedded and stored in a vector index. In a Retrieval-Augmented Generation (RAG) system, the retriever searches these pieces and gives the most relevant ones to the language model. Chunking usually follows document ingestion and directly affects retrieval quality. If a chunk is too large, it may contain too much unrelated information. If it is too small, it may lose the context needed to understand the answer. Strong text embeddings cannot fully fix poorly chosen chunk boundaries.

This article explains the trade-offs between chunk size and overlap, then covers five common strategies: fixed-size, sliding window, recursive, semantic, and hierarchical chunking. Each strategy includes runnable Python examples.

1. Why Chunking Matters

A typical RAG pipeline follows this shape: split documents into chunks, embed each chunk into a vector, store the vectors in an index, and at query time retrieve the top-$k$ chunks scored as most similar to the query embedding. That retrieved evidence is then supplied to the language model as context before answering.

rag-chunking-pipeline

Chunking sits at the very first step, and every downstream component can be affected by poor boundaries. If a chunk is too large, its embedding can be diluted: a single vector must represent several ideas at once and may become a weaker match for a focused query, similar to trying to summarize an entire chapter in one sentence. If a chunk is too small, it can lose context: a sentence like “It reduces the false positive rate by 40 percent” is meaningless on its own if the previous sentence, which explained what technique was being discussed, was sliced away into a different chunk.

You can think about this as a signal-to-noise problem for the retriever.

  • Chunks that are too large contain the right answer, but it is surrounded by so much irrelevant text that the embedding vector for the whole chunk points in a “smeared” or averaged direction, hurting the retriever’s ability to associate the chunk with a sharply focused query.
  • Chunks that are too small are cleanly focused, but they may not carry the discourse context needed for the generator to actually understand the retrieved fact. This is a common cause of the frustrating failure mode covered in debugging RAG workflows, where the correct chunk was retrieved yet the model still answers incorrectly, because that chunk had no idea what “it” or “this technique” referred to.

Chunking decisions also ripple into cost and latency. Larger chunks mean fewer chunks per document, which reduces index storage and can reduce retrieval work, but it also means each retrieved chunk consumes more of the generator’s limited context window, leaving less room for other useful evidence. Smaller chunks mean you can pack more distinct, focused pieces of evidence into the same context budget, but your index grows larger and you may need to retrieve a larger $k$ to reconstruct enough context around any single fact. Every chunking decision, in other words, is really a decision about how you want to spend a fixed context budget.

rag-chunking-size-illustration

2. The Core Trade-off: Size, Overlap, and Recall

Before comparing strategies, it helps to make the size trade-off a little more precise. Suppose a document has $N$ tokens and you split it into chunks of size $c$ tokens with an overlap of $o$ tokens between consecutive chunks, where $0 \le o < c$. With a simple sliding window that keeps the final partial chunk, the number of chunks is:

$$
n_{\mathrm{chunks}} =
\begin{cases}
0, & N = 0 \\
1, & 0 < N \le c \\
1 + \left\lceil \dfrac{N-c}{c-o} \right\rceil, & N > c
\end{cases}
$$

Two consequences follow from this formula, and they explain many practical tuning decisions:

  1. Smaller $c$ (chunk size) increases $n_{\text{chunks}}$. More, smaller chunks means your retriever has to work harder to rank the right one among a larger candidate pool, but each candidate is more topically focused.
  2. Larger $o$ (overlap) also increases $n_{\mathrm{chunks}}$, for a fixed chunk size, because consecutive chunks now share more tokens. The benefit is that evidence near a boundary is more likely to appear with enough surrounding context in at least one chunk.

A simple way to reason about overlap is the overlap ratio, $\rho = o / c$. A ratio of $\rho = 0$ means no overlap at all (chunks are cut back to back), while a ratio like $\rho = 0.2$ means each chunk repeats 20 percent of its neighbor’s content. For prose, an overlap around $0.1$ to $0.2$ of the target chunk size is a reasonable starting point, not a universal optimum. Very large overlaps often duplicate storage and can fill the top-$k$ with near-duplicates, so they should be justified with retrieval evaluation.

It is also useful to connect this back to retrieval quality itself, which you can measure using the metrics discussed in evaluating RAG systems, such as recall@$k$ and context precision. Smaller chunks and additional overlap can make a relevant fact easier to contain cleanly, but they do not guarantee higher recall@$k$; a larger candidate pool and near-duplicate results can also make ranking harder. Measure both recall and precision on representative queries, because the trade-off depends on the corpus, embedding model, and retriever.

rag-chunk-size-overlap-tradeoff

3. Fixed and Sliding Window Chunking

Fixed window and sliding window chunking use the same idea: divide a document into chunks of a chosen size. The size can be measured in tokens, words, sentences, or paragraphs. The defining difference is overlap. Fixed windows create disjoint chunks, while sliding windows repeat some content in adjacent chunks.

AspectFixed WindowSliding Window
OverlapNoYes
Chunk size can be based onTokens, words, sentences, paragraphsTokens, words, sentences, paragraphs
Main differenceDisjoint chunksOverlapping chunks

Sentence-based example. Suppose a document has eight sentences, $S_1$ through $S_8$.

Fixed window, chunk size = 2 sentences

Chunk 1: S1 S2
Chunk 2: S3 S4
Chunk 3: S5 S6
Chunk 4: S7 S8

Sliding window, chunk size = 3 sentences, stride = 2 sentences

Chunk 1: S1 S2 S3
Chunk 2:       S3 S4 S5
Chunk 3:             S5 S6 S7
Chunk 4:                   S7 S8

In the sliding window, $S_3$, $S_5$, and $S_7$ appear in two chunks. That overlap gives facts near a boundary more surrounding context. The unit is still sentences in both examples. The same distinction applies when the unit is tokens, words, or paragraphs.

The final sliding chunk has only two sentences because this example keeps the remaining partial chunk instead of discarding it.

Fixed windows are fast and predictable, but they can separate related information at a boundary. Sliding windows reduce that risk, but overlap increases the number of chunks, index storage, and the chance of retrieving near-duplicate results. For token-based chunking, use the embedding model’s tokenizer, such as byte-pair encoding or SentencePiece, because character counts do not map reliably to token limits across languages and technical text.

Python
import tiktoken
from math import ceil

def fixed_window_chunk(
    text: str,
    chunk_size: int = 256,
    overlap_percentage: float = 0.0,
    encoding_name: str = "cl100k_base",
):
    """Split text into token chunks, optionally with overlap.

    Args:
        text: The raw document text to split.
        chunk_size: Number of tokens per chunk.
        overlap_percentage: Requested percentage of each chunk repeated in the
            next one. The value is rounded up to a whole token. Use 0 for
            fixed, non-overlapping windows; use a value above 0 for sliding,
            overlapping windows. It must leave at least one new token for
            the next chunk.
        encoding_name: Tokenizer encoding to use for measuring length.

    Returns:
        A list of text chunks reconstructed from token slices.
    """
    tokenizer = tiktoken.get_encoding(encoding_name)
    tokens = tokenizer.encode(text)

    if chunk_size <= 0:
        raise ValueError("chunk_size must be positive")
    if not 0 <= overlap_percentage < 100:
        raise ValueError("overlap_percentage must be at least 0 and less than 100")

    chunks = []
    overlap_tokens = ceil(chunk_size * overlap_percentage / 100)
    if overlap_tokens >= chunk_size:
        raise ValueError("overlap_percentage is too large for chunk_size")
    step = chunk_size - overlap_tokens
    start = 0

    while start < len(tokens):
        end = min(start + chunk_size, len(tokens))
        chunk_tokens = tokens[start:end]
        chunks.append(tokenizer.decode(chunk_tokens))

        if end == len(tokens):
            break
        start += step

    return chunks


document = "Refund requests must be filed within 30 days of purchase. " * 20
fixed_chunks = fixed_window_chunk(document, chunk_size=64, overlap_percentage=0)
sliding_chunks = fixed_window_chunk(document, chunk_size=64, overlap_percentage=30)
print(f"Fixed windows: {len(fixed_chunks)} chunks")
print(f"Sliding windows: {len(sliding_chunks)} chunks")
# Expected output:
#     Fixed windows: 5 chunks
#     Sliding windows: 6 chunks

The same function implements both approaches: set overlap_percentage=0 for fixed windows and use a value above 0 for sliding windows. This example uses tokens, but the same calculation applies to words, sentences, or paragraphs once the document has been split into those units. Neither approach detects topic changes, so both can be less suitable for structured documents, such as legal contracts, source code, or Markdown with headers and tables. Recursive and semantic chunking address those limitations differently.

4. Recursive Chunking

Recursive chunking (popularized by frameworks like LangChain’s RecursiveCharacterTextSplitter) takes a more structure-aware approach. Instead of committing to a single separator like “split every $c$ characters,” it tries a prioritized list of separators, from coarse to fine, and only falls back to a finer-grained separator when a chunk produced by a coarser one is still too large.

A typical separator priority list looks like this:

  1. Double newline (\n\n), which usually marks a paragraph boundary.
  2. Single newline (\n), which usually marks a line or list-item boundary.
  3. Sentence-ending punctuation (., ?, !), which marks a sentence boundary.
  4. Whitespace (), as a last resort word-level split.
  5. Character-level split, only if a single “word” (like a long URL or a huge concatenated string) still exceeds the target size.

The algorithm recursively tries the first separator, checks whether the resulting pieces already fit under chunk_size; if a piece is still too large, it recurses into that piece using the next separator in the list, and so on. This gives you the predictability of fixed-size chunking (an upper bound on chunk length) combined with a strong preference for cutting along natural structural seams whenever the source text allows it.

Python
from langchain_text_splitters import RecursiveCharacterTextSplitter

# 1. Initialize the text splitter
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=200,       # The maximum number of characters per chunk
    chunk_overlap=100,     # Overlapping characters between consecutive chunks
    length_function=len,  # Use standard character length to measure sizes
)

# Sample long document text
document_text = """
Text chunking is important for large language models.
It helps in processing long documents efficiently.

Different strategies exist. For example, fixed-size chunks or overlapping chunks.
Overlapping chunks help retain context across boundaries.
This method improves retrieval-augmented generation (RAG).
It ensures that relevant information is not split awkwardly.
"""

# 2. Split into raw strings
text_chunks = text_splitter.split_text(document_text)

# 3. Alternatively, split into LangChain Document objects (retains metadata)
documents = text_splitter.create_documents([document_text])

print(f"Total chunks created: {len(text_chunks)}\n")
for i, chunk in enumerate(text_chunks):
    print(f"Chunk {i}: {chunk}\n")
    print()

# Expected output:
#     Total chunks created: 3

#     Chunk 0 :
#     Text chunking is important for large language models.
#     It helps in processing long documents efficiently.

#     Chunk 1 :
#     Different strategies exist. For example, fixed-size chunks or overlapping chunks.
#     Overlapping chunks help retain context across boundaries.
#     This method improves retrieval-augmented generation (RAG).

#     Chunk 2 :
#     This method improves retrieval-augmented generation (RAG).
#     It ensures that relevant information is not split awkwardly.

Recursive chunking is a strong, low-cost default for most text corpora (documentation, wikis, transcripts) because it respects the document’s existing structure without requiring any extra model calls or embedding computation. It is usually the right starting point before you invest in something more expensive like semantic chunking.

5. Semantic Chunking

Semantic chunking asks a more ambitious question: instead of relying only on punctuation or a fixed length, can we infer likely topic boundaries from the text? The intuition is that adjacent sentences discussing the same idea should usually end up in the same chunk, while a shift to a new idea, even mid-paragraph, may justify a new chunk boundary.

The typical implementation works like this:

  1. Split the document into individual sentences.
  2. Embed each sentence (or a small sliding group of a few sentences) using a text embedding model.
  3. Compute the cosine similarity between each pair of consecutive sentence embeddings.
  4. Wherever the similarity drops below a threshold, or drops sharply relative to the local average, insert a chunk boundary.

The cosine similarity between two embedding vectors $\mathbf{e}_i$ and $\mathbf{e}_{i+1}$ is:

$$
\text{sim}(\mathbf{e}_i, \mathbf{e}_{i+1}) = \frac{\mathbf{e}_i \cdot \mathbf{e}_{i+1}}{\lVert \mathbf{e}_i \rVert \, \lVert \mathbf{e}_{i+1} \rVert}
$$

One heuristic for choosing a threshold is to inspect the distribution of “similarity drops” (that is, $1 – \text{sim}$) within a document and cut where a drop exceeds a high percentile, such as the 95th. This is not inherently robust: short documents, repeated phrasing, and the embedding model can make percentile thresholds unstable. Treat it as a starting point and validate it against labeled retrieval queries and sensible minimum and maximum chunk sizes.

rag-semantic-chunking-boundaries
Python
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

def semantic_chunk(sentences: list[str], percentile_threshold: float = 95.0) -> list[str]:
    """Group sentences into chunks based on where topical similarity drops sharply.

    Args:
        sentences: The document already split into sentences.
        percentile_threshold: Similarity-drop percentile used as the cut threshold;
            higher values produce fewer, larger chunks.

    Returns:
        A list of chunk strings, each a topically coherent group of sentences.
    """
    if not sentences:
        return []
    if len(sentences) == 1:
        return sentences.copy()
    if not 0 < percentile_threshold < 100:
        raise ValueError("percentile_threshold must be between 0 and 100")

    embeddings = np.asarray(model.encode(sentences), dtype=float)
    norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
    if np.any(norms == 0):
        raise ValueError("embedding model returned a zero-length vector")
    embeddings = embeddings / norms

    # Cosine similarity between each sentence and the one right after it.
    # Explicit normalization keeps the calculation correct for models that
    # return unnormalized embeddings.
    sims = np.sum(embeddings[:-1] * embeddings[1:], axis=1)
    drops = 1 - sims

    cut_threshold = np.percentile(drops, percentile_threshold)
    # Each index identifies the first sentence of the next chunk.
    boundaries = set(np.where(drops > cut_threshold)[0] + 1)

    chunks, buffer = [], []
    for idx, sentence in enumerate(sentences):
        if idx in boundaries and buffer:
            chunks.append(" ".join(buffer))
            buffer = []
        buffer.append(sentence)
    if buffer:
        chunks.append(" ".join(buffer))

    return chunks


sentences = [
    "The refund policy applies to all digital purchases.",
    "Requests must be filed within 30 days of purchase.",
    "Approved refunds are processed within 5 business days.",
    "Our mobile app was redesigned last quarter with a new dashboard.",
    "The dashboard now shows order history and delivery tracking.",
    "Push notifications can be configured from the settings menu.",
]
chunks = semantic_chunk(sentences, percentile_threshold=80.0)
for i, c in enumerate(chunks):
    print(f"Chunk {i}: {c}\n")

Semantic chunking can improve topical coherence when documents mix subjects without clean structural markers, such as meeting transcripts or long-form narrative reports. Its quality depends on the sentence segmentation, embedding model, threshold, and guardrails on chunk length; it can also make poor boundaries. The cost is real, though: each sentence or sentence group must be embedded (usually in batches), adding compute during ingestion, and the resulting chunk sizes become variable, which makes token budgets harder to manage. In practice, teams may run recursive chunking first to get structurally sensible sections, then apply a semantic pass only within selected long sections.

6. Hierarchical Chunking

All the strategies above produce a single flat list of chunks. Hierarchical chunking instead builds a tree: large “parent” chunks (say, whole sections or pages) each contain several smaller “child” chunks (say, paragraphs or a few sentences). At query time, you can search over the smaller child chunks for specificity, then hand the parent chunk or a bounded window around the child to the generator for surrounding context. Whether this improves results depends on the corpus, embedding model, retrieval method, and prompt budget.

This can mitigate the core tension from Section 1: small chunks may improve retrieval specificity, while a selected parent or local context window can provide the generator with more context. It does not remove all trade-offs: a full parent can be too long, redundant parents can consume the prompt budget, and parent selection still needs evaluation. It is similar in spirit to how a knowledge graph based RAG system separates “the unit you search over” from “the unit you reason over,” except here the hierarchy is built directly from document structure rather than from extracted entities and relations. See also when to use a knowledge-graph RAG if your corpus has rich, explicit entity relationships that a pure hierarchy would not capture.

Python
from dataclasses import dataclass, field
from langchain_text_splitters import RecursiveCharacterTextSplitter

@dataclass
class Chunk:
    text: str
    chunk_id: str
    parent_id: str | None = None
    children: list["Chunk"] = field(default_factory=list)


def build_hierarchical_chunks(document_sections: dict[str, str], child_chunk_size: int = 200) -> list[Chunk]:
    """Build a two-level chunk hierarchy: one parent chunk per section, split into child chunks.

    Args:
        document_sections: Mapping of section title to the section's raw text.
        child_chunk_size: Target character length for each child chunk.

    Returns:
        A flat list of parent Chunk objects, each with populated `children`.
    """
    parents = []

    # Initialize a text splitter for child chunks
    child_text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=child_chunk_size,
        chunk_overlap=0, # Assuming no overlap for child chunks unless specified
        length_function=len,
    )

    for section_idx, (title, section_text) in enumerate(document_sections.items()):
        parent_id = f"parent-{section_idx}"
        parent = Chunk(text=section_text, chunk_id=parent_id)

        # Children are retrieved against; the parent text is what gets sent to the generator.
        child_texts = child_text_splitter.split_text(section_text)
        parent.children = [
            Chunk(text=child_text, chunk_id=f"{parent_id}-child-{i}", parent_id=parent_id)
            for i, child_text in enumerate(child_texts)
        ]
        parents.append(parent)

    return parents


sections = {
    "Refund Policy": "Digital purchases are eligible for a refund within 30 days. Requests are reviewed within 2 business days.",
    "Shipping Policy": "Physical products ship within 3 to 5 business days. International orders may take up to 3 weeks.",
}
parents = build_hierarchical_chunks(sections, child_chunk_size=60)
for parent in parents:
    print(f"Parent: {parent.chunk_id} ({len(parent.children)} children)")
    for child in parent.children:
        print(f"  -> {child.chunk_id}: {child.text}")

# Expected output:
#     Parent: parent-0 (2 children)
#     -> parent-0-child-0: Digital purchases are eligible for a refund within 30 days.
#     -> parent-0-child-1: Requests are reviewed within 2 business days.
#     Parent: parent-1 (2 children)
#     -> parent-1-child-0: Physical products ship within 3 to 5 business days.
#     -> parent-1-child-1: International orders may take up to 3 weeks.

At query time, you would embed and index the children, search over them, and then, once you have the top matching child chunk IDs, look up and pass along a parent or a bounded context window to the generator (often called “small-to-big” retrieval). This pattern adds engineering complexity because it requires reliable parent-child metadata and deduplication, but it does not require two vector indices unless the design also indexes parents. Its impact depends on the corpus and should be verified with retrieval and answer-quality evaluation after a chunking issue is identified during debugging a RAG workflow.

rag-hierarchical-retrieval

7. Comparing the Strategies

The following table summarizes when each strategy tends to be the right tool.

StrategyStructure awarenessCompute costBest forMain risk
Fixed and sliding windowNone by itselfVery lowUniform prose, quick prototypesFixed windows can lose boundary context; sliding windows create duplicate chunks
RecursiveStructural (headers, paragraphs, sentences)LowMarkdown, docs, general-purpose defaultStill length-bound, ignores topic shifts
SemanticTopic-levelHigh (embedding per sentence)Transcripts, mixed-topic long-form textLatency, variable chunk sizes
HierarchicalStructural + retrieval/generation splitMedium (metadata lookup; optionally a parent index)Long documents needing both precision and contextMore moving parts to maintain

A pragmatic rule of thumb: start with recursive chunking as your default, add hierarchical chunking once you notice the generator lacks context around otherwise correctly retrieved chunks, and reach for semantic chunking only for corpora where topic boundaries genuinely do not align with punctuation or headers, since it is the most expensive strategy to run at ingestion time.

8. Practical Tips and Best Practices

  • Measure chunk length in tokens, not characters, using the exact tokenizer and input limit for the model that will embed the chunk. Separately budget retrieved text with the generator’s tokenizer and context limit; character counts are misleading across languages and technical notation.
  • Keep metadata attached to every chunk: source document, section title, page number, and chunk index. This metadata is what lets you show citations, and it is invaluable when debugging RAG workflows where the answer looks wrong but you cannot tell if retrieval or generation failed.
  • Do not chunk tables and code the same way you chunk prose. A table row split away from its header is unreadable; prefer chunking tables and code blocks as atomic units, or serializing tables into a compact textual form before chunking.
  • Tune chunk size against your actual embedding model’s documented input limit and retrieval behavior. Models vary widely: some truncate short inputs while others support much longer sequences. A chunk that fits is not automatically a good retrieval unit, so compare candidate sizes on an evaluation set rather than assuming a universal 256-to-512-token range.
  • Validate with retrieval metrics, not vibes. Track recall@$k$ and context precision, as covered in evaluating RAG systems, whenever you change chunk size or overlap, since improvements are often non-obvious from spot-checking a handful of queries.
  • Evaluate the retrieval method together with the chunking method. For example, lexical retrievers such as BM25 and TF-IDF can favor short chunks with concentrated query terms, while embedding retrieval can reward chunks that preserve the semantic context around those terms.
  • Reconsider chunking whenever your corpus changes shape. A chunking configuration tuned for clean product documentation will likely underperform on customer support transcripts or scanned PDFs converted through OCR, since noise and structure differ substantially.
  • Remember that chunking interacts with your generator’s context window. Even with a large context budget, retrieving many overlapping chunks (high $\rho$) can waste tokens on duplicated text instead of diverse evidence. Longer inputs also raise latency and memory cost; for standard full-attention Transformer prefill, attention work grows quadratically with sequence length, although exact costs vary by architecture and implementation.

Summary

Chunking is a major factor in whether a retriever can find useful evidence. Fixed-size chunking is a fast baseline but ignores document structure. Sliding windows can reduce boundary-related context loss through overlap. Recursive chunking respects available structure and is often a practical baseline. Semantic chunking can infer topic boundaries at the cost of extra embedding work, while hierarchical chunking separates the retrieval unit from the context supplied to the generator. Neither approach guarantees better retrieval or answers; evaluate the choices on the target corpus.

If you are building or maintaining a RAG system, treat chunking strategy as a tunable hyperparameter, not a one-time preprocessing script. Start simple with recursive chunking, measure retrieval quality using the framework in evaluating RAG systems, and graduate to semantic or hierarchical chunking only where the evidence shows a real gap. If you have not yet decided whether RAG is even the right architecture for your problem, it is worth first reading when to use RAG before investing time in chunking tuning.

Website |  + posts

Silpa brings 5 years of experience in working on diverse ML projects, specializing in designing end-to-end ML systems tailored for real-time applications. Her background in statistics (Bachelor of Technology) provides a strong foundation for her work in the field. Silpa is also the driving force behind the development of the content you find on this site.

Machine Learning Engineer at HP | Website |  + posts

Happy is a seasoned ML professional with over 15 years of experience. His expertise spans various domains, including Computer Vision, Natural Language Processing (NLP), and Time Series analysis. He holds a PhD in Machine Learning from IIT Kharagpur and has furthered his research with postdoctoral experience at INRIA-Sophia Antipolis, France. Happy has a proven track record of delivering impactful ML solutions to clients.

Subscribe to our newsletter!

Related Posts