Text Chunking Strategies

Once raw documents are loaded, they must be broken down into smaller, cohesive pieces called chunks. This page explains why chunking is critical for RAG systems, details how to set chunk parameters, and demonstrates different splitting mechanisms.

1. What is Chunking and Why is it Required?

Chunking is the process of partitioning large bodies of text into smaller, meaningful segments (chunks). It is required for several key reasons:

Context Window Limitations: Large Language Models (LLMs) have a maximum token capacity. We cannot feed a whole book or manual into the prompt.
Retrieval Precision: Embedding models capture the overall theme of a text. If a document is too long and covers many topics, its embedding will be diluted. Splitting documents into specific topic-focused segments allows vector search to find exact answers.
Reduced Noise and Cost: Passing smaller, relevant chunks rather than whole documents to the LLM reduces generation latency, keeps costs low, and prevents the model from hallucinating or losing context.

1.1 Chunk Size, Overlap, and Their Significance

When configuring splitters, you typically define two primary parameters:

Chunk Size: The maximum size of each chunk (measured in characters or tokens).
Chunk Overlap: The amount of text shared between consecutive chunks.

The Significance of Overlap

Without overlap, a critical sentence or definition might be sliced right down the middle across two chunks. This breaks semantic meaning, making it impossible for the embedding model to represent the split concept accurately.

Preserves Context: Overlap acts as a bridge, ensuring that boundary context is preserved on both sides of the split.
Improves Retrieval: If a user’s query matches keywords or semantic concepts at the split boundary, having overlap ensures the complete information is retrieved.

Most Preferred Configurations

While there is no one-size-fits-all, the industry standards for general text and documentation retrieval are:

Chunk Size: 500 to 1000 characters (or ~150 to 250 tokens). This is large enough to contain complete paragraphs and logical points, but small enough to remain precise during similarity search.
Chunk Overlap: 10% to 20% of the chunk size (typically 50 to 200 characters or 15 to 50 tokens). This provides an optimal safety buffer at the boundaries without introducing excessive duplicate text.

2. Chunking Mechanisms & Examples

LangChain provides several splitters under the langchain_text_splitters library. Let’s explore the most common strategies.

2.1 Character Splitting (`CharacterTextSplitter`)

This is the simplest splitting strategy. It splits text based on a single character separator (e.g. a newline or a space) and measures the chunk size by a fixed character count.

Pros: Extremely simple and fast.
Cons: Does not respect natural boundaries like paragraphs or sentences, which can cut words in half and destroy context.

Implementation Code:

from langchain_text_splitters import CharacterTextSplitter

text = "This is sentence one. This is sentence two. This is sentence three."

splitter = CharacterTextSplitter(
    separator=" ",
    chunk_size=25,
    chunk_overlap=5
)

chunks = splitter.create_documents([text])
for idx, chunk in enumerate(chunks):
    print(f"Chunk {idx+1}: '{chunk.page_content}'")

Expected Output:

Chunk 1: 'This is sentence one.'
Chunk 2: 'This is sentence two.'
Chunk 3: 'This is sentence three.'

2.2 Recursive Character Splitting (`RecursiveCharacterTextSplitter`)

This is the recommended default splitter for generic text. It attempts to split text hierarchically using a list of default separators: double newlines (\n\n), single newlines (\n), spaces ( ), and finally individual characters (""). It tries to keep paragraphs together first, then sentences, and then words.

Pros: Preserves semantic context by keeping natural paragraphs and sentences intact.
Cons: Requires tuning two parameters: chunk_size (max characters per chunk) and chunk_overlap (redundancy at the boundary).

Implementation Code:

from langchain_text_splitters import RecursiveCharacterTextSplitter

text = """
Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.
Passwords must be at least 12 characters long and changed every 90 days.

Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.
Clean up your dishes after use.
"""

splitter = RecursiveCharacterTextSplitter(
    chunk_size=120,
    chunk_overlap=20
)

chunks = splitter.create_documents([text])
for idx, chunk in enumerate(chunks):
    print(f"Chunk {idx+1}: '{chunk.page_content.strip()}'")

Expected Output:

Chunk 1: 'Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.'
Chunk 2: 'Passwords must be at least 12 characters long and changed every 90 days.'
Chunk 3: 'Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.
Clean up your dishes after use.'

2.3 Token-Based Splitting (`TokenTextSplitter`)

LLMs do not see characters; they see tokens (word fragments). Token-based splitters measure the chunk size in terms of token counts (e.g., using OpenAI’s tiktoken encoder).

Pros: Directly maps to the model’s actual context limit, avoiding unexpected truncation errors.
Cons: Can cut sentences mid-way, making it less readable for human debuggers.

Implementation Code:

from langchain_text_splitters import TokenTextSplitter

text = "Retrieval-Augmented Generation (RAG) is an architectural pattern that optimizes LLM outputs."

# Split by 5 tokens per chunk
splitter = TokenTextSplitter(chunk_size=5, chunk_overlap=1)

chunks = splitter.create_documents([text])
for idx, chunk in enumerate(chunks):
    print(f"Chunk {idx+1}: '{chunk.page_content}'")

Expected Output:

Chunk 1: 'Retrieval-Augmented Generation ('
Chunk 2: ' Generation (RAG) is'
Chunk 3: ' is an architectural pattern'
Chunk 4: ' pattern that optimizes LLM'
Chunk 5: ' LLM outputs.'

2.4 Semantic Chunking (`SemanticChunker`)

Instead of splitting by static lengths, semantic chunking calculates the embeddings of each sentence and looks for significant drops in similarity between adjacent sentences. When a drop is found, it marks a topic shift and splits the text.

Pros: Splits documents dynamically and naturally based on thematic changes rather than character counts.
Cons: Very slow and computationally expensive because it requires running an embedding model on every sentence during ingestion.

[!NOTE] Ensure you have langchain-experimental and a vector embedding package installed.

Implementation Code:

from langchain_experimental.text_splitter import SemanticChunker
from langchain_google_genai import GoogleGenAIEmbeddings

# Define the embedding model to measure sentence similarity
embeddings = GoogleGenAIEmbeddings(model="models/text-embedding-004")

splitter = SemanticChunker(embeddings)

text = """
The company's stock price surged by 15% following a positive quarterly earnings report.
Investors expressed confidence in the new CEO's strategic direction.
On the other side of the campus, researchers developed a new quantum computing chip.
This chip achieves quantum coherence times that double previous benchmarks.
"""

chunks = splitter.create_documents([text])
for idx, chunk in enumerate(chunks):
    print(f"Chunk {idx+1}:\n{chunk.page_content.strip()}\n")

Expected Output:

Chunk 1:
The company's stock price surged by 15% following a positive quarterly earnings report.
Investors expressed confidence in the new CEO's strategic direction.

Chunk 2:
On the other side of the campus, researchers developed a new quantum computing chip.
This chip achieves quantum coherence times that double previous benchmarks.

3. Practice Exercises

Practice 1: Optimizing Splitter Overlap

You are loading a technical manual where definitions are critical. If you use a RecursiveCharacterTextSplitter with chunk_size=200, explain why you might set chunk_overlap=50 instead of chunk_overlap=0. Write a small snippet showing this setup.

Solution

Setting chunk_overlap=50 ensures that definitions or sentences that cross the boundary of a chunk are not completely split in half. The overlap acts as a buffer, preserving context across adjacent chunks.

from langchain_text_splitters import RecursiveCharacterTextSplitter

# Splitting with a 50-character overlap buffer
splitter = RecursiveCharacterTextSplitter(
    chunk_size=200,
    chunk_overlap=50
)

Summary

Importance of Chunking: Resolves LLM context window limits, increases retrieval precision, and prevents generation noise or hallucinations.
Chunk Parameters: Setting a 500 to 1000 character size with a 10% to 20% overlap is standard practice. Overlap serves as a safety buffer to prevent context from being severed at chunk boundaries.
Splitting Strategies:
- CharacterTextSplitter: Splits by a single character.
- RecursiveCharacterTextSplitter: Recursively evaluates separators hierarchically to keep paragraphs and sentences intact.
- TokenTextSplitter: Splits based on LLM token counts rather than character counts.
- SemanticChunker: Analyzes sentence-to-sentence embedding similarity to split dynamically based on topic shifts.

Foundations

RAG Implementation

1. What is Chunking and Why is it Required?

1.1 Chunk Size, Overlap, and Their Significance

The Significance of Overlap

Most Preferred Configurations

2. Chunking Mechanisms & Examples

2.1 Character Splitting (`CharacterTextSplitter`)

2.2 Recursive Character Splitting (`RecursiveCharacterTextSplitter`)

2.3 Token-Based Splitting (`TokenTextSplitter`)

2.4 Semantic Chunking (`SemanticChunker`)

3. Practice Exercises

Practice 1: Optimizing Splitter Overlap

Summary

​1. What is Chunking and Why is it Required?

​1.1 Chunk Size, Overlap, and Their Significance

The Significance of Overlap

Most Preferred Configurations

​2. Chunking Mechanisms & Examples

​2.1 Character Splitting (CharacterTextSplitter)

​2.2 Recursive Character Splitting (RecursiveCharacterTextSplitter)

​2.3 Token-Based Splitting (TokenTextSplitter)

​2.4 Semantic Chunking (SemanticChunker)

​3. Practice Exercises

​Practice 1: Optimizing Splitter Overlap

​Summary

1. What is Chunking and Why is it Required?

1.1 Chunk Size, Overlap, and Their Significance

2. Chunking Mechanisms & Examples

2.1 Character Splitting (`CharacterTextSplitter`)

2.2 Recursive Character Splitting (`RecursiveCharacterTextSplitter`)

2.3 Token-Based Splitting (`TokenTextSplitter`)

2.4 Semantic Chunking (`SemanticChunker`)

3. Practice Exercises

Practice 1: Optimizing Splitter Overlap

Summary