Document Ingestion & Chunking

The first phase of the RAG pipeline is Ingestion, which handles loading files into memory and partitioning them into semantically cohesive chunks that models can query.

1. Document Loading

Before chunking, raw files (PDFs, Markdown, HTML, JSON) must be loaded into LangChain Document objects containing page_content (text) and metadata (source, page number, title). LangChain provides specialized Document Loaders for different file types:

TextLoader: Reads plain text files.
PyPDFLoader: Parses and reads PDF files page-by-page.
UnstructuredHTMLLoader: Loads and extracts text clean of HTML tags.
JSONLoader: Selectively extracts fields from structured JSON files using JSONPath.

2. Chunking Strategies

We cannot feed massive documents to LLMs due to context window limits. Splitting them into topic-focused chunks makes similarity searches far more accurate. Different chunking strategies are suited for different tasks:

2.1 Character Splitting

Splits text by a fixed character count (e.g., every 500 characters).

Pros: Simple to calculate.
Cons: Cuts words, sentences, or paragraphs in half, destroying context.

2.2 Recursive Character Splitting (Recommended)

Splits text using a list of separator characters hierarchically (paragraphs \n\n, then lines \n, then spaces , and finally empty strings "").

Pros: Keeps paragraphs and sentences intact wherever possible.
Cons: Still requires tuning chunk sizes and overlap margins.

2.3 Token-Based Splitting

Splits text by token counts instead of characters.

Pros: Directly matches the LLM’s token limits, preventing context window overflow.
Cons: Harder to read visually for humans.

2.4 Semantic Chunking

Analyzes the semantic meaning of sentences (using embeddings) and splits text only when there is a significant shift in meaning.

Pros: Highly accurate; groups topics dynamically.
Cons: Requires executing an embedding model for every sentence during ingestion, making it slow and computationally expensive.

3. Ingestion Python Implementation

Below is a complete implementation that loads a local text file and chunks it recursively.

3.1 Creating a Sample File

First, let’s write a small sample database policy file (knowledge.txt):

with open("knowledge.txt", "w") as f:
    f.write("""
Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.
Passwords must be at least 12 characters long and changed every 90 days.

Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.
Lunch hours run from 12:00 PM to 1:30 PM. Clean up your dishes after use.
""")

3.2 Loading and Chunking Code

Now, load this file and split it using RecursiveCharacterTextSplitter:

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

# 1. Load the document
loader = TextLoader("knowledge.txt")
documents = loader.load()

# 2. Configure the Splitter
splitter = RecursiveCharacterTextSplitter(
    chunk_size=120,      # Max characters per chunk
    chunk_overlap=20     # Overlap boundary
)

# 3. Perform Ingestion Split
chunks = splitter.split_documents(documents)

# 4. Inspect Results
print(f"Loaded {len(documents)} document.")
print(f"Created {len(chunks)} chunks.")

for idx, chunk in enumerate(chunks):
    print(f"\n--- Chunk {idx + 1} ---")
    print(chunk.page_content)
    print(f"Metadata: {chunk.metadata}")

Output:

Loaded 1 document.
Created 3 chunks.

--- Chunk 1 ---
Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.
Metadata: {'source': 'knowledge.txt'}

--- Chunk 2 ---
Passwords must be at least 12 characters long and changed every 90 days.
Metadata: {'source': 'knowledge.txt'}

--- Chunk 3 ---
Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.
Metadata: {'source': 'knowledge.txt'}

4. Practice Exercises

Practice 1: PDF Document Loading Setup

Assume you have a PDF file named "report.pdf". Write the code to load it page-by-page using LangChain’s PyPDFLoader and print the content of the first page. Instructions:

Import PyPDFLoader from langchain_community.document_loaders.
Initialize it with "report.pdf".
Call .load() to get the document list.
Access the first page document and print its page_content.

Solution

from langchain_community.document_loaders import PyPDFLoader

# Initialize the PDF loader
loader = PyPDFLoader("report.pdf")

# Load all pages
pages = loader.load()

# Print first page content
print(pages[0].page_content)
print(pages[0].metadata) # Displays page index number

Foundations

RAG Implementation

1. Document Loading

2. Chunking Strategies

2.1 Character Splitting

2.2 Recursive Character Splitting (Recommended)

2.3 Token-Based Splitting

2.4 Semantic Chunking

3. Ingestion Python Implementation

3.1 Creating a Sample File

3.2 Loading and Chunking Code

4. Practice Exercises

Practice 1: PDF Document Loading Setup

​1. Document Loading

​2. Chunking Strategies

​2.1 Character Splitting

​2.2 Recursive Character Splitting (Recommended)

​2.3 Token-Based Splitting

​2.4 Semantic Chunking

​3. Ingestion Python Implementation

​3.1 Creating a Sample File

​3.2 Loading and Chunking Code

​4. Practice Exercises

​Practice 1: PDF Document Loading Setup

1. Document Loading

2. Chunking Strategies

2.1 Character Splitting

2.2 Recursive Character Splitting (Recommended)

2.3 Token-Based Splitting

2.4 Semantic Chunking

3. Ingestion Python Implementation

3.1 Creating a Sample File

3.2 Loading and Chunking Code

4. Practice Exercises

Practice 1: PDF Document Loading Setup