> ## Documentation Index
> Fetch the complete documentation index at: https://genai.codewithsiva.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Document Ingestion & Chunking

> Load private documents and explore character, token, and semantic text chunking strategies

The first phase of the RAG pipeline is **Ingestion**, which handles loading files into memory and partitioning them into semantically cohesive **chunks** that models can query.

## 1. Document Loading

Before chunking, raw files (PDFs, Markdown, HTML, JSON) must be loaded into LangChain `Document` objects containing `page_content` (text) and `metadata` (source, page number, title).

LangChain provides specialized **Document Loaders** for different file types:

* **`TextLoader`**: Reads plain text files.
* **`PyPDFLoader`**: Parses and reads PDF files page-by-page.
* **`UnstructuredHTMLLoader`**: Loads and extracts text clean of HTML tags.
* **`JSONLoader`**: Selectively extracts fields from structured JSON files using JSONPath.

## 2. Chunking Strategies

We cannot feed massive documents to LLMs due to context window limits. Splitting them into topic-focused chunks makes similarity searches far more accurate.

Different chunking strategies are suited for different tasks:

### 2.1 Character Splitting

Splits text by a fixed character count (e.g., every 500 characters).

* **Pros**: Simple to calculate.
* **Cons**: Cuts words, sentences, or paragraphs in half, destroying context.

### 2.2 Recursive Character Splitting (Recommended)

Splits text using a list of separator characters hierarchically (paragraphs `\n\n`, then lines `\n`, then spaces ` `, and finally empty strings `""`).

* **Pros**: Keeps paragraphs and sentences intact wherever possible.
* **Cons**: Still requires tuning chunk sizes and overlap margins.

### 2.3 Token-Based Splitting

Splits text by token counts instead of characters.

* **Pros**: Directly matches the LLM's token limits, preventing context window overflow.
* **Cons**: Harder to read visually for humans.

### 2.4 Semantic Chunking

Analyzes the semantic meaning of sentences (using embeddings) and splits text only when there is a significant shift in meaning.

* **Pros**: Highly accurate; groups topics dynamically.
* **Cons**: Requires executing an embedding model for every sentence during ingestion, making it slow and computationally expensive.

## 3. Ingestion Python Implementation

Below is a complete implementation that loads a local text file and chunks it recursively.

### 3.1 Creating a Sample File

First, let's write a small sample database policy file (`knowledge.txt`):

```python theme={null}
with open("knowledge.txt", "w") as f:
    f.write("""
Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.
Passwords must be at least 12 characters long and changed every 90 days.

Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.
Lunch hours run from 12:00 PM to 1:30 PM. Clean up your dishes after use.
""")
```

### 3.2 Loading and Chunking Code

Now, load this file and split it using `RecursiveCharacterTextSplitter`:

```python theme={null}
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

# 1. Load the document
loader = TextLoader("knowledge.txt")
documents = loader.load()

# 2. Configure the Splitter
splitter = RecursiveCharacterTextSplitter(
    chunk_size=120,      # Max characters per chunk
    chunk_overlap=20     # Overlap boundary
)

# 3. Perform Ingestion Split
chunks = splitter.split_documents(documents)

# 4. Inspect Results
print(f"Loaded {len(documents)} document.")
print(f"Created {len(chunks)} chunks.")

for idx, chunk in enumerate(chunks):
    print(f"\n--- Chunk {idx + 1} ---")
    print(chunk.page_content)
    print(f"Metadata: {chunk.metadata}")
```

**Output:**

```text theme={null}
Loaded 1 document.
Created 3 chunks.

--- Chunk 1 ---
Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.
Metadata: {'source': 'knowledge.txt'}

--- Chunk 2 ---
Passwords must be at least 12 characters long and changed every 90 days.
Metadata: {'source': 'knowledge.txt'}

--- Chunk 3 ---
Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.
Metadata: {'source': 'knowledge.txt'}
```

## 4. Practice Exercises

### Practice 1: PDF Document Loading Setup

Assume you have a PDF file named `"report.pdf"`. Write the code to load it page-by-page using LangChain's `PyPDFLoader` and print the content of the first page.

**Instructions:**

1. Import `PyPDFLoader` from `langchain_community.document_loaders`.
2. Initialize it with `"report.pdf"`.
3. Call `.load()` to get the document list.
4. Access the first page document and print its `page_content`.

<Accordion title="Solution">
  ```python theme={null}
  from langchain_community.document_loaders import PyPDFLoader

  # Initialize the PDF loader
  loader = PyPDFLoader("report.pdf")

  # Load all pages
  pages = loader.load()

  # Print first page content
  print(pages[0].page_content)
  print(pages[0].metadata) # Displays page index number
  ```
</Accordion>