> ## Documentation Index
> Fetch the complete documentation index at: https://genai.codewithsiva.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Document Loaders

> Learn about LangChain's Document representation and how to load unstructured data from files.

The first step in any RAG (Retrieval-Augmented Generation) pipeline is loading raw data from external sources (such as text files, PDFs, spreadsheets, or webpages) into a format that LangChain can process. This format is centered around the **`Document`** object.

### 1. The `Document` Object

In LangChain, all text data is wrapped in a standard `Document` class. A `Document` has two main attributes:

1. **`page_content`** (`str`): The actual textual content.
2. **`metadata`** (`dict`): A dictionary of key-value pairs storing contextual information (e.g., the source file path, page number, creation date, author).

#### 💡 Example: Manually Preparing Documents

Before loading files directly, you can construct `Document` objects manually. This is extremely useful when loading data from databases, custom APIs, or hardcoded strings.

```python theme={null}
from langchain_core.documents import Document

# Create a list of documents manually
documents = [
    Document(
        page_content="Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.",
        metadata={"source": "security_manual.txt", "category": "IT Security"}
    ),
    Document(
        page_content="Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.",
        metadata={"source": "office_guide.txt", "category": "Facilities"}
    )
]

# Inspecting the documents
for doc in documents:
    print(f"Content: {doc.page_content}")
    print(f"Metadata: {doc.metadata}\n")
```

**Expected Output:**

```text theme={null}
Content: Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.
Metadata: {'source': 'security_manual.txt', 'category': 'IT Security'}

Content: Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.
Metadata: {'source': 'office_guide.txt', 'category': 'Facilities'}
```

### 2. Loading Documents from Files

For real-world applications, you'll load data from physical files. LangChain provides specialized **Document Loader** subclasses that handle parsing logic for different formats. Let's explore five commonly used file loaders.

#### 2.1 Plain Text Loader (`TextLoader`)

The `TextLoader` reads plain text files (`.txt`, `.md`, etc.) directly into a single `Document` object.

**Sample File (`knowledge.txt`):**

```text theme={null}
Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.
Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.
```

**Implementation Code:**

```python theme={null}
from langchain_community.document_loaders import TextLoader

# Initialize the loader
loader = TextLoader("knowledge.txt")

# Load the file
docs = loader.load()

# Inspecting output
print(f"Loaded {len(docs)} document.")
print(f"Content:\n{docs[0].page_content}")
print(f"Metadata: {docs[0].metadata}")
```

**Expected Output:**

```text theme={null}
Loaded 1 document.
Content:
Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.
Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.
Metadata: {'source': 'knowledge.txt'}
```

#### 2.2 PDF Loader (`PyPDFLoader`)

The `PyPDFLoader` parses PDF documents page-by-page. It places the content of each page into a separate `Document` object and automatically populates the `page` number in the metadata.

**Sample File (`annual_report.pdf`):**
A PDF containing two pages:

* Page 1: "Annual Financial Report 2026. Page 1 content summary."
* Page 2: "Revenue increased by 15% year-over-year. Page 2 content details."

**Implementation Code:**

```python theme={null}
from langchain_community.document_loaders import PyPDFLoader

# Initialize the loader
loader = PyPDFLoader("annual_report.pdf")

# Load page-by-page
pages = loader.load()

print(f"Loaded {len(pages)} pages.")

# Inspecting Page 1
print(f"Page 1 Content: {pages[0].page_content.strip()}")
print(f"Page 1 Metadata: {pages[0].metadata}")

# Inspecting Page 2
print(f"Page 2 Content: {pages[1].page_content.strip()}")
print(f"Page 2 Metadata: {pages[1].metadata}")
```

**Expected Output:**

```text theme={null}
Loaded 2 pages.
Page 1 Content: Annual Financial Report 2026. Page 1 content summary.
Page 1 Metadata: {'source': 'annual_report.pdf', 'page': 0}
Page 2 Content: Revenue increased by 15% year-over-year. Page 2 content details.
Page 2 Metadata: {'source': 'annual_report.pdf', 'page': 1}
```

#### 2.3 CSV Loader (`CSVLoader`)

The `CSVLoader` reads tabular data. It creates a separate `Document` object for each row, presenting the column headers and row values as line pairs inside `page_content`.

**Sample File (`users.csv`):**

```csv theme={null}
User ID,Name,Role
101,Alice,Administrator
102,Bob,Developer
```

**Implementation Code:**

```python theme={null}
from langchain_community.document_loaders import CSVLoader

# Initialize the loader, using the "User ID" column as the metadata source key
loader = CSVLoader(file_path="users.csv", source_column="User ID")

# Load rows
rows = loader.load()

print(f"Loaded {len(rows)} rows.")

# Inspecting the first row Document
print(f"Row 1 Content:\n{rows[0].page_content}")
print(f"Row 1 Metadata: {rows[0].metadata}")
```

**Expected Output:**

```text theme={null}
Loaded 2 rows.
Row 1 Content:
User ID: 101
Name: Alice
Role: Administrator
Row 1 Metadata: {'source': '101', 'row': 0}
```

#### 2.4 JSON Loader (`JSONLoader`)

The `JSONLoader` parses JSON files. It uses a `jq` query filter to select fields to load into `page_content`.

**Sample File (`messages.json`):**

```json theme={null}
[
  {"text": "Hello World", "author": "Alice"},
  {"text": "LangChain is great", "author": "Bob"}
]
```

**Implementation Code:**

```python theme={null}
from langchain_community.document_loaders import JSONLoader

# Suppose we want to load only the "text" fields
loader = JSONLoader(
    file_path="messages.json",
    jq_schema=".[] | .text",
    text_content=True
)

docs = loader.load()
print(f"Loaded {len(docs)} JSON documents.")

# Inspecting the first document
print(f"Content 1: {docs[0].page_content}")
print(f"Metadata 1: {docs[0].metadata}")
```

**Expected Output:**

```text theme={null}
Loaded 2 JSON documents.
Content 1: Hello World
Metadata 1: {'source': '/absolute/path/to/messages.json', 'seq_num': 1}
```

#### 2.5 HTML Loader (`UnstructuredHTMLLoader`)

The `UnstructuredHTMLLoader` parses HTML pages, extracting the main text content free of scripts, CSS, and HTML tags.

**Sample File (`webpage.html`):**

```html theme={null}
<!DOCTYPE html>
<html>
<head>
    <title>About Us</title>
</head>
<body>
    <h1>Welcome to our Company</h1>
    <p>We build agentic AI systems that streamline developer workflows.</p>
</body>
</html>
```

**Implementation Code:**

```python theme={null}
from langchain_community.document_loaders import UnstructuredHTMLLoader

# Initialize the loader
loader = UnstructuredHTMLLoader("webpage.html")

# Load HTML content
docs = loader.load()

print(f"Loaded {len(docs)} HTML document.")
print(f"Cleaned Text:\n{docs[0].page_content.strip()}")
print(f"Metadata: {docs[0].metadata}")
```

**Expected Output:**

```text theme={null}
Loaded 1 HTML document.
Cleaned Text:
Welcome to our Company

We build agentic AI systems that streamline developer workflows.
Metadata: {'source': 'webpage.html'}
```

### 3. Practice Exercises

#### Practice 1: Multi-File Text Loading

Write a script that searches the current directory for all `.txt` files and loads their content into a list of LangChain `Document` objects using `TextLoader`.

<Accordion title="Solution">
  ```python theme={null}
  import glob
  from langchain_community.document_loaders import TextLoader

  all_documents = []

  # Find all text files in the current folder
  for file_path in glob.glob("*.txt"):
      try:
          loader = TextLoader(file_path)
          all_documents.extend(loader.load())
      except Exception as e:
          print(f"Error loading {file_path}: {e}")

  print(f"Total documents loaded: {len(all_documents)}")
  ```
</Accordion>

### Summary

* **The `Document` Object**: The standard data container in LangChain representing text (`page_content`) and associated source context (`metadata`).
* **Document Loaders**: Specialized parsers that extract text from specific formats and wrap them into standard `Document` objects:
  * **`TextLoader`**: Reads plain text files into a single Document.
  * **`PyPDFLoader`**: Automatically parses PDFs page-by-page, recording page indexes in metadata.
  * **`CSVLoader`**: Converts structured rows into key-value pairs per Document.
  * **`JSONLoader`**: Selectively queries text nodes using `jq` schemas.
  * **`UnstructuredHTMLLoader`**: Extracts clean, main text content free of markup tags, scripts, and styles.