Document Loaders

The first step in any RAG (Retrieval-Augmented Generation) pipeline is loading raw data from external sources (such as text files, PDFs, spreadsheets, or webpages) into a format that LangChain can process. This format is centered around the Document object.

1. The `Document` Object

In LangChain, all text data is wrapped in a standard Document class. A Document has two main attributes:

page_content (str): The actual textual content.
metadata (dict): A dictionary of key-value pairs storing contextual information (e.g., the source file path, page number, creation date, author).

💡 Example: Manually Preparing Documents

Before loading files directly, you can construct Document objects manually. This is extremely useful when loading data from databases, custom APIs, or hardcoded strings.

from langchain_core.documents import Document

# Create a list of documents manually
documents = [
    Document(
        page_content="Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.",
        metadata={"source": "security_manual.txt", "category": "IT Security"}
    ),
    Document(
        page_content="Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.",
        metadata={"source": "office_guide.txt", "category": "Facilities"}
    )
]

# Inspecting the documents
for doc in documents:
    print(f"Content: {doc.page_content}")
    print(f"Metadata: {doc.metadata}\n")

Expected Output:

Content: Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.
Metadata: {'source': 'security_manual.txt', 'category': 'IT Security'}

Content: Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.
Metadata: {'source': 'office_guide.txt', 'category': 'Facilities'}

2. Loading Documents from Files

For real-world applications, you’ll load data from physical files. LangChain provides specialized Document Loader subclasses that handle parsing logic for different formats. Let’s explore five commonly used file loaders.

2.1 Plain Text Loader (`TextLoader`)

The TextLoader reads plain text files (.txt, .md, etc.) directly into a single Document object. Sample File (knowledge.txt):

Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.
Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.

Implementation Code:

from langchain_community.document_loaders import TextLoader

# Initialize the loader
loader = TextLoader("knowledge.txt")

# Load the file
docs = loader.load()

# Inspecting output
print(f"Loaded {len(docs)} document.")
print(f"Content:\n{docs[0].page_content}")
print(f"Metadata: {docs[0].metadata}")

Expected Output:

Loaded 1 document.
Content:
Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins.
Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area.
Metadata: {'source': 'knowledge.txt'}

2.2 PDF Loader (`PyPDFLoader`)

The PyPDFLoader parses PDF documents page-by-page. It places the content of each page into a separate Document object and automatically populates the page number in the metadata. Sample File (annual_report.pdf): A PDF containing two pages:

Page 1: “Annual Financial Report 2026. Page 1 content summary.”
Page 2: “Revenue increased by 15% year-over-year. Page 2 content details.”

Implementation Code:

from langchain_community.document_loaders import PyPDFLoader

# Initialize the loader
loader = PyPDFLoader("annual_report.pdf")

# Load page-by-page
pages = loader.load()

print(f"Loaded {len(pages)} pages.")

# Inspecting Page 1
print(f"Page 1 Content: {pages[0].page_content.strip()}")
print(f"Page 1 Metadata: {pages[0].metadata}")

# Inspecting Page 2
print(f"Page 2 Content: {pages[1].page_content.strip()}")
print(f"Page 2 Metadata: {pages[1].metadata}")

Expected Output:

Loaded 2 pages.
Page 1 Content: Annual Financial Report 2026. Page 1 content summary.
Page 1 Metadata: {'source': 'annual_report.pdf', 'page': 0}
Page 2 Content: Revenue increased by 15% year-over-year. Page 2 content details.
Page 2 Metadata: {'source': 'annual_report.pdf', 'page': 1}

2.3 CSV Loader (`CSVLoader`)

The CSVLoader reads tabular data. It creates a separate Document object for each row, presenting the column headers and row values as line pairs inside page_content. Sample File (users.csv):

User ID,Name,Role
101,Alice,Administrator
102,Bob,Developer

Implementation Code:

from langchain_community.document_loaders import CSVLoader

# Initialize the loader, using the "User ID" column as the metadata source key
loader = CSVLoader(file_path="users.csv", source_column="User ID")

# Load rows
rows = loader.load()

print(f"Loaded {len(rows)} rows.")

# Inspecting the first row Document
print(f"Row 1 Content:\n{rows[0].page_content}")
print(f"Row 1 Metadata: {rows[0].metadata}")

Expected Output:

Loaded 2 rows.
Row 1 Content:
User ID: 101
Name: Alice
Role: Administrator
Row 1 Metadata: {'source': '101', 'row': 0}

2.4 JSON Loader (`JSONLoader`)

The JSONLoader parses JSON files. It uses a jq query filter to select fields to load into page_content. Sample File (messages.json):

[
  {"text": "Hello World", "author": "Alice"},
  {"text": "LangChain is great", "author": "Bob"}
]

Implementation Code:

from langchain_community.document_loaders import JSONLoader

# Suppose we want to load only the "text" fields
loader = JSONLoader(
    file_path="messages.json",
    jq_schema=".[] | .text",
    text_content=True
)

docs = loader.load()
print(f"Loaded {len(docs)} JSON documents.")

# Inspecting the first document
print(f"Content 1: {docs[0].page_content}")
print(f"Metadata 1: {docs[0].metadata}")

Expected Output:

Loaded 2 JSON documents.
Content 1: Hello World
Metadata 1: {'source': '/absolute/path/to/messages.json', 'seq_num': 1}

2.5 HTML Loader (`UnstructuredHTMLLoader`)

The UnstructuredHTMLLoader parses HTML pages, extracting the main text content free of scripts, CSS, and HTML tags. Sample File (webpage.html):

<!DOCTYPE html>
<html>
<head>
    <title>About Us</title>
</head>
<body>
    <h1>Welcome to our Company</h1>
    <p>We build agentic AI systems that streamline developer workflows.</p>
</body>
</html>

Implementation Code:

from langchain_community.document_loaders import UnstructuredHTMLLoader

# Initialize the loader
loader = UnstructuredHTMLLoader("webpage.html")

# Load HTML content
docs = loader.load()

print(f"Loaded {len(docs)} HTML document.")
print(f"Cleaned Text:\n{docs[0].page_content.strip()}")
print(f"Metadata: {docs[0].metadata}")

Expected Output:

Loaded 1 HTML document.
Cleaned Text:
Welcome to our Company

We build agentic AI systems that streamline developer workflows.
Metadata: {'source': 'webpage.html'}

3. Practice Exercises

Practice 1: Multi-File Text Loading

Write a script that searches the current directory for all .txt files and loads their content into a list of LangChain Document objects using TextLoader.

Solution

import glob
from langchain_community.document_loaders import TextLoader

all_documents = []

# Find all text files in the current folder
for file_path in glob.glob("*.txt"):
    try:
        loader = TextLoader(file_path)
        all_documents.extend(loader.load())
    except Exception as e:
        print(f"Error loading {file_path}: {e}")

print(f"Total documents loaded: {len(all_documents)}")

Summary

The Document Object: The standard data container in LangChain representing text (page_content) and associated source context (metadata).
Document Loaders: Specialized parsers that extract text from specific formats and wrap them into standard Document objects:
- TextLoader: Reads plain text files into a single Document.
- PyPDFLoader: Automatically parses PDFs page-by-page, recording page indexes in metadata.
- CSVLoader: Converts structured rows into key-value pairs per Document.
- JSONLoader: Selectively queries text nodes using jq schemas.
- UnstructuredHTMLLoader: Extracts clean, main text content free of markup tags, scripts, and styles.

Foundations

RAG Implementation

1. The `Document` Object

💡 Example: Manually Preparing Documents

2. Loading Documents from Files

2.1 Plain Text Loader (`TextLoader`)

2.2 PDF Loader (`PyPDFLoader`)

2.3 CSV Loader (`CSVLoader`)

2.4 JSON Loader (`JSONLoader`)

2.5 HTML Loader (`UnstructuredHTMLLoader`)

3. Practice Exercises

Practice 1: Multi-File Text Loading

Summary

​1. The Document Object

​💡 Example: Manually Preparing Documents

​2. Loading Documents from Files

​2.1 Plain Text Loader (TextLoader)

​2.2 PDF Loader (PyPDFLoader)

​2.3 CSV Loader (CSVLoader)

​2.4 JSON Loader (JSONLoader)

​2.5 HTML Loader (UnstructuredHTMLLoader)

​3. Practice Exercises

​Practice 1: Multi-File Text Loading

​Summary

1. The `Document` Object

💡 Example: Manually Preparing Documents

2. Loading Documents from Files

2.1 Plain Text Loader (`TextLoader`)

2.2 PDF Loader (`PyPDFLoader`)

2.3 CSV Loader (`CSVLoader`)

2.4 JSON Loader (`JSONLoader`)

2.5 HTML Loader (`UnstructuredHTMLLoader`)

3. Practice Exercises

Practice 1: Multi-File Text Loading

Summary