> ## Documentation Index > Fetch the complete documentation index at: https://genai.codewithsiva.dev/llms.txt > Use this file to discover all available pages before exploring further. # RAG Systems > Design and build Retrieval-Augmented Generation systems to query custom data ## 📋 Table of Contents * [Chapter 1: Introduction to RAG](#chapter-1-introduction-to-rag) * [Chapter 2: Document Ingestion & Chunking](#chapter-2-document-ingestion--chunking) * [Chapter 3: Vector Databases & Retrieval](#chapter-3-vector-databases--retrieval) * [Chapter 4: Generation, Advanced RAG & Evaluation](#chapter-4-generation,-advanced-rag--evaluation) ## 💻 Workshop Practice Notebook Master all the concepts from this guide with hands-on practice: * **Practice in VS Code**: Open the notebook in your local editor. Requires a local `.env` file containing your API keys. * **Practice in Google Colab**: Open the notebook directly in Colab. Setup cells are included to install packages and request API keys. [💻 VS Code](vscode://file/Users/sivaprasad/Downloads/GenAI%20With%20Python/public/notebooks/rag/rag-practice-vscode.ipynb) | [🚀 Colab](https://colab.research.google.com/github/prasad230776/genai-course/blob/master/public/notebooks/rag/rag-practice-colab.ipynb) | 📥 Download Notebook ## Chapter 1: Introduction to RAG Large Language Models (LLMs) are incredibly powerful, but they have two core limitations: they suffer from **hallucinations** (fabricating facts convincingly) and their knowledge is static (limited to their pre-training training cut-off date). **Retrieval-Augmented Generation (RAG)** solves these problems by grounding model answers in private, external data. ## 1. What is RAG & Why is it Needed? Instead of relying solely on the LLM's internal weights to generate answers, RAG queries an external datasource to retrieve relevant documents matching the user's question, and then passes those documents to the LLM as context. ### Why not just fine-tune? * **Real-time Updates**: RAG can access live data (like database records or APIs) instantly. Fine-tuning is static and slow. * **Cost & Time**: Fine-tuning requires renting GPUs and running training runs. RAG connects to databases dynamically at runtime. * **Access Control**: RAG lets you filter documents based on user roles (e.g. Employee A cannot query Employee B's salary documents). Fine-tuned weights expose all information to all users. * **Verifiability**: RAG outputs can cite source documents (e.g., *"According to Page 12 of the HR Manual..."*), whereas fine-tuned outputs cannot be traced. ## 2. The 3 Pillars of RAG Every RAG application follows a standard three-step workflow pipeline: ```text theme={null} User Query ──> [ 1. Ingestion (Index Documents) ] ──> [ 2. Retrieval (Find Matches) ] ──> [ 3. Generation (LLM Answer) ] ``` 1. **Ingestion**: Reading raw documents (PDFs, Markdown, wikis), breaking them into smaller chunks, converting those chunks into vector embeddings, and indexing them in a database. 2. **Retrieval**: When a user asks a question, the system converts the query into a vector and searches the database to find the top-K most similar document chunks. 3. **Generation**: The retrieved document chunks are formatted into a prompt along with the user's question, and sent to the LLM to generate a factual, grounded response. ## 3. Lexical Search vs. Semantic Search RAG systems rely on **Semantic Search** (dense retrieval) rather than traditional keyword-matching search: * **Traditional (Lexical) Search**: Looks for exact word matches (e.g. TF-IDF or BM25). If you search for *"automobile repair"*, it will miss documents containing *"car mechanics"* because the exact characters do not match. * **Semantic Search**: Converts text into vector embeddings representing meaning. It knows that *"automobile"* and `"car"` are conceptually close, returning relevant matches even without direct keyword intersections. ## 4. Practice Exercises ### Practice 1: Hallucination Mitigation Explain how RAG reduces the occurrence of model hallucinations compared to open-ended generation. * **Open-ended Generation**: The model relies on predicting the next token based purely on its training parameters. If it doesn't know a fact, it continues predicting the most statistically probable next words, generating incorrect facts (hallucinations). * **RAG**: The model is restricted to a prompt template instruction (e.g., *"Answer the query ONLY using the provided context."*). Since the facts are supplied directly in the prompt, the model acts as a summarization/synthesizer, drastically reducing fabricated claims. [Back to Top](#table-of-contents) ## Chapter 2: Document Ingestion & Chunking The first phase of the RAG pipeline is **Ingestion**, which handles loading files into memory and partitioning them into semantically cohesive **chunks** that models can query. ## 1. Document Loading Before chunking, raw files (PDFs, Markdown, HTML, JSON) must be loaded into LangChain `Document` objects containing `page_content` (text) and `metadata` (source, page number, title). LangChain provides specialized **Document Loaders** for different file types: * **`TextLoader`**: Reads plain text files. * **`PyPDFLoader`**: Parses and reads PDF files page-by-page. * **`UnstructuredHTMLLoader`**: Loads and extracts text clean of HTML tags. * **`JSONLoader`**: Selectively extracts fields from structured JSON files using JSONPath. ## 2. Chunking Strategies We cannot feed massive documents to LLMs due to context window limits. Splitting them into topic-focused chunks makes similarity searches far more accurate. Different chunking strategies are suited for different tasks: ### 2.1 Character Splitting Splits text by a fixed character count (e.g., every 500 characters). * **Pros**: Simple to calculate. * **Cons**: Cuts words, sentences, or paragraphs in half, destroying context. ### 2.2 Recursive Character Splitting (Recommended) Splits text using a list of separator characters hierarchically (paragraphs `\n\n`, then lines `\n`, then spaces ` `, and finally empty strings `""`). * **Pros**: Keeps paragraphs and sentences intact wherever possible. * **Cons**: Still requires tuning chunk sizes and overlap margins. ### 2.3 Token-Based Splitting Splits text by token counts instead of characters. * **Pros**: Directly matches the LLM's token limits, preventing context window overflow. * **Cons**: Harder to read visually for humans. ### 2.4 Semantic Chunking Analyzes the semantic meaning of sentences (using embeddings) and splits text only when there is a significant shift in meaning. * **Pros**: Highly accurate; groups topics dynamically. * **Cons**: Requires executing an embedding model for every sentence during ingestion, making it slow and computationally expensive. ## 3. Ingestion Python Implementation Below is a complete implementation that loads a local text file and chunks it recursively. ### 3.1 Creating a Sample File First, let's write a small sample database policy file (`knowledge.txt`): ```python theme={null} with open("knowledge.txt", "w") as f: f.write(""" Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins. Passwords must be at least 12 characters long and changed every 90 days. Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area. Lunch hours run from 12:00 PM to 1:30 PM. Clean up your dishes after use. """) ``` ### 3.2 Loading and Chunking Code Now, load this file and split it using `RecursiveCharacterTextSplitter`: ```python theme={null} from langchain_community.document_loaders import TextLoader from langchain_text_splitters import RecursiveCharacterTextSplitter # 1. Load the document loader = TextLoader("knowledge.txt") documents = loader.load() # 2. Configure the Splitter splitter = RecursiveCharacterTextSplitter( chunk_size=120, # Max characters per chunk chunk_overlap=20 # Overlap boundary ) # 3. Perform Ingestion Split chunks = splitter.split_documents(documents) # 4. Inspect Results print(f"Loaded {len(documents)} document.") print(f"Created {len(chunks)} chunks.") for idx, chunk in enumerate(chunks): print(f"\n--- Chunk {idx + 1} ---") print(chunk.page_content) print(f"Metadata: {chunk.metadata}") ``` **Output:** ```text theme={null} Loaded 1 document. Created 3 chunks. --- Chunk 1 --- Security Policy: Employees must use Multi-Factor Authentication (MFA) for all system logins. Metadata: {'source': 'knowledge.txt'} --- Chunk 2 --- Passwords must be at least 12 characters long and changed every 90 days. Metadata: {'source': 'knowledge.txt'} --- Chunk 3 --- Pantry Policy: Coffee, tea, and fresh fruit are provided in the main kitchen area. Metadata: {'source': 'knowledge.txt'} ``` ## 4. Practice Exercises ### Practice 1: PDF Document Loading Setup Assume you have a PDF file named `"report.pdf"`. Write the code to load it page-by-page using LangChain's `PyPDFLoader` and print the content of the first page. **Instructions:** 1. Import `PyPDFLoader` from `langchain_community.document_loaders`. 2. Initialize it with `"report.pdf"`. 3. Call `.load()` to get the document list. 4. Access the first page document and print its `page_content`. ```python theme={null} from langchain_community.document_loaders import PyPDFLoader # Initialize the PDF loader loader = PyPDFLoader("report.pdf") # Load all pages pages = loader.load() # Print first page content print(pages[0].page_content) print(pages[0].metadata) # Displays page index number ``` [Back to Top](#table-of-contents) ## Chapter 3: Vector Databases & Retrieval After documents are chunked, they must be converted into numerical vectors using an **Embedding Model** and stored in a **Vector Database** for similarity retrieval. By the end of this page, you will have a working semantic search engine project. ## 1. Vector Spaces & Distance Metrics An embedding model maps text chunks to coordinate vectors in a high-dimensional space. To retrieve the best matches, the vector database calculates distance metrics between the user's query vector ($q$) and the stored document vectors ($d$): * **Cosine Similarity**: Measures the cosine of the angle between two vectors. It ranges from -1 to 1 (where 1 means identical direction). Ideal for text retrieval because it is independent of document length. * **L2 Distance (Euclidean)**: Measures the straight-line distance between two points. Closer to 0 means higher similarity. * **Dot Product**: Multiplies corresponding coordinates. If vectors are normalized, dot product equals cosine similarity. ## 2. Ingestion + Vector Search Working Project Let's build a working database search project that loads a text document, chunks it, generates embeddings using Gemini, stores them in ChromaDB, and runs semantic query searches. ### 2.1 Install Dependencies Run in your terminal: ```bash theme={null} uv add chromadb langchain-community langchain-google-genai ``` Ensure your API key is in your environment: ```bash theme={null} export GOOGLE_API_KEY="your_actual_api_key_here" ``` ### 2.2 Complete Code Implementation Save and run this code: ```python theme={null} import os from dotenv import load_dotenv from langchain_community.document_loaders import TextLoader from langchain_text_splitters import RecursiveCharacterTextSplitter from langchain_google_genai import GoogleGenAIEmbeddings from langchain_community.vectorstores import Chroma # 1. Load API keys load_dotenv() # 2. Write and Load Document Data with open("office_policy.txt", "w") as f: f.write(""" Reimbursements: Submit all travel receipts by the 25th of each month. Office Rules: Do not write passwords on post-it notes. Lock your screen when leaving your desk. Health Policy: Sick leave of more than 2 consecutive days requires a medical certificate. """) loader = TextLoader("office_policy.txt") documents = loader.load() # 3. Chunk the documents splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=15) chunks = splitter.split_documents(documents) # 4. Generate Embeddings # text-embedding-004 maps chunks to 768-dimension vectors embeddings = GoogleGenAIEmbeddings(model="models/text-embedding-004") # 5. Load into ChromaDB (In-Memory Vector DB) print("Generating embeddings and indexing into ChromaDB...") vector_db = Chroma.from_documents( documents=chunks, embedding=embeddings ) # 6. Perform a Semantic Similarity Search query = "What happens if I am sick for three days?" print(f"\nQuery: '{query}'") # Retrieve the top semantic match (k=1) results = vector_db.similarity_search(query, k=1) # 7. Print Output print("\n--- Semantic Retrieval Match ---") for doc in results: print(f"Content: {doc.page_content}") print(f"Source: {doc.metadata['source']}") ``` **Output:** ```text theme={null} Generating embeddings and indexing into ChromaDB... Query: 'What happens if I am sick for three days?' --- Semantic Retrieval Match --- Content: Health Policy: Sick leave of more than 2 consecutive days requires a medical certificate. Source: office_policy.txt ``` ## 3. Practice Exercises ### Practice 1: Search Scope (Top-K) Modify the search query step in the working project to retrieve the top **2** matches (`k=2`). Run a query searching for `"travel refunds and screen security"` and print both returned chunks. **Instructions:** 1. Call `.similarity_search(query, k=2)` on the `vector_db` object. 2. Iterate through the returned list and print each chunk's content. ```python theme={null} query = "travel refunds and screen security" results = vector_db.similarity_search(query, k=2) for i, doc in enumerate(results): print(f"Match {i+1}: {doc.page_content}") ``` [Back to Top](#table-of-contents) ## Chapter 4: Generation, Advanced RAG & Evaluation The final stage of the RAG pipeline is **Generation**, where the retrieved chunks are formatted into a prompt for the LLM. In production, simple RAG systems must be optimized using **Advanced RAG** architectures and evaluated using **RAG metrics** to ensure quality. ## 1. End-to-End RAG Synthesis Pipeline We compose the retriever, augmented prompt template, and Chat Model using LangChain Expression Language (LCEL): ```python theme={null} import os from dotenv import load_dotenv from langchain_text_splitters import RecursiveCharacterTextSplitter from langchain_google_genai import GoogleGenAIEmbeddings from langchain_community.vectorstores import Chroma from langchain_core.prompts import ChatPromptTemplate from langchain_core.output_parsers import StrOutputParser from langchain.chat_models import init_chat_model load_dotenv() # 1. Ingest Data policies = "Sick leave of more than 2 days requires a medical certificate. Submit travel expense receipts by the 25th." splitter = RecursiveCharacterTextSplitter(chunk_size=80, chunk_overlap=10) docs = splitter.create_documents([policies]) embeddings = GoogleGenAIEmbeddings(model="models/text-embedding-004") vector_db = Chroma.from_documents(docs, embeddings) retriever = vector_db.as_retriever(search_kwargs={"k": 1}) # 2. Define Prompt Template prompt = ChatPromptTemplate.from_messages([ ("system", "Answer the question using ONLY the provided context:\n\n{context}"), ("human", "{question}") ]) llm = init_chat_model("gemini-2.5-flash", model_provider="google_genai") def format_docs(docs): return "\n\n".join([doc.page_content for doc in docs]) # 3. Build RAG Chain rag_chain = ( { "context": retriever | format_docs, "question": lambda x: x } | prompt | llm | StrOutputParser() ) # 4. Invoke print(rag_chain.invoke("When should I hand in travel receipts?")) ``` ## 2. Types of RAG Architectures As RAG applications scale, they transition through three architectural paradigms: ### 2.1 Naive RAG The standard pipeline: Ingest $\rightarrow$ Embed $\rightarrow$ Retrieve $\rightarrow$ Generate. * **Limitations**: Low retrieval precision (retrieving noise), poor recall (missing details), and model hallucinations if the context is too long. ### 2.2 Advanced RAG Introduces optimizations before and after retrieval to improve answer relevance: * **Hybrid Search**: Combines keyword search (BM25) with vector search (semantic) to find both exact term matches (e.g. product IDs) and conceptual synonyms. * **Reranking**: Uses a secondary Cross-Encoder model to calculate exact relevance scores for the top-N retrieved documents, sorting the most important context to the top before sending it to the LLM. * **Metadata Filtering**: Restricts searches to specific tags (e.g. `{"department": "HR"}` or `{"year": 2026}`), preventing the retriever from pulling irrelevant documents. ### 2.3 Agentic RAG Uses LLMs as agents that decide when to query databases, reformulate queries, and self-correct answers if the retrieved data is insufficient. ## 3. RAG Evaluation Metrics To measure RAG performance (using frameworks like **Ragas** or **TruLens**), systems are evaluated across two halves of the pipeline: ```text theme={null} ┌───────────────────────┐ │ User Query │ └──────────┬────────────┘ │ Context │ Answer Precision │ Relevance & Context Recall │ & Faithfulness ▼ ┌───────────────────────┐ │ Vector Context │ └───────────────────────┘ ``` ### 3.1 Retrieval Metrics * **Context Recall**: Measures if the retriever found *all* the necessary facts needed to answer the question. * **Context Precision**: Measures if the retrieved chunks are highly relevant, or if they contain too much irrelevant noise. ### 3.2 Generation Metrics * **Faithfulness (Groundedness)**: Measures if the LLM's response is based *only* on the retrieved context. A high score means no hallucinations. * **Answer Relevance**: Measures if the generated response directly answers the user's question, rather than talking about unrelated topics. ## 4. Practice Exercises ### Practice 1: Identifying RAG Failure Modes Identify which evaluation metric is failing in the following scenarios: 1. The LLM answers a query by making up facts that were not in the retrieved documents. 2. The user asks about sick leave policies, but the database retriever returns documents about office lunch hours, leading to a blank answer. 1) **Faithfulness (Groundedness)** is failing, because the model is hallucinating facts outside of the provided context. 2) **Context Recall** is failing (and consequently **Context Precision** is low), because the retriever failed to find the correct sick leave documents.