Retrieval, Augmentation & Generation (RAG)

Phase 2: The RAG Flow

Runtime Execution: Triggered once document embeddings are stored in the database.
Acronym Breakdown:
- Retrieval (R): Queries the database to fetch context matching the user’s question.
- Augmentation (A): Merges the question and retrieved context into a single prompt template.
- Generation (G): Sends the compiled prompt to the LLM to synthesize the final answer.

Main Concepts Covered

1. Step 1: Retrieval
2. Step 2: Augmentation
3. Step 3: Generation
4. End-to-End Code: Combined Ingestion & RAG Flow
5. Types of RAG Architectures
6. Practice Exercises

1. Step 1: Retrieval

Under the Hood:
- Query Embedding: The user’s text query is converted into a numerical vector using the exact same embedding model configured during the Ingestion stage.
- Similarity Matching: The database calculates a numerical similarity score between the query vector and each stored chunk vector using a distance metric (like Cosine Similarity).
- Top-K Filtering: The database sorts all chunks by their similarity scores in descending order and extracts the top k highest-ranked chunks.
Top-K Parameter (k): A configuration parameter specifying the number of chunks to retrieve.
- Common Choice: k=3 to k=5 is preferred to balance context completeness and token efficiency.
- Evaluation & Selection: Determined empirically by running evaluations over different values:
  - Low k (e.g. 1-2): Reduces token costs and latency, but risks low Context Recall (missing critical facts).
  - High k (e.g. 10+): Maximizes recall, but introduces noise, increases API costs, and slows generation times.
  - Selection: The final k is selected where Context Recall stabilizes without inflating token overhead.

# Convert the existing vector database into a retriever interface specifying k
retriever = vector_db.as_retriever(search_kwargs={"k": 1})

# Query the retriever to fetch the most relevant document
retrieved_docs = retriever.invoke("When should I hand in travel receipts?")

2. Step 2: Augmentation

Goal: Combines the user’s question and retrieved text chunks into a unified prompt.
Mechanism: Injects context directly into structured system/human templates, preventing LLM hallucinations.

from langchain_core.prompts import ChatPromptTemplate

# Helper function to merge retrieved document contents
def format_docs(docs):
    return "\n\n".join([doc.page_content for doc in docs])

# System instructions strictly ground the model in the retrieved context
prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer the question using ONLY the provided context:\n\n{context}"),
    ("human", "{question}")
])

3. Step 3: Generation

Goal: Synthesizes a factual response grounded in the provided context.
Pipeline Mechanism: Chains the retriever context mapping, prompt template, LLM, and string parser using LCEL.

from langchain.chat_models import init_chat_model
from langchain_core.output_parsers import StrOutputParser

# Initialize the chat model
llm = init_chat_model("gemini-2.5-flash", model_provider="google_genai")

# LCEL Chain links Retrieval, Augmentation, and Generation
rag_chain = (
    {
        "context": retriever | format_docs,
        "question": lambda x: x
    }
    | prompt
    | llm
    | StrOutputParser()
)

# Run the chain to generate the final answer
response = rag_chain.invoke("When should I hand in travel receipts?")
print(response)

4. End-to-End Code: Combined Ingestion & RAG Flow

Here is the complete, self-contained script combining Phase 1 (Ingestion) and Phase 2 (Retrieval, Augmentation, and Generation) into a single execution flow:

import os
from dotenv import load_dotenv
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.embeddings import HuggingFaceBgeEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain.chat_models import init_chat_model
from langchain_core.output_parsers import StrOutputParser

# 1. Load API keys
load_dotenv()

# ==========================================
# PHASE 1: INGESTION (Build Vector Database)
# ==========================================

# Load and chunk raw documents
loader = TextLoader("policy.txt")
documents = loader.load()
splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=15)
docs = splitter.split_documents(documents)

# Create local vector store with BGE embeddings
model_name = "BAAI/bge-small-en-v1.5"
model_kwargs = {"device": "cpu"}
encode_kwargs = {"normalize_embeddings": True}
embeddings = HuggingFaceBgeEmbeddings(
    model_name=model_name,
    model_kwargs=model_kwargs,
    encode_kwargs=encode_kwargs
)
vector_db = Chroma.from_documents(docs, embeddings)

# ==========================================
# PHASE 2: THE RAG FLOW (Retrieve & Generate)
# ==========================================

# 1. Retrieval Setup
retriever = vector_db.as_retriever(search_kwargs={"k": 1})

# 2. Augmentation (Prompt) Setup
prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer the question using ONLY the provided context:\n\n{context}"),
    ("human", "{question}")
])

# 3. Generation (LLM & Output Parser) Setup
llm = init_chat_model("gemini-2.5-flash", model_provider="google_genai")

def format_docs(docs):
    return "\n\n".join([doc.page_content for doc in docs])

# LCEL Pipeline Definition
rag_chain = (
    {
        "context": retriever | format_docs,
        "question": lambda x: x
    }
    | prompt
    | llm
    | StrOutputParser()
)

# Execute Chain
response = rag_chain.invoke("When should I hand in travel receipts?")
print(f"Response: {response}")

5. Types of RAG Architectures

As pipelines scale, they transition through three architectural paradigms:

Naive RAG: A standard linear pipeline (Ingest ➔ Embed ➔ Retrieve ➔ Generate). High risk of retrieving noise or hallucinating.
Advanced RAG: Adds pre-retrieval and post-retrieval optimizations like Hybrid Search (vector + keyword search), Reranking (using cross-encoders to sort relevance), and Metadata Filtering (restricting database search scope).
Agentic RAG: Uses LLMs as autonomous agents that determine when to query databases, rewrite queries, and evaluate retrieval relevance.

6. Practice Exercises

Practice 1: Multi-Document RAG Chain

Expand the RAG pipeline to query over multiple policy files. Set up the retriever to return the top 2 matches (k=2), format them using the format_docs helper, and invoke the chain.

Solution

# Configure retriever to pull top 2 matches
retriever = vector_db.as_retriever(search_kwargs={"k": 2})

# Complete RAG Chain remains identical
rag_chain = (
    {
        "context": retriever | format_docs,
        "question": lambda x: x
    }
    | prompt
    | llm
    | StrOutputParser()
)

result = rag_chain.invoke("Detail sick leave and password security policies.")
print(result)

Summary

RAG Steps: RAG runtime flow consists of Retrieval (fetching documents), Augmentation (injecting documents into a template), and Generation (synthesizing LLM response).
LCEL RAG Pipeline: Chains retriever data mapping, prompt augmentation, LLM inference, and parsing using the | operator.
RAG Architectures: Scales from Naive RAG (simple linear lookup) to Advanced RAG (hybrid search, metadata filters, rerankers) and Agentic RAG (autonomous decision loops).

​1. Step 1: Retrieval

​2. Step 2: Augmentation

​3. Step 3: Generation

​4. End-to-End Code: Combined Ingestion & RAG Flow

​5. Types of RAG Architectures

​6. Practice Exercises

​Practice 1: Multi-Document RAG Chain

​Summary