> ## Documentation Index
> Fetch the complete documentation index at: https://genai.codewithsiva.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# RAG Evaluation

> Evaluate RAG systems using core metrics like context precision, context recall, faithfulness, and answer relevance.

After building a RAG pipeline, it is crucial to evaluate its performance. RAG evaluation measures the quality of both the retrieved documents (Retrieval stage) and the model's synthesized response (Generation stage).

**Main Concepts Covered**

* [1. RAG Evaluation Metrics](#1-rag-evaluation-metrics)
  * [1.1 Retrieval Metrics](#11-retrieval-metrics)
  * [1.2 Generation Metrics](#12-generation-metrics)
* [2. Evaluation Frameworks](#2-evaluation-frameworks)
* [3. Practice Exercises](#3-practice-exercises)

***

### 1. RAG Evaluation Metrics

To measure RAG performance, systems are evaluated across two halves of the pipeline:

```mermaid theme={null}
graph TD
    Query["User Query"] --> Retrieval["Retrieval Stage"]
    Retrieval --> Generation["Generation Stage"]
    
    subgraph Retrieval_Metrics["Retrieval Quality"]
        direction LR
        Recall["Context Recall"]
        Precision["Context Precision"]
    end
    
    subgraph Generation_Metrics["Generation Quality"]
        direction LR
        Faith["Faithfulness / Groundedness"]
        Relevance["Answer Relevance"]
    end
    
    Retrieval --> Retrieval_Metrics
    Generation --> Generation_Metrics
```

#### 1.1 Retrieval Metrics

Retrieval metrics evaluate the quality of the documents fetched from the vector database:

* **Context Recall**: Measures if the retriever found **all** the necessary facts required to answer the user query.
* **Context Precision**: Measures if the retrieved text chunks are highly relevant, minimizing noise and distractors.

#### 1.2 Generation Metrics

Generation metrics evaluate the quality of the LLM's synthesized response based on the retrieved context:

* **Faithfulness (Groundedness)**: Measures if the generated response is based **only** on the retrieved context, indicating the absence of hallucinations.
* **Answer Relevance**: Measures if the response directly addresses the user's question, rather than introducing irrelevant topics.

***

### 2. Evaluation Frameworks

In production environments, developers automate these evaluations using specialized LLM-as-a-judge frameworks:

* **Ragas**: An open-source framework specifically designed for evaluating RAG pipelines, offering built-in metrics for precision, recall, and faithfulness.
* **TruLens**: An evaluation tool that uses a "Feedback Functions" paradigm to track and score RAG triad metrics over time.

***

### 3. Practice Exercises

#### Practice 1: Identifying RAG Failure Modes

Identify which evaluation metric is failing in the following scenarios:

1. The LLM answers a query by fabricating customer support numbers that were not present in the retrieved policy documents.
2. A user asks about office hours, but the retriever returns documents about reimbursement guidelines, leading to a generic "I don't know" answer.

<Accordion title="Solution">
  1) **Faithfulness (Groundedness)** is failing because the model is hallucinating information outside of the provided context.
  2) **Context Recall** is failing because the database retriever failed to locate and return the correct document chunks regarding office hours.
</Accordion>

### Summary

* **RAG Evaluation Stages**: Divided into **Retrieval** (evaluating document relevance) and **Generation** (evaluating LLM response quality).
* **Key Metrics**:
  * *Context Recall*: Finding all necessary facts.
  * *Context Precision*: Avoiding irrelevant text chunks.
  * *Faithfulness*: Grounding answers in the context to prevent hallucinations.
  * *Answer Relevance*: Directly addressing the query.
* **Frameworks**: Tools like **Ragas** and **TruLens** automate evaluations using LLMs to score pipeline quality.