> ## Documentation Index
> Fetch the complete documentation index at: https://genai.codewithsiva.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Vector Databases & Retrieval

> Generate vector embeddings, evaluate cosine vs. L2 distance metrics, and query ChromaDB

After documents are chunked, they must be converted into numerical vectors using an **Embedding Model** and stored in a **Vector Database** for similarity retrieval. By the end of this page, you will have a working semantic search engine project.

### 1. Vector Embeddings

#### Concept

A **vector embedding** is a dense, low-dimensional mathematical representation of unstructured data (e.g., text, images) mapping semantic meaning to coordinates in a high-dimensional space. Words or sentences with similar contextual meanings reside close to each other in this space.

#### Objectives

* **Semantic Retrieval**: Resolves synonym matching (e.g., matching "automobile" with "car") and phrase intent, bypassing the limitations of exact keyword matching.
* **Dimensionality Reduction**: Converts thousands of vocabulary words into fixed-size numeric arrays (e.g., 384, 768, or 1536 elements) optimized for vector computations.

#### Embedding Models

* **Cloud API Models**: Managed services offering state-of-the-art accuracy (e.g., OpenAI `text-embedding-3-small`, Google `text-embedding-004`).
* **Open-Source (Hugging Face) Models**: Hosted locally using libraries like `sentence-transformers` for offline execution and cost efficiency.
  * *`all-MiniLM-L6-v2`*: Lightweight, fast inference (384 dimensions).
  * *`bge-large-en-v1.5`*: State-of-the-art performance for English retrieval (1024 dimensions).

#### Vector Database Options

Vector databases index and search embeddings efficiently. Choosing the right database depends on scale and search requirements:

| Database                | Primary Use Case                    | Search Capabilities                         | Pros                                                      | Cons                                                    |
| :---------------------- | :---------------------------------- | :------------------------------------------ | :-------------------------------------------------------- | :------------------------------------------------------ |
| **ChromaDB**            | Local development, prototyping      | Dense vector search                         | In-memory, Python-native, zero setup configuration        | Limited scaling, not suitable for production clustering |
| **Pinecone**            | Production SaaS applications        | Dense + Sparse (Hybrid), metadata filtering | Fully managed, highly scalable, cloud-native              | Closed-source, vendor lock-in, latency over network     |
| **Qdrant / Milvus**     | Large-scale self-hosted enterprise  | Dense, Sparse, Hybrid search                | Open-source, high-throughput, robust clustering           | Complex infrastructure setup and maintenance            |
| **pgvector (Postgres)** | Unified relational & vector storage | Relational queries + Vector search          | Integrates vectors directly into existing Postgres tables | Heavy indexing strains relational database performance  |

### 2. Vector Spaces & Distance Metrics

An embedding model maps text chunks to coordinate vectors in a high-dimensional space. To retrieve the best matches, the vector database calculates distance metrics between the user's query vector ($q$) and the stored document vectors ($d$):

* **Cosine Similarity**: Measures the cosine of the angle between two vectors. It ranges from -1 to 1.
  * **Highest Similarity Case**: A value closer to **1** (with exactly **1** representing vectors pointing in the identical direction). Ideal for text retrieval because it is independent of document length.
* **L2 Distance (Euclidean)**: Measures the straight-line distance between two points in vector space.
  * **Highest Similarity Case**: A value closer to **0** (with exactly **0** representing identical vector coordinates/matching points).
* **Dot Product (Inner Product)**: Multiplies corresponding coordinates of the query and document vectors and sums them up ($\sum q_i d_i$).
  * **Highest Similarity Case**: **Larger positive values** indicate higher similarity (and a value closer to **1** for normalized vectors).
  * **Significance**: If vectors are pre-normalized to a length of 1 (as most modern embedding models do), the dot product is mathematically identical to Cosine Similarity.
  * **Database Advantage**: It is computationally cheaper and much faster to calculate than Cosine or L2 because it avoids expensive square root calculations, making it the preferred metric for large-scale production retrieval.

### 3. Ingestion + Vector Search Working Project

Let's build a working database search project step-by-step.

#### 3.1 Install Dependencies

Run in your terminal:

```bash theme={null}
uv add chromadb langchain-community langchain-google-genai
```

Ensure your API key is in your environment:

```bash theme={null}
export GOOGLE_API_KEY="your_actual_api_key_here"
```

**Sample File (`office_policy.txt`):**

```text theme={null}
Reimbursements: Submit all travel receipts by the 25th of each month.
Office Rules: Do not write passwords on post-it notes. Lock your screen when leaving your desk.
Health Policy: Sick leave of more than 2 consecutive days requires a medical certificate.
```

#### 3.2 Step-by-Step Implementation

##### Stage 1: Load the Document Data

The document is read using `TextLoader` and wrapped into a LangChain `Document` list.

```python theme={null}
from langchain_community.document_loaders import TextLoader

loader = TextLoader("office_policy.txt")
documents = loader.load()

print(f"Loaded {len(documents)} document(s).")
```

**Sample Output:**

```text theme={null}
Loaded 1 document(s).
```

##### Stage 2: Chunk the Documents

Split the loaded document into smaller pieces using `RecursiveCharacterTextSplitter`.

```python theme={null}
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=15)
chunks = splitter.split_documents(documents)

print(f"Created {len(chunks)} text chunks.")
print(f"Chunk 1: {chunks[0].page_content}")
```

**Sample Output:**

```text theme={null}
Created 3 text chunks.
Chunk 1: Reimbursements: Submit all travel receipts by the 25th of each month.
```

##### Stage 3: Generate Embeddings

Initialize the embedding model (`text-embedding-004`) to map text chunks to numerical vectors.

```python theme={null}
from langchain_google_genai import GoogleGenAIEmbeddings

embeddings = GoogleGenAIEmbeddings(model="models/text-embedding-004")

# Test embedding generation on a single word
sample_vector = embeddings.embed_query("hello")
print(f"Embedding dimension: {len(sample_vector)}")
```

**Sample Output:**

```text theme={null}
Embedding dimension: 768
```

##### Stage 4: Index into ChromaDB

Store the generated embeddings and their original text chunks in ChromaDB.

```python theme={null}
from langchain_community.vectorstores import Chroma

print("Indexing documents into ChromaDB...")
vector_db = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings
)
```

**Sample Output:**

```text theme={null}
Indexing documents into ChromaDB...
```

##### Stage 5: Execute Similarity Search

Query the database with a natural language search to retrieve the most semantically relevant text chunk.

```python theme={null}
query = "What happens if I am sick for three days?"
results = vector_db.similarity_search(query, k=1)

print(f"Top Match: {results[0].page_content}")
print(f"Metadata: {results[0].metadata}")
```

**Sample Output:**

```text theme={null}
Top Match: Health Policy: Sick leave of more than 2 consecutive days requires a medical certificate.
Metadata: {'source': 'office_policy.txt'}
```

#### 3.3 Assembled Complete Script

Here is the complete, unified script combining all stages:

```python theme={null}
import os
from dotenv import load_dotenv
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_google_genai import GoogleGenAIEmbeddings
from langchain_community.vectorstores import Chroma

# 1. Load Environment Variables
load_dotenv()

# 2. Load the Document
loader = TextLoader("office_policy.txt")
documents = loader.load()

# 3. Chunk the Document
splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=15)
chunks = splitter.split_documents(documents)

# 4. Initialize Embedding Model
embeddings = GoogleGenAIEmbeddings(model="models/text-embedding-004")

# 5. Load chunks and embeddings into ChromaDB
vector_db = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings
)

# 6. Execute Semantic Search
query = "What happens if I am sick for three days?"
results = vector_db.similarity_search(query, k=1)

# 7. Print Results
print("\n--- Semantic Retrieval Match ---")
for doc in results:
    print(f"Content: {doc.page_content}")
    print(f"Source: {doc.metadata['source']}")
```

### 4. Practice Exercises

#### Practice 1: Search Scope (Top-K)

Modify the search query step in the working project to retrieve the top **2** matches (`k=2`). Run a query searching for `"travel refunds and screen security"` and print both returned chunks.

**Instructions:**

1. Call `.similarity_search(query, k=2)` on the `vector_db` object.
2. Iterate through the returned list and print each chunk's content.

<Accordion title="Solution">
  ```python theme={null}
  query = "travel refunds and screen security"
  results = vector_db.similarity_search(query, k=2)

  for i, doc in enumerate(results):
      print(f"Match {i+1}: {doc.page_content}")
  ```
</Accordion>

### Summary

* **Vector Embeddings**: Dense numerical representations mapping semantic meaning to multi-dimensional coordinate space. Open-source Hugging Face models (like `all-MiniLM-L6-v2`) and cloud APIs are used to compute them.
* **Vector Databases**: Specialized engines to index and query embeddings. Options include ChromaDB (prototyping), Pinecone (SaaS/hybrid search), Qdrant/Milvus (large scale self-hosted), and pgvector (relational vector integration).
* **Distance Metrics**:
  * **Cosine Similarity**: Measures vector direction (closer to `1` is highly similar).
  * **L2 Distance**: Measures physical distance (closer to `0` is highly similar).
  * **Dot Product**: Multiplies elements. Identical to Cosine for normalized vectors, offering high database computation speeds.