> ## Documentation Index
> Fetch the complete documentation index at: https://genai.codewithsiva.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Generative AI Foundations

> A comprehensive consolidated guide covering the evolution of AI, Transformer architectures, training lifecycles, and model selection metrics

<a id="table-of-contents" />

## 📋 Table of Contents

* [Chapter 1: The Evolution of AI](#chapter-1-the-evolution-of-ai)
* [Chapter 2: Transformer Architectures](#chapter-2-transformer-architectures)
* [Chapter 3: How LLMs Work](#chapter-3-how-llms-work)
* [Chapter 4: Designing LLM Applications](#chapter-4-designing-llm-applications)

## <a id="chapter-1-the-evolution-of-ai" />Chapter 1: The Evolution of AI

Artificial Intelligence (AI) has evolved from rigid, manually coded expert systems to statistical patterns, deep neural networks, and finally to modern Generative AI models capable of generating human-like text and code.

## 1. What is Artificial Intelligence?

**Artificial Intelligence (AI)** is the broad field of computer science dedicated to building systems capable of performing tasks that normally require human cognitive capabilities, such as reasoning, learning, and decision-making.

```text theme={null}
Artificial Intelligence  
        ↓  
Perception → Learning → Reasoning → Decision → Action
```

The AI field represents a layered hierarchy of technologies:

* **AI**: The overall landscape (includes rule-based systems and search algorithms).
* **Machine Learning (ML)**: Statistical models that learn from data.
* **Deep Learning (DL)**: Stacks of neural networks that learn features automatically.
* **Natural Language Processing (NLP)**: The branch focused on human language.
* **Generative AI**: Systems that generate new, unstructured content (text, images, audio).

## 2. The AI Evolution Timeline

AI has progressed through several milestones, each addressing the core limitations of the previous stage:

```mermaid theme={null}
graph TD
    A["Rule-Based Systems<br/>(Hardcoded Logic)"]
    --> B["Machine Learning (ML)<br/>(Statistical Patterns)"]
    --> C["Deep Learning (DL)<br/>(Neural Networks)"]
    --> D["Neural NLP<br/>(Sequential Models)"]
    --> E["Attention & Transformers<br/>(Parallelized Context)"]
```

## 3. Rule-Based Systems

Early AI systems relied entirely on manual rules written by human domain experts.

```text theme={null}
IF condition  
THEN action
```

### 3.1 Example: Temperature Controller

```python theme={null}
if temperature > 30:  
    recommendation = "Turn on AC"  
else:  
    recommendation = "AC not required"
```

### 3.2 Characteristics

* **Deterministic**: Behavior is 100% predictable; identical inputs follow the exact same hardcoded logic path.
* **No Learning**: The system cannot adapt to new patterns or correct its own errors.
* **High Maintenance**: Scaling requires manually writing rules for every possible edge case.

### 3.3 Limitations

Real-world scenarios have too much ambiguity. For instance, parsing the sentence:

> *"I don't think the weather is going to be particularly pleasant today."*

A rule-based parser would struggle to extract the negative sentiment without a massive dictionary of rules mapping "don't think", "particularly", and "pleasant."

## 4. Machine Learning (ML)

Machine Learning shifted the paradigm: instead of humans writing the rules, **the computer learns rules from data**.

```text theme={null}
Data + Expected Outputs ──> [ ML Learning ] ──> Trained Model ──> New Data ──> Prediction
```

### 4.1 Types of Machine Learning

* **Supervised Learning**: The algorithm learns from labeled data (Input $\rightarrow$ Correct Output).
  * *Examples*: Spam detection, house-price forecasting, classification.
* **Unsupervised Learning**: The algorithm groups unlabeled data by identifying hidden patterns.
  * *Examples*: Customer segmentation, anomaly detection, clustering.
* **Reinforcement Learning**: An agent learns to make decisions by interacting with an environment to maximize a reward.
  * *Examples*: Robotics, game-playing models (AlphaGo), financial trading.

## 5. Deep Learning (DL)

Deep Learning is a subset of ML based on multi-layered **Artificial Neural Networks** inspired by biological brains.

```text theme={null}
Input ──> [ Hidden Layer 1 ] ──> [ Hidden Layer 2 ] ──> Output
```

### 5.1 Why Deep Learning Succeeded

* **Automatic Feature Extraction**: Unlike traditional ML (which requires manual feature engineering), DL networks learn representations directly from raw inputs.
* **Scalability**: Deep learning performance continues to improve as you feed it more data and compute (GPUs).

## 6. Natural Language Processing (NLP)

NLP focuses on bridging human communication and computer comprehension.

### 6.1 Why Human Language is Difficult for Computers

* **Ambiguity**: Words have different meanings depending on context (e.g., *"The bank of the river"* vs. *"A deposit in the bank"*).
* **Coreference Resolution**: Identifying what pronouns refer to. For example, in *"The student told the teacher that he was tired,"* who is *"he"*?
* **Long-Range Dependencies**: Contextual clues at the beginning of a paragraph can alter the meaning of a word at the end.

Traditional sequence-to-sequence neural networks (like Recurrent Neural Networks - RNNs, and Long Short-Term Memory - LSTMs) processed language **sequentially** (word-by-word). This sequential approach was slow to train and struggled to retain context over long text distances.

## 7. The Attention Mechanism & Transformers

To solve sequential limitations, the **Attention Mechanism** was introduced. It allows the model to look at the entire sentence at once and calculate how much focus or "attention" each word should pay to other words in the same sequence.

### 7.1 Self-Attention Illustration

In the sentence:

> *"The cat sat on the mat because it was tired."*

To understand what **"it"** refers to, the attention mechanism calculates relationships across the sequence, mapping **"it"** to **"cat"** with high attention weights.

This attention concept led to the **Transformer** architecture (introduced in the 2017 paper *"Attention Is All You Need"*), which parallelized model training and laid the foundation for modern Large Language Models (LLMs).

## 8. Practice Exercises

### Practice 1: Traditional vs. ML Sentiment

Explain how a rule-based system and a machine learning model would approach classifying whether a product review is "Positive" or "Negative."

<Accordion title="Solution">
  * **Rule-Based System**: A developer creates a list of positive words (e.g., "good", "great", "excellent") and negative words. The program counts these words in the review and classifies the sentiment based on the highest count.
  * **Machine Learning Model**: You feed the model thousands of reviews already labeled as "Positive" or "Negative." The model automatically learns which combinations of words and semantic patterns correspond to positive or negative sentiments.
</Accordion>

[Back to Top](#table-of-contents)

## <a id="chapter-2-transformer-architectures" />Chapter 2: Transformer Architectures

The **Transformer** (introduced in the 2017 research paper *"Attention Is All You Need"*) is the core neural network architecture behind all modern Large Language Models (LLMs). This page explores its primary components and architectural variants.

## 1. Core Transformer Components

The Transformer architecture reads whole sequences of tokens in parallel, processing them through several key layers:

```text theme={null}
Input Tokens ──> [ Token Embeddings ] ──> [ Positional Information ] ──> [ Self-Attention ] ──> [ Feed-Forward Networks ] ──> Output
```

### 1.1 Token Embeddings

Computers cannot read text; they only understand numbers. Embeddings map each token into a high-dimensional vector of real numbers (e.g. 768 or 1536 dimensions) representing its semantic meaning.

* **Semantic Mapping**: Words with similar meanings are mapped to vectors that are physically close in vector space (e.g., the distance between `"cat"` and `"feline"` is very small).

### 1.2 Positional Information (Positional Encodings)

Because Transformers process all tokens in parallel, they do not inherently know the order of words.

* For example: *"Dog bites man"* and *"Man bites dog"* contain the exact same tokens, but opposite meanings.
* **Positional Encodings** are vectors added directly to the token embeddings to represent the position of each word in the sequence, preserving syntactic order.

### 1.3 Self-Attention

Self-attention calculates how much context each word should gather from all other words in the same sequence.

* In the sentence: *"The server was down, so the database crashed,"* self-attention links the context of `"crashed"` back to `"database"` and `"server"`.

## 2. Encoder vs. Decoder Architectures

The original Transformer consists of two main halves: an **Encoder** and a **Decoder**. Depending on the task, models are built using one or both of these components:

| Architecture        | Focus                        | Primary Goal                                                                                 | Examples                       |
| ------------------- | ---------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------ |
| **Encoder-Only**    | Bidirectional Understanding  | Read context left-to-right and right-to-left to construct rich representations of input.     | BERT, RoBERTa                  |
| **Decoder-Only**    | Autoregressive Generation    | Predict the next token sequentially from left to right. Ideal for text generation and chat.  | GPT-4, Llama 3, Mistral        |
| **Encoder-Decoder** | Sequence-to-Sequence Mapping | The Encoder understands the input sequence, and the Decoder generates a new output sequence. | T5, BART, Original Transformer |

## 3. Deep-Dive: Architectural Roles

### 3.1 Encoder-Only Models

* **Mechanism**: Reads input bidirectionally to output a dense representation.
* **Typical Tasks**: Classification, semantic search, sentiment analysis, entity extraction.
* **Illustration**:
  ```text theme={null}
  "Python is easy to learn" ──> [ Encoder ] ──> [ Semantic Vector Representation ]
  ```

### 3.2 Decoder-Only Models

* **Mechanism**: Predicts the next token based on the prompt history. Each generated token is appended to the prompt in an autoregressive loop.
* **Typical Tasks**: Chatbots, creative writing, code generation, step-by-step reasoning.
* **Illustration**:
  ```text theme={null}
  Prompt ──> [ Decoder ] ──> Next Token ──> Append ──> Next Token ...
  ```

### 3.3 Encoder-Decoder Models

* **Mechanism**: The encoder maps input text into a vector representation, which the decoder uses to generate a new output sequence.
* **Typical Tasks**: Language translation, summarization, document refactoring.
  ```text theme={null}
  Input Text ──> [ Encoder ] ──> [ Latent Space Representation ] ──> [ Decoder ] ──> Translated Text
  ```

## 4. What is a Large Language Model (LLM)?

An **LLM** is a scaled-up, Decoder-Only or Encoder-Decoder Transformer model trained on massive text corpora.

### Why are they called "Large"?

"Large" refers to several dimensions of the model's footprint:

* **Parameters**: The millions or billions of mathematical weights inside the neural network that are adjusted during training.
* **Training Data**: Petabytes of text scraped from the internet, books, and code.
* **Compute Power**: Thousands of specialized GPUs running for weeks or months.

```text theme={null}
Model Architecture + Training Data + Compute + Training Method = Final LLM Capability
```

## 5. Practice Exercises

### Practice 1: Choosing Architectural Variants

For each task below, choose whether an **Encoder-Only** or a **Decoder-Only** model is more appropriate:

1. Detecting spam emails.
2. Answering a user's question conversationally.
3. Extracting the names of companies from a legal contract.

<Accordion title="Solution">
  1) **Detecting spam**: **Encoder-Only**, because it excels at classification and reading the entire email context bidirectionally.
  2) **Answering conversationally**: **Decoder-Only**, because it is optimized for autoregressive next-token text generation.
  3) **Extracting names**: **Encoder-Only**, because entity extraction is an understanding and labeling task over the input tokens.
</Accordion>

[Back to Top](#table-of-contents)

## <a id="chapter-3-how-llms-work" />Chapter 3: How LLMs Work

Large Language Models (LLMs) process text via tokenization, operate within strict memory boundaries (context windows), and go through multiple training stages. This page explains how models are trained and hosted.

## 1. Tokenization & Token IDs

LLMs do not understand raw characters or words. They process text in chunks called **tokens**.

```text theme={null}
Text: "Hello world" ──> [ Tokenizer ] ──> Tokens: ["Hello", " world"] ──> IDs: [15496, 995]
```

### 1.1 What is a Token?

A token can represent:

* A complete word (e.g., `"Hello"`)
* Part of a word (e.g., `"ing"`)
* A single character, punctuation mark, or whitespace.

Each token is mapped to a unique integer ID from the model's vocabulary list. These IDs are then fed into the model's embedding layer.

## 2. The Context Window

The **context window** is the maximum memory capacity (measured in tokens) that an LLM can process in a single request.

### 2.1 What the Context Window Contains:

* System Instructions
* User Prompts
* Conversation History
* Retrieval Documents (e.g. RAG context)
* Output Generated Response

```text theme={null}
┌────────────────────────────────────────────────────────┐
│                      Context Window                    │
│ [System Prompt] [History] [Docs] [User Query] [Response]│
└────────────────────────────────────────────────────────┘
```

> \[!IMPORTANT]
> A larger context window allows you to feed more documents to the model, but it does not automatically improve reasoning. In addition, processing larger context windows increases latency and token cost.

## 3. The LLM Training Lifecycle

Training an LLM requires three core ingredients: **massive datasets**, **large-scale compute clusters (GPUs/TPUs)**, and **training algorithms**.

### 3.1 Pre-Training (Self-Supervised)

* **Goal**: Learn language grammar, syntax, world facts, and reasoning patterns.
* **Method**: Predict the next token over billions of webpages, books, and code repos.
* **Output**: A **Base Model** (autocomplete engine) that has deep language understanding but does not know how to follow instructions.

### 3.2 Supervised Fine-Tuning (SFT)

* **Goal**: Teach the base model to follow commands, answer questions, and output structured templates.
* **Method**: Train on high-quality, curated pairs of instructions and ideal responses.

### 3.3 Alignment (RLHF / DPO)

* **Goal**: Teach the model safety, helpfulness, and style preference.
* **Method**: Use Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO) based on human reviews of model completions.

## 4. Training vs. Inference Requirements

* **Training** is the process of learning the model parameters (weights).
  * *Requirements*: Hundreds of GPUs (like Nvidia H100s) connected by high-bandwidth networks, petabytes of data, and millions of dollars in capital. This is why only a few large companies (Google, Meta, OpenAI) train frontier base models.
* **Inference** is the process of calling an already-trained model to generate responses.
  * *Requirements*: A single GPU or a cloud API. While much cheaper than training, large-scale inference still incurs computing costs, which is why providers charge per token.

## 5. Model Hosting & Access Options

When building applications, you can access LLMs via three hosting patterns:

### 5.1 Proprietary APIs (Closed-Source)

* **How it works**: Models hosted and managed by third-party providers (Google, OpenAI, Anthropic).
* **Pros**: Best-in-class capability, easy API integration, no server maintenance.
* **Cons**: Per-token fees, dependency on vendors, data privacy constraints.

### 5.2 Open-Weight Models

* **How it works**: Models whose weights are released publicly (Llama 3, Gemma, Mistral) for you to host on your own cloud servers.
* **Pros**: Full control over data privacy, customizable model weights.
* **Cons**: You must configure and pay for server hosting infrastructure (GPUs).

### 5.3 Local Models

* **How it works**: Running small models directly on your developer machine using tools like **Ollama**, **LM Studio**, or **vLLM**.
* **Pros**: Free, completely private, works offline.
* **Cons**: Limited by your computer's RAM and GPU capability.

## 6. Practice Exercises

### Practice 1: Pre-trained vs. Chat Models

Why is a raw "pre-trained" base model unsuitable for a conversational chatbot application?

<Accordion title="Solution">
  A pre-trained base model is designed strictly as a text autocompletion engine. If a user asks a question like *"What is the capital of France?"*, the base model might respond by autocompleting it with a list of other questions (e.g., *"What is the capital of Germany? What is the capital of Italy?"*), rather than providing the answer. It requires Supervised Fine-Tuning (SFT) to learn instruction-following chat behavior.
</Accordion>

[Back to Top](#table-of-contents)

## <a id="chapter-4-designing-llm-applications" />Chapter 4: Designing LLM Applications

Integrating LLMs into production requires understanding the cost structures, choosing models based on performance metrics, and learning how LLMs compose the intelligence layer of modern GenAI applications.

## 1. LLM API Cost Structure

Most commercial LLM APIs use **token-based pricing**, separating charges based on text direction:

```text theme={null}
Input Tokens (Prompt + Context) + Output Tokens (Model Generation) = Total Request Cost
```

* **Input Tokens**: Cheaper. Includes system prompts, conversation history, injected context documents, and user queries.
* **Output Tokens**: More expensive (often $3\times$ to $4\times$ the cost of input tokens). Includes the tokens generated by the model.

### 1.1 Factors Affecting Application Cost

1. **Request Volume**: Total number of daily active user queries.
2. **Context Growth**: In conversational chat apps, each turn appends previous messages, causing input token size to compound.
3. **Agent Loops**: Multi-agent architectures or loop-based thinking (ReAct) can make several LLM calls for a single user query.
   > One user interaction does not necessarily equal one API call.

## 2. Choosing the Right Model Class

There is no single "best" model. Models are categorized into classes balancing speed, cost, and capability:

```text theme={null}
Lightweight / Flash Models ──> High-speed + Low Cost ──> Simple tasks (Translation, classification)
Medium / Balanced Models   ──> Balanced speed & reasoning ──> General Q&A, simple extractions
Large / Frontier Models     ──> Advanced reasoning + High cost ──> Complex coding, multi-step math
```

### 2.1 Model Evaluation Dimensions

Before selecting a model, evaluate the following requirements:

* **Task Complexity**: Does it need advanced logic or simple classification?
* **Latency**: How fast must the response stream back?
* **Context Size**: How many documents are you injecting?
* **Data Privacy**: Can you send data to external APIs, or must you host open-weights locally?
* **Tool Calling**: Does the model support function calling to run databases or APIs?

### 2.2 Model Selection Strategy

1. **Start small**: Test your task with the cheapest, fastest model class (e.g. Gemini 2.5 Flash).
2. **Establish a test dataset**: Measure model outputs against a gold standard set of answers.
3. **Scale up if needed**: Only upgrade to larger, more expensive frontier models if the lightweight model fails to hit accuracy targets.

## 3. What Can We Build with LLMs?

LLMs serve as the semantic engine for various application architectures:

* **Chatbots**: Conversational assistants for customer support, HR, or training.
* **Document Q\&A (RAG)**: Indexing local PDF manuals, code repositories, or company wikis so the model answers questions factually.
* **Summarization**: Condensing research papers, call transcripts, or legal briefs.
* **Information Extraction**: Converting unstructured text (invoices, emails, resumes) into structured JSON formats.
* **Code Assistants**: Automated code generation, refactoring, code explanation, and unit test creation.
* **Autonomous Agents**: Goal-oriented loops combining memory, planning, and tools (e.g., booking flights, editing files, or running code).

## 4. The GenAI Bootcamp Learning Path

The four modules of this bootcamp are designed to build upon each other logically:

```text theme={null}
Module 1: GENAI FOUNDATIONS
AI, Machine Learning, Deep Learning, Transformers, and LLM mechanics.
        ↓
Module 2: PROMPT ENGINEERING
Prompt Templates, Chat Dialogue structures, LCEL pipes, and Output Parsers.
        ↓
Module 3: RAG SYSTEMS
Document loading, chunking strategies, vector databases, and semantic search.
        ↓
Module 4: AGENTIC AI
Autonomy, tool usage (function calling), memory architectures, and multi-agent coordination.
```

The core progression is:

> **Understand the model $\rightarrow$ Learn to communicate with it $\rightarrow$ Give it custom knowledge $\rightarrow$ Give it the ability to act.**