Generative AI Foundations

​

📋 Table of Contents

• Chapter 1: The Evolution of AI
• Chapter 2: Transformer Architectures
• Chapter 3: How LLMs Work
• Chapter 4: Designing LLM Applications

​

Chapter 1: The Evolution of AI

​

1. What is Artificial Intelligence?

Artificial Intelligence  
        ↓  
Perception → Learning → Reasoning → Decision → Action

• AI: The overall landscape (includes rule-based systems and search algorithms).
• Machine Learning (ML): Statistical models that learn from data.
• Deep Learning (DL): Stacks of neural networks that learn features automatically.
• Natural Language Processing (NLP): The branch focused on human language.
• Generative AI: Systems that generate new, unstructured content (text, images, audio).

​

2. The AI Evolution Timeline

​

3. Rule-Based Systems

IF condition  
THEN action

​

3.1 Example: Temperature Controller

if temperature > 30:  
    recommendation = "Turn on AC"  
else:  
    recommendation = "AC not required"

​

3.2 Characteristics

• Deterministic: Behavior is 100% predictable; identical inputs follow the exact same hardcoded logic path.
• No Learning: The system cannot adapt to new patterns or correct its own errors.
• High Maintenance: Scaling requires manually writing rules for every possible edge case.

​

3.3 Limitations

“I don’t think the weather is going to be particularly pleasant today.”

​

4. Machine Learning (ML)

Data + Expected Outputs ──> [ ML Learning ] ──> Trained Model ──> New Data ──> Prediction

​

4.1 Types of Machine Learning

• Supervised Learning: The algorithm learns from labeled data (Input $\rightarrow$ Correct Output).
  • Examples: Spam detection, house-price forecasting, classification.
• Unsupervised Learning: The algorithm groups unlabeled data by identifying hidden patterns.
  • Examples: Customer segmentation, anomaly detection, clustering.
• Reinforcement Learning: An agent learns to make decisions by interacting with an environment to maximize a reward.
  • Examples: Robotics, game-playing models (AlphaGo), financial trading.

​

5. Deep Learning (DL)

Input ──> [ Hidden Layer 1 ] ──> [ Hidden Layer 2 ] ──> Output

​

5.1 Why Deep Learning Succeeded

• Automatic Feature Extraction: Unlike traditional ML (which requires manual feature engineering), DL networks learn representations directly from raw inputs.
• Scalability: Deep learning performance continues to improve as you feed it more data and compute (GPUs).

​

6. Natural Language Processing (NLP)

​

6.1 Why Human Language is Difficult for Computers

• Ambiguity: Words have different meanings depending on context (e.g., “The bank of the river” vs. “A deposit in the bank”).
• Coreference Resolution: Identifying what pronouns refer to. For example, in “The student told the teacher that he was tired,” who is “he”?
• Long-Range Dependencies: Contextual clues at the beginning of a paragraph can alter the meaning of a word at the end.

​

7. The Attention Mechanism & Transformers

​

7.1 Self-Attention Illustration

“The cat sat on the mat because it was tired.”

​

8. Practice Exercises

​

Practice 1: Traditional vs. ML Sentiment

​

Chapter 2: Transformer Architectures

​

1. Core Transformer Components

Input Tokens ──> [ Token Embeddings ] ──> [ Positional Information ] ──> [ Self-Attention ] ──> [ Feed-Forward Networks ] ──> Output

​

1.1 Token Embeddings

• Semantic Mapping: Words with similar meanings are mapped to vectors that are physically close in vector space (e.g., the distance between "cat" and "feline" is very small).

​

1.2 Positional Information (Positional Encodings)

• For example: “Dog bites man” and “Man bites dog” contain the exact same tokens, but opposite meanings.
• Positional Encodings are vectors added directly to the token embeddings to represent the position of each word in the sequence, preserving syntactic order.

​

1.3 Self-Attention

• In the sentence: “The server was down, so the database crashed,” self-attention links the context of "crashed" back to "database" and "server".

​

2. Encoder vs. Decoder Architectures

​

3. Deep-Dive: Architectural Roles

​

3.1 Encoder-Only Models

• Mechanism: Reads input bidirectionally to output a dense representation.
• Typical Tasks: Classification, semantic search, sentiment analysis, entity extraction.
• Illustration:
  "Python is easy to learn" ──> [ Encoder ] ──> [ Semantic Vector Representation ]
  

​

3.2 Decoder-Only Models

• Mechanism: Predicts the next token based on the prompt history. Each generated token is appended to the prompt in an autoregressive loop.
• Typical Tasks: Chatbots, creative writing, code generation, step-by-step reasoning.
• Illustration:
  Prompt ──> [ Decoder ] ──> Next Token ──> Append ──> Next Token ...
  

​

3.3 Encoder-Decoder Models

• Mechanism: The encoder maps input text into a vector representation, which the decoder uses to generate a new output sequence.
• Typical Tasks: Language translation, summarization, document refactoring.
  Input Text ──> [ Encoder ] ──> [ Latent Space Representation ] ──> [ Decoder ] ──> Translated Text
  

​

4. What is a Large Language Model (LLM)?

​

Why are they called “Large”?

• Parameters: The millions or billions of mathematical weights inside the neural network that are adjusted during training.
• Training Data: Petabytes of text scraped from the internet, books, and code.
• Compute Power: Thousands of specialized GPUs running for weeks or months.

Model Architecture + Training Data + Compute + Training Method = Final LLM Capability

​

5. Practice Exercises

​

Practice 1: Choosing Architectural Variants

1. Detecting spam emails.
2. Answering a user’s question conversationally.
3. Extracting the names of companies from a legal contract.

​

Chapter 3: How LLMs Work

​

1. Tokenization & Token IDs

Text: "Hello world" ──> [ Tokenizer ] ──> Tokens: ["Hello", " world"] ──> IDs: [15496, 995]

​

1.1 What is a Token?

• A complete word (e.g., "Hello")
• Part of a word (e.g., "ing")
• A single character, punctuation mark, or whitespace.

​

2. The Context Window

​

2.1 What the Context Window Contains:

• System Instructions
• User Prompts
• Conversation History
• Retrieval Documents (e.g. RAG context)
• Output Generated Response

┌────────────────────────────────────────────────────────┐
│                      Context Window                    │
│ [System Prompt] [History] [Docs] [User Query] [Response]│
└────────────────────────────────────────────────────────┘

[!IMPORTANT] A larger context window allows you to feed more documents to the model, but it does not automatically improve reasoning. In addition, processing larger context windows increases latency and token cost.

​

3. The LLM Training Lifecycle

​

3.1 Pre-Training (Self-Supervised)

• Goal: Learn language grammar, syntax, world facts, and reasoning patterns.
• Method: Predict the next token over billions of webpages, books, and code repos.
• Output: A Base Model (autocomplete engine) that has deep language understanding but does not know how to follow instructions.

​

3.2 Supervised Fine-Tuning (SFT)

• Goal: Teach the base model to follow commands, answer questions, and output structured templates.
• Method: Train on high-quality, curated pairs of instructions and ideal responses.

​

3.3 Alignment (RLHF / DPO)

• Goal: Teach the model safety, helpfulness, and style preference.
• Method: Use Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO) based on human reviews of model completions.

​

4. Training vs. Inference Requirements

• Training is the process of learning the model parameters (weights).
  • Requirements: Hundreds of GPUs (like Nvidia H100s) connected by high-bandwidth networks, petabytes of data, and millions of dollars in capital. This is why only a few large companies (Google, Meta, OpenAI) train frontier base models.
• Inference is the process of calling an already-trained model to generate responses.
  • Requirements: A single GPU or a cloud API. While much cheaper than training, large-scale inference still incurs computing costs, which is why providers charge per token.

​

5. Model Hosting & Access Options

​

5.1 Proprietary APIs (Closed-Source)

• How it works: Models hosted and managed by third-party providers (Google, OpenAI, Anthropic).
• Pros: Best-in-class capability, easy API integration, no server maintenance.
• Cons: Per-token fees, dependency on vendors, data privacy constraints.

​

5.2 Open-Weight Models

• How it works: Models whose weights are released publicly (Llama 3, Gemma, Mistral) for you to host on your own cloud servers.
• Pros: Full control over data privacy, customizable model weights.
• Cons: You must configure and pay for server hosting infrastructure (GPUs).

​

5.3 Local Models

• How it works: Running small models directly on your developer machine using tools like Ollama, LM Studio, or vLLM.
• Pros: Free, completely private, works offline.
• Cons: Limited by your computer’s RAM and GPU capability.

​

6. Practice Exercises

​

Practice 1: Pre-trained vs. Chat Models

​

Chapter 4: Designing LLM Applications

​

1. LLM API Cost Structure

Input Tokens (Prompt + Context) + Output Tokens (Model Generation) = Total Request Cost

• Input Tokens: Cheaper. Includes system prompts, conversation history, injected context documents, and user queries.
• Output Tokens: More expensive (often $3\times$ to $4\times$ the cost of input tokens). Includes the tokens generated by the model.

​

1.1 Factors Affecting Application Cost

1. Request Volume: Total number of daily active user queries.
2. Context Growth: In conversational chat apps, each turn appends previous messages, causing input token size to compound.
3. Agent Loops: Multi-agent architectures or loop-based thinking (ReAct) can make several LLM calls for a single user query.

   One user interaction does not necessarily equal one API call.

​

2. Choosing the Right Model Class

Lightweight / Flash Models ──> High-speed + Low Cost ──> Simple tasks (Translation, classification)
Medium / Balanced Models   ──> Balanced speed & reasoning ──> General Q&A, simple extractions
Large / Frontier Models     ──> Advanced reasoning + High cost ──> Complex coding, multi-step math

​

2.1 Model Evaluation Dimensions

• Task Complexity: Does it need advanced logic or simple classification?
• Latency: How fast must the response stream back?
• Context Size: How many documents are you injecting?
• Data Privacy: Can you send data to external APIs, or must you host open-weights locally?
• Tool Calling: Does the model support function calling to run databases or APIs?

​

2.2 Model Selection Strategy

1. Start small: Test your task with the cheapest, fastest model class (e.g. Gemini 2.5 Flash).
2. Establish a test dataset: Measure model outputs against a gold standard set of answers.
3. Scale up if needed: Only upgrade to larger, more expensive frontier models if the lightweight model fails to hit accuracy targets.

​

3. What Can We Build with LLMs?

• Chatbots: Conversational assistants for customer support, HR, or training.
• Document Q&A (RAG): Indexing local PDF manuals, code repositories, or company wikis so the model answers questions factually.
• Summarization: Condensing research papers, call transcripts, or legal briefs.
• Information Extraction: Converting unstructured text (invoices, emails, resumes) into structured JSON formats.
• Code Assistants: Automated code generation, refactoring, code explanation, and unit test creation.
• Autonomous Agents: Goal-oriented loops combining memory, planning, and tools (e.g., booking flights, editing files, or running code).

​

4. The GenAI Bootcamp Learning Path

Module 1: GENAI FOUNDATIONS
AI, Machine Learning, Deep Learning, Transformers, and LLM mechanics.
        ↓
Module 2: PROMPT ENGINEERING
Prompt Templates, Chat Dialogue structures, LCEL pipes, and Output Parsers.
        ↓
Module 3: RAG SYSTEMS
Document loading, chunking strategies, vector databases, and semantic search.
        ↓
Module 4: AGENTIC AI
Autonomy, tool usage (function calling), memory architectures, and multi-agent coordination.

Understand the model $\rightarrow$ Learn to communicate with it $\rightarrow$ Give it custom knowledge $\rightarrow$ Give it the ability to act.