From AI to Generative AI - GenAI Engineering

Artificial Intelligence (AI) has evolved from rigid, manually coded expert systems to statistical patterns, deep neural networks, and finally to modern Generative AI models capable of generating human-like text and code.

1. What is Artificial Intelligence?

Artificial Intelligence (AI) is the broad field of computer science dedicated to building systems capable of performing tasks that normally require human cognitive capabilities, such as reasoning, learning, and decision-making.

Artificial Intelligence  
        ↓  
Perception → Learning → Reasoning → Decision → Action

The AI field represents a layered hierarchy of technologies:

AI: The overall landscape (includes rule-based systems and search algorithms).
Machine Learning (ML): Statistical models that learn from data.
Deep Learning (DL): Stacks of neural networks that learn features automatically.
Natural Language Processing (NLP): The branch focused on human language.
Generative AI: Systems that generate new, unstructured content (text, images, audio).

2. The AI Evolution Timeline

AI has progressed through several milestones, each addressing the core limitations of the previous stage:

3. Rule-Based Systems

Early AI systems relied entirely on manual rules written by human domain experts.

IF condition  
THEN action

3.1 Example: Temperature Controller

if temperature > 30:  
    recommendation = "Turn on AC"  
else:  
    recommendation = "AC not required"

3.2 Characteristics

Deterministic: Behavior is 100% predictable; identical inputs follow the exact same hardcoded logic path.
No Learning: The system cannot adapt to new patterns or correct its own errors.
High Maintenance: Scaling requires manually writing rules for every possible edge case.

3.3 Limitations

Real-world scenarios have too much ambiguity. For instance, parsing the sentence:

“I don’t think the weather is going to be particularly pleasant today.”

A rule-based parser would struggle to extract the negative sentiment without a massive dictionary of rules mapping “don’t think”, “particularly”, and “pleasant.”

4. Machine Learning (ML)

Machine Learning shifted the paradigm: instead of humans writing the rules, the computer learns rules from data.

Data + Expected Outputs ──> [ ML Learning ] ──> Trained Model ──> New Data ──> Prediction

4.1 Types of Machine Learning

Supervised Learning: The algorithm learns from labeled data (Input $\rightarrow$ $\to$ Correct Output).
- Examples: Spam detection, house-price forecasting, classification.
Unsupervised Learning: The algorithm groups unlabeled data by identifying hidden patterns.
- Examples: Customer segmentation, anomaly detection, clustering.
Reinforcement Learning: An agent learns to make decisions by interacting with an environment to maximize a reward.
- Examples: Robotics, game-playing models (AlphaGo), financial trading.

5. Deep Learning (DL)

Deep Learning is a subset of ML based on multi-layered Artificial Neural Networks inspired by biological brains.

Input ──> [ Hidden Layer 1 ] ──> [ Hidden Layer 2 ] ──> Output

5.1 Why Deep Learning Succeeded

Automatic Feature Extraction: Unlike traditional ML (which requires manual feature engineering), DL networks learn representations directly from raw inputs.
Scalability: Deep learning performance continues to improve as you feed it more data and compute (GPUs).

6. Natural Language Processing (NLP)

NLP focuses on bridging human communication and computer comprehension.

6.1 Why Human Language is Difficult for Computers

Ambiguity: Words have different meanings depending on context (e.g., “The bank of the river” vs. “A deposit in the bank”).
Coreference Resolution: Identifying what pronouns refer to. For example, in “The student told the teacher that he was tired,” who is “he”?
Long-Range Dependencies: Contextual clues at the beginning of a paragraph can alter the meaning of a word at the end.

Traditional sequence-to-sequence neural networks (like Recurrent Neural Networks - RNNs, and Long Short-Term Memory - LSTMs) processed language sequentially (word-by-word). This sequential approach was slow to train and struggled to retain context over long text distances.

7. The Attention Mechanism & Transformers

To solve sequential limitations, the Attention Mechanism was introduced. It allows the model to look at the entire sentence at once and calculate how much focus or “attention” each word should pay to other words in the same sequence.

7.1 Self-Attention Illustration

In the sentence:

“The cat sat on the mat because it was tired.”

To understand what “it” refers to, the attention mechanism calculates relationships across the sequence, mapping “it” to “cat” with high attention weights. This attention concept led to the Transformer architecture (introduced in the 2017 paper “Attention Is All You Need”), which parallelized model training and laid the foundation for modern Large Language Models (LLMs).

8. Practice Exercises

Practice 1: Traditional vs. ML Sentiment

Explain how a rule-based system and a machine learning model would approach classifying whether a product review is “Positive” or “Negative.”

Solution

Rule-Based System: A developer creates a list of positive words (e.g., “good”, “great”, “excellent”) and negative words. The program counts these words in the review and classifies the sentiment based on the highest count.
Machine Learning Model: You feed the model thousands of reviews already labeled as “Positive” or “Negative.” The model automatically learns which combinations of words and semantic patterns correspond to positive or negative sentiments.

​1. What is Artificial Intelligence?

​2. The AI Evolution Timeline

​3. Rule-Based Systems

​3.1 Example: Temperature Controller

​3.2 Characteristics

​3.3 Limitations

​4. Machine Learning (ML)

​4.1 Types of Machine Learning

​5. Deep Learning (DL)

​5.1 Why Deep Learning Succeeded

​6. Natural Language Processing (NLP)

​6.1 Why Human Language is Difficult for Computers

​7. The Attention Mechanism & Transformers

​7.1 Self-Attention Illustration

​8. Practice Exercises

​Practice 1: Traditional vs. ML Sentiment