> ## Documentation Index
> Fetch the complete documentation index at: https://genai.codewithsiva.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# The Transformer Model & Architectures

> Dive into Transformer components and compare encoder, decoder, and sequence-to-sequence model variants

The **Transformer** (introduced in the 2017 research paper *"Attention Is All You Need"*) is the core neural network architecture behind all modern Large Language Models (LLMs). This page explores its primary components and architectural variants.

## 1. Core Transformer Components

The Transformer architecture reads whole sequences of tokens in parallel, processing them through several key layers:

```text theme={null}
Input Tokens ──> [ Token Embeddings ] ──> [ Positional Information ] ──> [ Self-Attention ] ──> [ Feed-Forward Networks ] ──> Output
```

### 1.1 Token Embeddings

Computers cannot read text; they only understand numbers. Embeddings map each token into a high-dimensional vector of real numbers (e.g. 768 or 1536 dimensions) representing its semantic meaning.

* **Semantic Mapping**: Words with similar meanings are mapped to vectors that are physically close in vector space (e.g., the distance between `"cat"` and `"feline"` is very small).

### 1.2 Positional Information (Positional Encodings)

Because Transformers process all tokens in parallel, they do not inherently know the order of words.

* For example: *"Dog bites man"* and *"Man bites dog"* contain the exact same tokens, but opposite meanings.
* **Positional Encodings** are vectors added directly to the token embeddings to represent the position of each word in the sequence, preserving syntactic order.

### 1.3 Self-Attention

Self-attention calculates how much context each word should gather from all other words in the same sequence.

* In the sentence: *"The server was down, so the database crashed,"* self-attention links the context of `"crashed"` back to `"database"` and `"server"`.

## 2. Encoder vs. Decoder Architectures

The original Transformer consists of two main halves: an **Encoder** and a **Decoder**. Depending on the task, models are built using one or both of these components:

| Architecture        | Focus                        | Primary Goal                                                                                 | Examples                       |
| ------------------- | ---------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------ |
| **Encoder-Only**    | Bidirectional Understanding  | Read context left-to-right and right-to-left to construct rich representations of input.     | BERT, RoBERTa                  |
| **Decoder-Only**    | Autoregressive Generation    | Predict the next token sequentially from left to right. Ideal for text generation and chat.  | GPT-4, Llama 3, Mistral        |
| **Encoder-Decoder** | Sequence-to-Sequence Mapping | The Encoder understands the input sequence, and the Decoder generates a new output sequence. | T5, BART, Original Transformer |

## 3. Deep-Dive: Architectural Roles

### 3.1 Encoder-Only Models

* **Mechanism**: Reads input bidirectionally to output a dense representation.
* **Typical Tasks**: Classification, semantic search, sentiment analysis, entity extraction.
* **Illustration**:
  ```text theme={null}
  "Python is easy to learn" ──> [ Encoder ] ──> [ Semantic Vector Representation ]
  ```

### 3.2 Decoder-Only Models

* **Mechanism**: Predicts the next token based on the prompt history. Each generated token is appended to the prompt in an autoregressive loop.
* **Typical Tasks**: Chatbots, creative writing, code generation, step-by-step reasoning.
* **Illustration**:
  ```text theme={null}
  Prompt ──> [ Decoder ] ──> Next Token ──> Append ──> Next Token ...
  ```

### 3.3 Encoder-Decoder Models

* **Mechanism**: The encoder maps input text into a vector representation, which the decoder uses to generate a new output sequence.
* **Typical Tasks**: Language translation, summarization, document refactoring.
  ```text theme={null}
  Input Text ──> [ Encoder ] ──> [ Latent Space Representation ] ──> [ Decoder ] ──> Translated Text
  ```

## 4. What is a Large Language Model (LLM)?

An **LLM** is a scaled-up, Decoder-Only or Encoder-Decoder Transformer model trained on massive text corpora.

### Why are they called "Large"?

"Large" refers to several dimensions of the model's footprint:

* **Parameters**: The millions or billions of mathematical weights inside the neural network that are adjusted during training.
* **Training Data**: Petabytes of text scraped from the internet, books, and code.
* **Compute Power**: Thousands of specialized GPUs running for weeks or months.

```text theme={null}
Model Architecture + Training Data + Compute + Training Method = Final LLM Capability
```

## 5. Practice Exercises

### Practice 1: Choosing Architectural Variants

For each task below, choose whether an **Encoder-Only** or a **Decoder-Only** model is more appropriate:

1. Detecting spam emails.
2. Answering a user's question conversationally.
3. Extracting the names of companies from a legal contract.

<Accordion title="Solution">
  1) **Detecting spam**: **Encoder-Only**, because it excels at classification and reading the entire email context bidirectionally.
  2) **Answering conversationally**: **Decoder-Only**, because it is optimized for autoregressive next-token text generation.
  3) **Extracting names**: **Encoder-Only**, because entity extraction is an understanding and labeling task over the input tokens.
</Accordion>