Skip to main content
The Transformer (introduced in the 2017 research paper “Attention Is All You Need”) is the core neural network architecture behind all modern Large Language Models (LLMs). This page explores its primary components and architectural variants.

​
1. Core Transformer Components

The Transformer architecture reads whole sequences of tokens in parallel, processing them through several key layers:

​
1.1 Token Embeddings

Computers cannot read text; they only understand numbers. Embeddings map each token into a high-dimensional vector of real numbers (e.g. 768 or 1536 dimensions) representing its semantic meaning.
  • Semantic Mapping: Words with similar meanings are mapped to vectors that are physically close in vector space (e.g., the distance between "cat" and "feline" is very small).

​
1.2 Positional Information (Positional Encodings)

Because Transformers process all tokens in parallel, they do not inherently know the order of words.
  • For example: “Dog bites man” and “Man bites dog” contain the exact same tokens, but opposite meanings.
  • Positional Encodings are vectors added directly to the token embeddings to represent the position of each word in the sequence, preserving syntactic order.

​
1.3 Self-Attention

Self-attention calculates how much context each word should gather from all other words in the same sequence.
  • In the sentence: “The server was down, so the database crashed,” self-attention links the context of "crashed" back to "database" and "server".

​
2. Encoder vs. Decoder Architectures

The original Transformer consists of two main halves: an Encoder and a Decoder. Depending on the task, models are built using one or both of these components:

​
3. Deep-Dive: Architectural Roles

​
3.1 Encoder-Only Models

  • Mechanism: Reads input bidirectionally to output a dense representation.
  • Typical Tasks: Classification, semantic search, sentiment analysis, entity extraction.
  • Illustration:

​
3.2 Decoder-Only Models

  • Mechanism: Predicts the next token based on the prompt history. Each generated token is appended to the prompt in an autoregressive loop.
  • Typical Tasks: Chatbots, creative writing, code generation, step-by-step reasoning.
  • Illustration:

​
3.3 Encoder-Decoder Models

  • Mechanism: The encoder maps input text into a vector representation, which the decoder uses to generate a new output sequence.
  • Typical Tasks: Language translation, summarization, document refactoring.

​
4. What is a Large Language Model (LLM)?

An LLM is a scaled-up, Decoder-Only or Encoder-Decoder Transformer model trained on massive text corpora.

​
Why are they called “Large”?

“Large” refers to several dimensions of the model’s footprint:
  • Parameters: The millions or billions of mathematical weights inside the neural network that are adjusted during training.
  • Training Data: Petabytes of text scraped from the internet, books, and code.
  • Compute Power: Thousands of specialized GPUs running for weeks or months.

​
5. Practice Exercises

​
Practice 1: Choosing Architectural Variants

For each task below, choose whether an Encoder-Only or a Decoder-Only model is more appropriate:
  1. Detecting spam emails.
  2. Answering a user’s question conversationally.
  3. Extracting the names of companies from a legal contract.
  1. Detecting spam: Encoder-Only, because it excels at classification and reading the entire email context bidirectionally.
  2. Answering conversationally: Decoder-Only, because it is optimized for autoregressive next-token text generation.
  3. Extracting names: Encoder-Only, because entity extraction is an understanding and labeling task over the input tokens.