1. Tokenization & Token IDs
LLMs do not understand raw characters or words. They process text in chunks called tokens.1.1 What is a Token?
A token can represent:- A complete word (e.g.,
"Hello") - Part of a word (e.g.,
"ing") - A single character, punctuation mark, or whitespace.
2. The Context Window
The context window is the maximum memory capacity (measured in tokens) that an LLM can process in a single request.2.1 What the Context Window Contains:
- System Instructions
- User Prompts
- Conversation History
- Retrieval Documents (e.g. RAG context)
- Output Generated Response
[!IMPORTANT] A larger context window allows you to feed more documents to the model, but it does not automatically improve reasoning. In addition, processing larger context windows increases latency and token cost.
3. The LLM Training Lifecycle
Training an LLM requires three core ingredients: massive datasets, large-scale compute clusters (GPUs/TPUs), and training algorithms.3.1 Pre-Training (Self-Supervised)
- Goal: Learn language grammar, syntax, world facts, and reasoning patterns.
- Method: Predict the next token over billions of webpages, books, and code repos.
- Output: A Base Model (autocomplete engine) that has deep language understanding but does not know how to follow instructions.
3.2 Supervised Fine-Tuning (SFT)
- Goal: Teach the base model to follow commands, answer questions, and output structured templates.
- Method: Train on high-quality, curated pairs of instructions and ideal responses.
3.3 Alignment (RLHF / DPO)
- Goal: Teach the model safety, helpfulness, and style preference.
- Method: Use Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO) based on human reviews of model completions.
4. Training vs. Inference Requirements
- Training is the process of learning the model parameters (weights).
- Requirements: Hundreds of GPUs (like Nvidia H100s) connected by high-bandwidth networks, petabytes of data, and millions of dollars in capital. This is why only a few large companies (Google, Meta, OpenAI) train frontier base models.
- Inference is the process of calling an already-trained model to generate responses.
- Requirements: A single GPU or a cloud API. While much cheaper than training, large-scale inference still incurs computing costs, which is why providers charge per token.
5. Model Hosting & Access Options
When building applications, you can access LLMs via three hosting patterns:5.1 Proprietary APIs (Closed-Source)
- How it works: Models hosted and managed by third-party providers (Google, OpenAI, Anthropic).
- Pros: Best-in-class capability, easy API integration, no server maintenance.
- Cons: Per-token fees, dependency on vendors, data privacy constraints.
5.2 Open-Weight Models
- How it works: Models whose weights are released publicly (Llama 3, Gemma, Mistral) for you to host on your own cloud servers.
- Pros: Full control over data privacy, customizable model weights.
- Cons: You must configure and pay for server hosting infrastructure (GPUs).
5.3 Local Models
- How it works: Running small models directly on your developer machine using tools like Ollama, LM Studio, or vLLM.
- Pros: Free, completely private, works offline.
- Cons: Limited by your computer’s RAM and GPU capability.
6. Practice Exercises
Practice 1: Pre-trained vs. Chat Models
Why is a raw “pre-trained” base model unsuitable for a conversational chatbot application?Solution
Solution
A pre-trained base model is designed strictly as a text autocompletion engine. If a user asks a question like “What is the capital of France?”, the base model might respond by autocompleting it with a list of other questions (e.g., “What is the capital of Germany? What is the capital of Italy?”), rather than providing the answer. It requires Supervised Fine-Tuning (SFT) to learn instruction-following chat behavior.