Skip to main content
Large Language Models (LLMs) are incredibly powerful, but they have two core limitations: they suffer from hallucinations (fabricating facts convincingly) and their knowledge is static (limited to their pre-training training cut-off date). Retrieval-Augmented Generation (RAG) solves these problems by grounding model answers in private, external data.

​
1. What is RAG & Why is it Needed?

Instead of relying solely on the LLM’s internal weights to generate answers, RAG queries an external datasource to retrieve relevant documents matching the user’s question, and then passes those documents to the LLM as context.

​
Why not just fine-tune?

  • Real-time Updates: RAG can access live data (like database records or APIs) instantly. Fine-tuning is static and slow.
  • Cost & Time: Fine-tuning requires renting GPUs and running training runs. RAG connects to databases dynamically at runtime.
  • Access Control: RAG lets you filter documents based on user roles (e.g. Employee A cannot query Employee B’s salary documents). Fine-tuned weights expose all information to all users.
  • Verifiability: RAG outputs can cite source documents (e.g., “According to Page 12 of the HR Manual…”), whereas fine-tuned outputs cannot be traced.

​
2. The 3 Pillars of RAG

Every RAG application follows a standard three-step workflow pipeline:
  1. Ingestion: Reading raw documents (PDFs, Markdown, wikis), breaking them into smaller chunks, converting those chunks into vector embeddings, and indexing them in a database.
  2. Retrieval: When a user asks a question, the system converts the query into a vector and searches the database to find the top-K most similar document chunks.
  3. Generation: The retrieved document chunks are formatted into a prompt along with the user’s question, and sent to the LLM to generate a factual, grounded response.
RAG systems rely on Semantic Search (dense retrieval) rather than traditional keyword-matching search:
  • Traditional (Lexical) Search: Looks for exact word matches (e.g. TF-IDF or BM25). If you search for “automobile repair”, it will miss documents containing “car mechanics” because the exact characters do not match.
  • Semantic Search: Converts text into vector embeddings representing meaning. It knows that “automobile” and "car" are conceptually close, returning relevant matches even without direct keyword intersections.

​
4. Practice Exercises

​
Practice 1: Hallucination Mitigation

Explain how RAG reduces the occurrence of model hallucinations compared to open-ended generation.
  • Open-ended Generation: The model relies on predicting the next token based purely on its training parameters. If it doesn’t know a fact, it continues predicting the most statistically probable next words, generating incorrect facts (hallucinations).
  • RAG: The model is restricted to a prompt template instruction (e.g., “Answer the query ONLY using the provided context.”). Since the facts are supplied directly in the prompt, the model acts as a summarization/synthesizer, drastically reducing fabricated claims.