RAG (Retrieval-Augmented Generation) tutorial

1. What is RAG (Retrieval-Augmented Generation)?

Idea:

Instead of asking an LLM to answer only from its own training, you:

Retrieve relevant documents from your data (PDFs, DB, website, etc.)
Augment the prompt with those docs
Let the Generator (LLM) answer using that context.

So pipeline is:

User query ➜ Retrieve top-k docs ➜ Build prompt with docs ➜ LLM answers

This solves:

Outdated model knowledge
Hallucinations (partially)
Domain customization (company docs, reports, codebase)

2. High-Level RAG Architecture

Think of 2 phases:

2.1. Offline / One-Time: Build Knowledge Store

Collect data
- PDFs, HTML pages, markdown, CSVs, Confluence, Notion, etc.
Chunking
- Split into small text chunks (e.g., 200–500 tokens) with overlap.
- Why: too big ➜ irrelevant noise; too small ➜ not enough context.
Embed chunks
- Use an embedding model (e.g., OpenAI text-embedding, or local like bge, all-MiniLM, etc.).
- For each chunk:
  - embedding = embedding_model(chunk_text)
Store in Vector DB
- Store: (id, chunk_text, metadata, embedding) in:
  - e.g. Chroma, Qdrant, Weaviate, Milvus, PostgreSQL + pgvector, Pinecone, etc.

2.2. Online / Query Time: Answer Workflow

User asks a question: q = "Explain tax rules for freelancers in India"
Embed the query:
- q_emb = embedding_model(q)
Vector search (kNN) in vector DB:
- Retrieve top-k chunks most similar to q_emb.
Build the prompt:
- System + Instructions
- Context (retrieved docs)
- User question
Call LLM with this prompt → generated answer.

3. Detailed RAG Steps (End-to-End)

Step 1: Data Ingestion

Load from:
- Local files (PDF, DOCX)
- Web pages (scraping)
- APIs / DB (SQL queries)
Clean & normalize:
- Remove boilerplate, menus, ads
- Fix encodings, newlines, etc.

Tips:

Keep source info: filename, URL, section headings → saves you later for citations.

Step 2: Chunking Strategy

Why chunk? LLMs + vector search work better with semantically coherent segments.

Common patterns:

Fixed token/word chunks
- e.g. 300 tokens with 50-token overlap.
By structure
- For docs/manuals: chunk by headings (H1/H2/H3) and then sub-split if large.
By semantic boundaries (advanced)
- Using sentence boundaries / semantic segmentation.

Trade-offs:

Large chunks: richer context but retrieval less precise, may hit token limits.
Small chunks: more precise retrieval but answers may need many chunks.

A good starting point:

300–500 tokens per chunk, 10–20% overlap.

Step 3: Embeddings

What: map text to high-dimensional vectors.
Goal: similar meaning → close vectors.

You need:

A query encoder: encode user queries.
A document encoder: encode chunks.

Sometimes same model does both.

Key points:

Choose embedding model based on:
- Language support
- License (open / closed)
- Latency & cost
Normalize vectors (L2 normalize) if your DB requires/can benefit.

Step 4: Vector Store Design

Each entry typically has:

id
text (chunk)
embedding
metadata:
- source (file, URL)
- page/section
- tags (doc type, date, language, etc.)

Index types:

HNSW, IVF, PQ, etc. (handled by the vector DB)

Popular choices:

Chroma, Qdrant, Weaviate, Milvus
Postgres + pgvector (good if you already use Postgres)
Pinecone, Elastic, etc. (managed services)

Step 5: Retrieval

At query time:

Embed the query.
Vector search:
- Top-k similar chunks (e.g., k = 5–10).
Optional filters:
- Metadata filter: {source="handbook", date > 2023-01-01}
- User/tenant filters for multi-tenant apps.

Advanced retrieval tricks:

Hybrid search: combine dense (vector) + sparse (BM25) scores.
Reranking:
- Use a cross-encoder to re-score top 20 docs and pick top 5.
- Increases quality with extra latency.

Step 6: Prompt Construction

Typical format:

You are a helpful assistant. Use ONLY the context below to answer.
If answer is not in the context, say you don't know.

Context:
[Doc 1]
<chunk text 1>

[Doc 2]
<chunk text 2>

[Doc 3]
<chunk text 3>

Question:
<user question>

Answer:

Guidelines:

Clearly tell LLM:
- Use only given context.
- Admit when info not present.
Include citations:
- E.g., “[Source: Doc 2]” or “[1] [2]” mapping to metadata.

Step 7: Generation & Post-Processing

LLM generates answer.

Post-processing optional:

Add citations (map “[1]” to URLs/pages).
Formatting:
- Markdown, bullets, tables, JSON, etc.
Guardrails:
- Check answer length, style, safety filters.

4. Types of RAG (Upgrades)

Once basic RAG works, you can move to more advanced patterns.

4.1. Naive RAG (Basic)

One query → one vector search → one prompt → one answer.

Good for:

FAQ bots
Simple knowledge base Q&A
Small to medium datasets

4.2. Conversational RAG

Add chat history:

Summarize previous turns or include last N messages.
Use history-aware retriever:
- Condense conversation into a standalone query (“query rewriting”).
- Then retrieval with that rewritten query.

Pipeline:

User says: “What about point 3?”
LLM rewrites: “What is point 3 in the refund policy for Indian customers?”
Do retrieval with rewritten query.
Answer with context.

4.3. Multi-Hop / Iterative RAG

For complex questions requiring multiple facts:

Do multiple retrieval + reasoning steps:
- Step 1: retrieve docs related to entity A.
- Step 2: from those, extract clue -> new query -> retrieve about entity B.
Implemented via:
- Tool-calling agents
- Chains / graphs (LangChain, LlamaIndex, custom)

4.4. Structured / Tool RAG

Instead of dumping raw text to LLM:

Transform retrieved data into structured JSON/tables.
Or let LLM call tools:
- “search_docs(query)”
- “get_user_profile(id)”
Very useful for:
- Analytics
- Code search
- Support bots integrated with backend.

5. RAG Evaluation (Very Important in Practice)

Don’t just eyeball outputs. Set up automatic evaluation.

5.1. Components to Evaluate

Retrieval Quality
- Are we retrieving correct/needed chunks?
- Metrics: Recall@k, MRR, nDCG (if you have labeled data).
Answer Quality
- Faithfulness: does answer stick to retrieved docs?
- Relevance: does it actually answer the question?
- Conciseness & style.

5.2. How to Evaluate

Build test set:
- (question, gold answer, optional gold evidence docs).
Use:
- LLM-as-judge to score (faithfulness, correctness).
- Or human evaluation on a subset.

5.3. Debugging RAG

If answer is wrong:

Did we retrieve the right docs?
If yes → prompting/LLM issue.
If no → improve:
- chunking
- embeddings
- retrieval parameters (k, filters, hybrid/rerank).

6. Common Pitfalls & How to Fix

Pitfall 1: Bad Chunking

Symptoms:
- RAG returns random or partial answers.
Fix:
- Use sentence-aware chunking.
- Preserve headings and structure.
- Add overlap.

Pitfall 2: Too Few / Too Many Documents

Too few (k=1–2): missing info.
Too many (k=20): noisy prompt, token overflow.

Fix:
Start with k=5–8, tune by evaluating.

Pitfall 3: Model Hallucinations Still Happen

LLM makes up answers even if context doesn’t support.

Fixes:

Strong instructions: “If not in context, say ‘I don’t know’.”
Ask LLM to cite exact source snippet.
Use a verification step: another LLM checks if answer is grounded in context.

Pitfall 4: Poor Relevance for Short or Vague Queries

“Tell me more” or “Explain that”

Fixes:

Query rewriting (history-aware retriever).
Prompt LLM: “Rewrite user’s last message as a standalone question using chat history.”

7. Minimal “Mental Model” of RAG

If you remember nothing else, remember this:

RAG = Search + LLM

Search = vector DB + retrieval logic
LLM = reasoning + language + formatting

Your job as ML/AI engineer:

Make search strong (data, chunking, embeddings, retrieval).
Make prompts clear and grounded.
Add evaluation to know when it fails.

8. If You Want a Coding Direction (High-Level Pseudocode)

Just to connect concepts:

# Offline
docs = load_documents()
chunks = chunk_documents(docs)
embs = embed(chunks)
store_in_vector_db(chunks, embs, metadata)

# Online
def answer_question(question):
    q_emb = embed([question])
    top_chunks = vector_db.search(q_emb, k=5)
    context = format_context(top_chunks)
    prompt = build_prompt(context, question)
    answer = call_llm(prompt)
    return answer

Perfect, let’s turn that RAG-with-PDF example into a clear step-by-step checklist you can literally follow on your machine.

🧩 Goal

We’ll build a minimal RAG pipeline:

Input: a PDF
Offline: read → chunk → embed → build FAISS index
Online: ask questions → retrieve top chunks → send as context to LLM (stub you can later replace)

✅ Step 1 – Create Project Folder

In your terminal:

mkdir rag_pdf_example
cd rag_pdf_example

Create subfolders:

mkdir data index

So structure is:

rag_pdf_example/
  data/
  index/

✅ Step 2 – Put Your PDF in `data/`

Copy any PDF you want to query into the data folder and rename it to my_doc.pdf (or keep another name but then update the code).

Example:

rag_pdf_example/
  data/
    my_doc.pdf   ← your document
  index/

✅ Step 3 – Create Virtual Environment (Optional but Recommended)

python -m venv .venv
source .venv/bin/activate    # Mac/Linux
# .venv\Scripts\activate     # Windows

✅ Step 4 – Install Required Libraries

pip install pypdf sentence-transformers faiss-cpu numpy

If faiss-cpu gives trouble on Windows, tell me and I’ll give you alternatives (like using a simple cosine similarity without FAISS).

✅ Step 5 – Create `build_index.py`

In rag_pdf_example/, create a file called build_index.py and paste this:

# build_index.py
import os
from pathlib import Path

import numpy as np
import faiss
from pypdf import PdfReader
from sentence_transformers import SentenceTransformer

# ---------- CONFIG ----------
PDF_PATH = "data/my_doc.pdf"
INDEX_DIR = "index"
CHUNK_SIZE = 500     # characters per chunk
CHUNK_OVERLAP = 100  # overlap between chunks
EMBEDDING_MODEL_NAME = "sentence-transformers/all-MiniLM-L6-v2"
# ----------------------------

def read_pdf(pdf_path: str) -> str:
    reader = PdfReader(pdf_path)
    pages = [page.extract_text() or "" for page in reader.pages]
    return "\n".join(pages)

def chunk_text(text: str, chunk_size: int, overlap: int):
    """
    Simple character-based chunking with overlap.
    """
    chunks = []
    start = 0
    text_length = len(text)

    while start < text_length:
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk.strip())
        start = end - overlap  # move back by overlap for next chunk

        if start < 0:
            start = 0

    return [c for c in chunks if c]  # remove empty chunks

def main():
    os.makedirs(INDEX_DIR, exist_ok=True)

    print(f"Reading PDF from {PDF_PATH}...")
    full_text = read_pdf(PDF_PATH)

    print("Chunking text...")
    chunks = chunk_text(full_text, CHUNK_SIZE, CHUNK_OVERLAP)
    print(f"Total chunks: {len(chunks)}")

    print(f"Loading embedding model: {EMBEDDING_MODEL_NAME}...")
    model = SentenceTransformer(EMBEDDING_MODEL_NAME)

    print("Embedding chunks...")
    embeddings = model.encode(
        chunks,
        batch_size=32,
        show_progress_bar=True,
        convert_to_numpy=True
    )
    dim = embeddings.shape[1]
    print(f"Embedding dimension: {dim}")

    print("Building FAISS index...")
    index = faiss.IndexFlatL2(dim)
    index.add(embeddings)

    # Save FAISS index
    faiss.write_index(index, os.path.join(INDEX_DIR, "docs.index"))

    # Save chunks as metadata
    np.save(os.path.join(INDEX_DIR, "chunks.npy"), np.array(chunks, dtype=object))

    print("✅ Index built and saved in 'index/'")

if __name__ == "__main__":
    main()

✅ Step 6 – Run the Index Builder

From inside rag_pdf_example/:

python build_index.py

You should see logs like:

Reading PDF from data/my_doc.pdf...
Chunking text...
Total chunks: X
Embedding chunks...
Building FAISS index...
✅ Index built and saved in 'index/'

After success, index/ will have:

docs.index
chunks.npy

✅ Step 7 – Create `query_rag.py`

Now create another file query_rag.py in the same folder and paste:

# query_rag.py
import os
import numpy as np
import faiss
from sentence_transformers import SentenceTransformer

# ---------- CONFIG ----------
INDEX_DIR = "index"
EMBEDDING_MODEL_NAME = "sentence-transformers/all-MiniLM-L6-v2"
TOP_K = 5
# ----------------------------

def load_index_and_chunks():
    index_path = os.path.join(INDEX_DIR, "docs.index")
    chunks_path = os.path.join(INDEX_DIR, "chunks.npy")

    if not (os.path.exists(index_path) and os.path.exists(chunks_path)):
        raise FileNotFoundError("Index not found. Run build_index.py first.")

    index = faiss.read_index(index_path)
    chunks = np.load(chunks_path, allow_pickle=True).tolist()
    return index, chunks

def retrieve_chunks(query: str, model, index, chunks, top_k: int = TOP_K):
    # Embed query
    query_emb = model.encode([query], convert_to_numpy=True)
    # Search FAISS
    distances, indices = index.search(query_emb, top_k)  # shape: (1, top_k)
    indices = indices[0]
    distances = distances[0]

    retrieved = []
    for rank, (idx, dist) in enumerate(zip(indices, distances), start=1):
        if idx == -1:
            continue
        retrieved.append(
            {
                "rank": rank,
                "chunk_index": int(idx),
                "distance": float(dist),
                "text": chunks[idx],
            }
        )
    return retrieved

def build_context(retrieved_chunks):
    """
    Concatenate retrieved chunks into a context string.
    """
    context_parts = []
    for item in retrieved_chunks:
        context_parts.append(f"[Chunk {item['rank']}]\n{item['text']}\n")
    return "\n".join(context_parts)

# ---------- LLM STUB ----------
def generate_answer(question: str, context: str) -> str:
    """
    Replace this stub with a real LLM call (OpenAI, local model, etc.).
    For now, we just show the prompt that would be sent.
    """
    prompt = f"""You are a helpful assistant. Use ONLY the context below to answer.

Context:
{context}

Question:
{question}

Answer in a concise, clear way. If the answer is not in the context, say you don't know.
"""
    # Here you would call your LLM. For demo, we just return the prompt:
    return f"(This is where your LLM answer would go.)\n\nPROMPT SENT TO LLM:\n\n{prompt}"
# ------------------------------

def main():
    print("Loading index and chunks...")
    index, chunks = load_index_and_chunks()

    print(f"Loading embedding model: {EMBEDDING_MODEL_NAME}...")
    model = SentenceTransformer(EMBEDDING_MODEL_NAME)

    while True:
        question = input("\nAsk a question about the PDF (or type 'exit'): ").strip()
        if question.lower() in {"exit", "quit"}:
            break

        print("Retrieving relevant chunks...")
        retrieved = retrieve_chunks(question, model, index, chunks, TOP_K)

        print(f"Retrieved {len(retrieved)} chunks.")
        context = build_context(retrieved)

        answer = generate_answer(question, context)
        print("\n--- RAG Answer ---")
        print(answer)
        print("------------------")

if __name__ == "__main__":
    main()

✅ Step 8 – Run the RAG Query Script

Again from inside rag_pdf_example/:

python query_rag.py

You’ll see:

Loading index and chunks...
Loading embedding model: sentence-transformers/all-MiniLM-L6-v2...

Then it will wait for your question:

Ask a question about the PDF (or type 'exit'):

Try questions like:

What is this document about?
Explain the main steps of the process described.
What does it say about refunds / chapter 2 / section 5? (depends on your PDF)

The script will:

Embed your question
Retrieve top-k similar chunks
Build a context
Show a “fake” answer with the full prompt (so you can see what would be sent to a real LLM)

✅ Step 9 – (Optional) Plug in a Real LLM

Right now generate_answer() is a stub.

If you’re using, say, OpenAI’s API, you could replace it like this (sketch):

from openai import OpenAI
client = OpenAI()

def generate_answer(question: str, context: str) -> str:
    system_msg = "You are a helpful assistant. Answer using ONLY the provided context."

    user_msg = f"""Context:
{context}

Question:
{question}

Answer in a concise, clear way. If the answer is not in the context, say you don't know.
"""

    completion = client.chat.completions.create(
        model="gpt-4.1-mini",
        messages=[
            {"role": "system", "content": system_msg},
            {"role": "user", "content": user_msg},
        ],
    )
    return completion.choices[0].message.content

Then when you run query_rag.py, you’ll get real, LLM-generated answers based on your PDF chunks.

✅ Step 10 – Recap (What You Just Built)

Data ingestion – read PDF with pypdf
Chunking – split into overlapping character chunks
Embeddings – use sentence-transformers to encode chunks
Indexing – store vectors in a FAISS index + chunks in .npy
Retrieval – at query time, embed question, search FAISS for top-k chunks
Prompt building – combine retrieved chunks into context string
Generation – send context + question to LLM (currently stub, easily replaceable)

That’s a full RAG loop around a single PDF 🎯