RAG Systems Explained: Python Guide

RAG Systems: How They Work and a Minimal Python Implementation

Language models are limited by their training data: knowledge is static, doesn't cover current events, and prone to hallucinations—making up facts. The context window makes it worse by dropping input when limits are hit. Retrieval-Augmented Generation (RAG) fixes this by pulling relevant context from external sources during generation, cutting errors and providing source citations.

RAG connects documents, databases, or code repos to make responses verifiable and up-to-date. It's essential for niche domains the model wasn't trained on.

Chunking Documents

Chunking breaks text into fragments (chunks) for indexing. LLMs handle a limited set of chunks, so chunk quality determines fact accessibility.

Large chunks add noise, raise costs, and hurt search accuracy. Small ones break context, splitting related ideas. Strategies:

• Fixed character or token length.
• Semantic boundaries (paragraphs, headings).
• Overlap to keep continuity.

Tune parameters empirically for your corpus and queries.

RAG Workflow Stages

RAG covers preprocessing, retrieval, and generation:

• Indexing: Chunks turn into vectors and store in a vector DB.
• Retrieval: Query gets vectorized, top relevant chunks pulled.
• Augmentation: Results ranked, duplicates and noise cleaned.
• Generation: Context added to query for the LLM.

Basic Naive RAG keeps it simple. Advanced RAG adds query rewriting and reranking. Modular RAG uses modules like routing and result fusion.

RAG vs. Fine-Tuning

RAG expands knowledge access without model changes, unlike fine-tuning which needs data and heavy compute. Prompt engineering tweaks queries but doesn't add external data.

| Approach | Pros | Cons |

|----------|------|------|

| RAG | Fresh data, low cost, citations | Relies on retrieval quality |

| Fine-tuning | Deep customization | Resource-heavy, gets outdated |

RAG shines with dynamic data; fine-tuning for static domains.

Minimal RAG Pipeline: Hands-On

We'll build a Naive RAG in Python using Wikipedia (500 articles) and Qwen2.5-3B-Instruct. Leveraging LangChain, Chroma, and sentence-transformers.

Step 1: Load and Chunk

import torch
from datasets import load_dataset
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.documents import Document

device = "cuda" if torch.cuda.is_available() else "cpu"


dataset = load_dataset(
    "wikimedia/wikipedia",
    "20231101.ru",
    split="train[:500]"
)

documents = [
    Document(
        page_content=row["text"],
        metadata={"title": row["title"]}
    )
    for row in dataset
]

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=100
)

chunks = splitter.split_documents(documents)

print("Chunks:", len(chunks))

Yields ~17,892 chunks of 800 characters with 100-char overlap.

Step 2: Vector Index

from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings

embedding_model = HuggingFaceEmbeddings(
    model_name="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
    model_kwargs={"device": device}
)

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embedding_model
)

Multilingual MiniLM for embeddings, Chroma for storage.

Step 3: Retrieval

import textwrap

def search(query, k=3, width=80):
    results = vectorstore.similarity_search(query, k=k)

    for i, doc in enumerate(results, 1):
        print(f"\n--- Result {i} ---")
        print("Title:", doc.metadata["title"])
        print(textwrap.fill(doc.page_content[:1000], width))

search("Who is Pushkin?")

Pulls relevant snippets from Pushkin articles.

Step 4: Generation

Add an LLM to generate from retrieved chunks (code omitted for brevity; full version in source).

Key Takeaways

• RAG cuts hallucinations with external context, no model tweaks needed.
• Success hinges on chunking and embeddings: tailor to your domain.
• Start with Naive RAG, scale to Advanced/Modular for production.
• LangChain simplifies prototyping.
• Test on real datasets to optimize.

— Editorial Team