RAG: smart document search for developers

Retrieval-Augmented Generation: Smart Document Search for Enterprises

RAG combines vector search with generative AI to deliver precise answers from private document repositories. The system retrieves relevant content from PDFs, Excel files, code, and more, then generates explanations strictly based on source material—significantly reducing hallucinations.

The process begins with preprocessing: documents are parsed, split into intelligent chunks considering content type, and converted into embeddings stored in a vector database.

Text Extraction from File Formats

The system supports parsing multiple formats without data loss. Each format uses a dedicated extractor.

SUPPORTED = {
    '.pdf':  lambda f: ''.join(p.extract_text() or '' for p in PyPDF2.PdfReader(f).pages),
    '.docx': lambda f: '\n'.join(p.text for p in Document(f).paragraphs),
    '.pptx': lambda f: '\n'.join(
        shape.text
        for slide in Presentation(f).slides
        for shape in slide.shapes
        if hasattr(shape, "text") and shape.text.strip()
    ),
    '.xlsx': extract_xlsx,
    '.csv':  extract_csv,
    # .txt, .json, .py, .js, etc.
}

Output is clean, ready-to-process text.

Intelligent Chunking by Content Type

Text isn’t split by fixed size—it’s segmented semantically. First, the content type is detected:

def detect_content_type(text):
    if re.search(r'<table|</td>|<tr>', text, re.IGNORECASE):
        return 'table'
    tab_lines = sum(1 for line in text.splitlines() if '\t' in line)
    if tab_lines > max(3, len(text.splitlines()) * 0.4):
        return 'table'
    if re.search(r'function\s+\w+\s*\(|\bdef\s+\w+\s*\(|\bclass\s+\w+\b', text):
        return 'code'
    return 'prose'

• Tables: Include repeated headers in each chunk
• Code: Split by functions/classes without breaking logic
• Prose: Semantic boundaries using sentence embeddings

For tables:

def split_table_by_rows(text, chunk_size):
    lines = [l for l in text.splitlines() if l.strip()]
    header = lines[0]
    chunks = []
    cur_lines = [header]
    for line in lines[1:]:
        if sum(len(l) for l in cur_lines) + len(line) >= chunk_size:
            chunks.append('\n'.join(cur_lines))
            cur_lines = [header, line]
        else:
            cur_lines.append(line)
    if cur_lines:
        chunks.append('\n'.join(cur_lines))
    return chunks

Semantic chunking uses cosine similarity between adjacent sentences with an adaptive threshold (25th percentile).

Embedding Generation

Chunks are vectorized using models like text-embedding-3-small (1536 dimensions). Vectors and metadata are stored in the database.

def get_embeddings(texts, batch_size=100):
    embeddings = []
    for i in range(0, len(texts), batch_size):
        resp = requests.post(
            "https://openrouter.ai/api/v1/embeddings",
            headers=HEADERS,
            json={
                "model": "openai/text-embedding-3-small",
                "input": texts[i:i + batch_size],
                "encoding_format": "float"
            },
        ).json()
        embeddings.extend(d["embedding"] for d in resp.get("data", []))
    return np.array(embeddings)

Multi-Stage Search

The query is transformed into a vector, reformulated into three variations via LLM, then processed through hybrid search.

Hybrid Search: Vectors + BM25

• Vector search: Cosine similarity in embedding space
• BM25: TF-IDF with k1=1.5, b=0.75 for exact matches

Results are fused using RRF:

# RRF fusion
for rank, idx in enumerate(sorted_by_vector):
    rrf_scores[idx] += 1.0 / (k_rrf + rank)
for rank, idx in enumerate(sorted_by_bm25):
    rrf_scores[idx] += bm25_normalized[idx] * (1.0 / (k_rrf + rank))

Re-ranking and Filters

Top 50 candidates are evaluated by LLM, selecting 5–10 best. Forced inclusion rules:

• Quotes in "quotation marks"
• Numbers/IDs (regex: \b(№|N|number)\s*\d+\b)

quote_matches = re.findall(r'«([^»]+)»|"([^"+])"', query)
for match in quote_matches:
    quote = (match[0] or match[1]).strip()
    if len(quote) > 5:
        for i, chunk in enumerate(chunks_data):
            if quote in chunk['content']:
                forced_indices.append(i)
                break

Performance Optimizations

Hierarchical Search

First, top documents are identified via annotations; then, chunks within them are searched.

doc_summaries = [chunks_data_by_doc[doc_id][0]['content'][:500] for doc_id in all_doc_ids]
doc_embs = get_embeddings(doc_summaries)
sims = np.dot(doc_embs_normalized, query_embedding_normalized)
top_doc_ids = [all_doc_ids[i] for i in np.argsort(sims)[::-1][:5]]
chunks_data = [ch for ch in chunks_data if ch['doc_id'] in top_doc_ids]

Iterative Search

For complex queries, multiple rounds refine results using LLM feedback.

Key Takeaways

• RAG minimizes hallucinations by generating answers strictly from relevant chunks
• Hybrid search (vectors + BM25 + RRF) improves accuracy by 20–30% over pure vector search
• Semantic chunking preserves context in tables and code, reducing noise
• Hierarchical approach speeds up search across large corpora (>10k docs)
• Forced filters ensure retrieval of facts containing numbers or quoted text

— Editorial Team