Agentic RAG at hackathon: chunking and search

Agentic RAG: Building a Legal Document Search System for DIFC Courts

The 'Sparks of Intelligence' team developed two RAG systems for the EORA AI's Agentic Legal RAG Challenge 2026. The goal: deliver precise answers to questions based on 300+ court documents from Dubai International Financial Centre (DIFC). With a prize pool of $32,000 and over 300 participants, the challenge featured two stages: preliminary (30 documents, 100 questions) and final (300 documents, 900 questions). Questions were categorized by type—boolean, name, date, number, free text—and scored on accuracy, speed, and token usage.

Challenges of Vector Search in RAG

Vector search excels at finding semantically similar content ('contract termination' ≈ 'contract cancellation'), but fails on specific facts like 'the city where the Jason vs. Kruger case was heard.' Document scale amplifies this issue—from fast retrieval in small libraries to complex indexing across massive corpora.

The key to success? Chunking: splitting text into meaningful segments. A naive approach (based on patterns like Article 1, Paragraph 2) works with strict hierarchies but demands optimal chunk size. Small chunks yield clear vectors lacking context; large ones produce blurry, ambiguous representations.

Example problem:

• Original text: "Attendees at the hearing included: Participant 1, Participant 2, Participant 3."
• Splitting by list markers → each item loses meaning.

Chunking Strategies and Hybrid Search

Hybrid search combines vector similarity, BM25, and metadata. A reranker (a model trained on question-answer pairs) refines top-k*10 results down to top-k for higher precision.

Chunking approaches:

| Approach | Description | Pros | Cons |

|--------|----------|-------|--------|

| B1. Fixed size + overlap | Chunks of N tokens with overlap | Simplicity | Risk of breaking context |

| B2. Hierarchical | Large chunks → smaller ones; context preserved from larger units | Context retention | Implementation complexity |

| B3. Semantic | ML-driven grouping by meaning | High relevance | High resource demand |

The team used B1 and B2, skipping B3 due to time constraints.

Simple Solution Architecture

Qdrant + LlamaIndex for vector storage, Unstructured for PDF parsing while preserving structure.

Chunking: by pages + overlap. Search: hybrid + regex for patterns + metadata filtering. Reranker: yes.

Pros:

• Fast implementation.
• Predictable performance.
• Grounding derived directly from page-level chunks.

Cons:

• Regex leads to false positives or missed matches.
• Rigid dependency on document structure.
• Initial grounding wasn’t in metadata (later fixed).

Results:

• Preliminary stage: accuracy dropped from 0.9 to 0.81 (after fix), grounding improved from 0.5 to 0.58.
• Final stage: accuracy 0.79, grounding 0.63, average speed.

Agentic RAG: Router and Tools

Chunking: hierarchical. Algorithm:

• LLM analyzes document structure on a sample.
• Identifies recurring patterns.
• Recursively merges small chunks into target size.

Challenges: inconsistent patterns across pages → noisy, low-quality chunks.

Search: agent router directs queries to four tools (metadata, exact match, document comparison, hybrid), filtered by case number or law reference. Architecture: router → tools (with reranker) → agent generator.

Pros:

• Correct tool selection based on context.
• Filtering worked effectively.

Cons:

• Unstable chunking.
• Noisy chunks produced.
• Not fully tested (API errors at deadline).

Preliminary results: accuracy 0.74, grounding 0.6, slow speed (2 LLM calls). Final submission not delivered.

Key Takeaways

• Hybrid search + reranker is essential for accuracy beyond semantic similarity.
• Page-based chunking is simpler and more stable than hierarchical chunking under tight deadlines.
• Agents are valuable for routing, but require a strict tool schema and thorough testing.
• Metadata-based grounding is critical for deterministic responses.
• Regex complements, but doesn’t replace, vector search—use with caution to avoid false triggers.

Overall verdict: the simple solution delivered consistent results (0.79/0.63). The agentic approach showed potential but carried risks in chunking and debugging. The experience confirmed: prioritize predictability over complexity.

— Editorial Team