Legal RAG ARLC 2026: 17 iterations to 0.791

Legal RAG at ARLC 2026: From 0.034 to 0.791 in 17 Iterations with Scaling Analysis

On the ARLC 2026 legal challenge, the RAG pipeline started with a grounding score of 0.05 and a total score of 0.034. Deterministic answers (S_det) had already reached 0.857, but the G factor was nullifying results. The issue was traced to the doc_id format: submission.json expected the filename without '.pdf', while the code passed the full name. Fixing a single line boosted G to 0.55 and the overall score to 0.438 — a 13x improvement.

Lesson for RAG developers: validating the submission format must come before retrieval optimization. Even perfect reranking is useless without correct mapping to (doc_id, page_number).

Pipeline Architecture: Page-Level Ingestion and Hybrid Search

The pipeline indexes entire document pages, avoiding chunking. This simplifies page-level grounding but requires context distillation before LLM processing.

PDF Parsing with OCR Fallback

def extract_pages(pdf_path, min_chars=50, ocr_dpi=300):
    pages = []
    doc = pdfplumber.open(pdf_path)
    for i, page in enumerate(doc.pages):
        text = page.extract_text() or ""
        if len(text.strip()) < min_chars:
            text = ocr_page(pdf_path, i + 1, dpi=ocr_dpi)
        pages.append({
            "doc_id": pdf_path.stem,      # without .pdf!
            "page_number": i + 1,
            "text": text
        })
    return pages

The code handles scanned PDFs via OCR when extracted text is minimal. doc_id uses the stem without extension; page_number is 1-based.

Hybrid Retrieval with RRF

Combining BM25 for exact matches and embeddings for semantics using Reciprocal Rank Fusion (k=60):

def hybrid_search(query, bm25_index, embedding_index, pages, top_k=20, rrf_k=60):
    bm25_scores = bm25_index.get_scores(tokenize(query))
    bm25_ranked = sorted(range(len(pages)), key=lambda i: -bm25_scores[i])

    q_emb = embed_model.encode([query[:512]], normalize_embeddings=True)
    sim_scores = (embedding_index @ q_emb.T).flatten()
    emb_ranked = sorted(range(len(pages)), key=lambda i: -sim_scores[i])

    combined = {}
    for rank, idx in enumerate(bm25_ranked[:top_k * 3]):
        combined[idx] = combined.get(idx, 0) + 1.0 / (rrf_k + rank)
    for rank, idx in enumerate(emb_ranked[:top_k * 3]):
        combined[idx] = combined.get(idx, 0) + 1.0 / (rrf_k + rank)

    return sorted(combined, key=lambda i: -combined[i])[:top_k]

Embedding model: all-MiniLM-L6-v2 (22M parameters). RRF requires no scale normalization—it works directly on ranks.

Reranking and Priority Tagging

A cross-encoder (ms-marco-MiniLM-L-6-v2) reranks the top 30 pages. A hard cap of 30 prevents O(N²) TTFT explosion.

def rerank(pages, query, top_k=5):
    HARD_CAP = 30
    priority = [p for p in pages if p.get("_priority")]
    non_priority = pages[:HARD_CAP - len(priority)]

    all_pages = priority + non_priority
    pairs = [(query, p["text"][:512]) for p in all_pages]
    scores = cross_encoder.predict(pairs)

    for i, p in enumerate(all_pages):
        if p.get("_priority"):
            scores[i] = 1000.0

    ranked = sorted(zip(scores, all_pages), key=lambda x: -x[0])
    return [p for _, p in ranked[:top_k]]

Priority pages (title pages, articles) get a score of 1000 — guaranteed inclusion in context.

Document Routing for Comparison Queries

For questions like "who was appointed earlier in CFI 001/2020 vs CFI 002/2021", a title-page index is built:

• Patterns: CFI 010/2024 → doc_id
• Dual query with round-robin interleaving of contexts

Deterministic Fast-Paths for Typed Answers

Types: number (±1%), boolean, name (exact), date (YYYY-MM-DD), names (Jaccard), free_text (LLM-judge).

Examples of regex extraction:

• Law number: re.search(r"(?:DIFC\s+)?Law\s+No\.\s*(\d+)")
• Issue date: parse first pages using Date of Issue[:\s]+(\w+ \d+,?\s+\d{4})
• Amounts: extract monetary claims via case_ref

Fast-paths save 700ms on Haiku calls, returning answer + grounding.

Scoring Formula and TTFT Optimization

Total = (0.7 × S_det + 0.3 × S_asst) × G × T × F

• G (F-beta, β=2.5): page-level grounding — the key multiplier
• F: <1000ms → +5%, >3000ms → -15%

| TTFT (ms) | Multiplier |

|-----------|------------|

| <1000 | 1.05 |

| <2000 | 1.02 |

| <3000 | 1.00 |

| >3000 | 0.85–0.99 |

Scaling Wall: Warmup vs Final

On 30 documents (warmup), score was 0.791. On 300 (final), it dropped by 42%. Reasons:

• Larger corpus dilutes retrieval quality
• TTFT grows linearly with size
• Reranking on 300+ pages harms F

Final optimizations: caching embeddings, batching cross-encoder, early stopping in hybrid search.

What Matters

• Page-level indexing: simplifies grounding but demands context distillation.
• RRF (k=60): optimal fusion of BM25 + embeddings without normalization.
• Priority tagging: ensures critical pages are always in context.
• Fast-paths: regex for 80% of typed queries cuts TTFT significantly.
• TTFT-first approach: speed gives +5% bonus for free at equal quality.

— Editorial Team