DRAGOn: test of RAG systems on dynamic data

# DRAGOn: Dynamic Methodology for Testing RAG Systems on Evolving Corpora

The teams from SberAI, MWS AI, and university researchers have introduced DRAGOn—an open dynamic methodology for testing Russian-language RAG (Retrieval-Augmented Generation) systems. It accounts for periodically updated data corpora, such as news feeds, and builds a knowledge map from them. The methodology has been accepted to the EACL 2026 conference in Morocco.

RAG systems integrate LLMs with external knowledge bases to generate responses based on up-to-date data. This minimizes hallucinations and boosts the reliability of AI agents in analysis and search tasks.

Problems with Existing Approaches and the DRAGOn Solution

Traditional benchmarks rely on static datasets that quickly become outdated. Corporate knowledge is dynamic, but tests aren't adapted to handle it. DRAGOn solves this:

• Automatically builds a knowledge graph from updating sources.
• Generates multi-level logical tasks involving fact matching across different documents.
• Evaluates responses using a separate LLM, checking accuracy, completeness, and semantic similarity.

The system is designed for Russian-language data, focusing on language-specific challenges in retrieval and generation.

Test Generation and Quality Evaluation

DRAGOn creates queries not as simple Q&A pairs, but as reasoning chains: from basic facts to complex inferences. Example workflow:

• Parsing the updated corpus (news, reports).
• Extracting entities and relations to build the knowledge graph.
• Synthesizing questions with logical dependencies.
• Running the RAG system and verifying results with a judge model.

Evaluation metrics include factual accuracy, completeness score, and logical coherence. Preliminary tests show that combining multiple LLMs with hybrid search (dense + sparse) delivers the best results, though complex connections remain a challenge.

Public Leaderboard and Practical Applications

An open leaderboard for Russian-language RAG systems has been launched. Top-performing models pair advanced retrieval (e.g., ColBERT) with fine-tuned LLMs. The results underscore the value of multi-model ensembles.

For developers, the methodology makes it easy to set up private test environments:

• Integration with internal data (docs, CRM).
• Automatic benchmark generation.
• Pipeline comparisons: embedding models, reranking, prompting strategies.

This helps mitigate risks during production deployment of AI agents.

Versatility and Future Prospects

DRAGOn extends beyond news to scientific publications, legal databases, and enterprise knowledge bases. A co-author from MWS AI highlights the AI industry's shift toward solution quality over sheer model scale.

Participants: SberAI, MWS AI, MBZUAI, ITMO, MISIS, HSE University, IITU, Yandex ShAD.

Key Takeaways:

• DRAGOn is the first open methodology for dynamic testing of Russian-language RAG systems.
• It handles updating corpora and generates logical tasks.
• The public leaderboard exposes strengths and weaknesses in current RAG setups.
• Ideal for enterprise use: private benchmarks on internal data.
• Versatile across domains—from news to legal fields.

— Editorial Team