DRAGOn: Designing RAG On Periodically Updated Corpus

TL;DR

DRAGOn presents a comprehensive framework for designing and evaluating RAG benchmarks on periodically updated corpora, including datasets, question generation, evaluation metrics, and a public leaderboard to foster community progress.

Contribution

It introduces a novel methodology for creating and maintaining RAG benchmarks with regular updates, ensuring fair comparison and reducing data leakage.

Findings

Effective automatic question generation from knowledge graphs

A diverse set of evaluation metrics for RAG systems

Successful deployment on Russian news datasets

Abstract

This paper introduces DRAGOn, method to design a RAG benchmark on a regularly updated corpus. It features recent reference datasets, a question generation framework, an automatic evaluation pipeline, and a public leaderboard. Specified reference datasets allow for uniform comparison of RAG systems, while newly generated dataset versions mitigate data leakage and ensure that all models are evaluated on unseen, comparable data. The pipeline for automatic question generation extracts the Knowledge Graph from the text corpus and produces multiple question-answer pairs utilizing modern LLM capabilities. A set of diverse LLM-as-Judge metrics is provided for a comprehensive model evaluation. We used Russian news outlets to form the datasets and demonstrate our methodology. We launch a public leaderboard to track the development of RAG systems and encourage community participation.