IndicRAGSuite: Large-Scale Datasets and a Benchmark for Indian Language RAG Systems

TL;DR

IndicRAGSuite provides essential large-scale datasets and a multilingual benchmark for developing and evaluating Retrieval-Augmented Generation systems tailored to Indian languages, addressing critical resource gaps.

Contribution

It introduces IndicMSMarco, a multilingual benchmark for 13 Indian languages, and large-scale datasets derived from Indian language Wikipedias and translated MS MARCO data.

Findings

Created IndicMSMarco benchmark with 1000 queries in 13 languages.

Built large-scale datasets from 19 Indian language Wikipedias.

Enriched training data with translated MS MARCO datasets.

Abstract

Retrieval-Augmented Generation (RAG) systems enable language models to access relevant information and generate accurate, well-grounded, and contextually informed responses. However, for Indian languages, the development of high-quality RAG systems is hindered by the lack of two critical resources: (1) evaluation benchmarks for retrieval and generation tasks, and (2) large-scale training datasets for multilingual retrieval. Most existing benchmarks and datasets are centered around English or high-resource languages, making it difficult to extend RAG capabilities to the diverse linguistic landscape of India. To address the lack of evaluation benchmarks, we create IndicMSMarco, a multilingual benchmark for evaluating retrieval quality and response generation in 13 Indian languages, created via manual translation of 1000 diverse queries from MS MARCO-dev set. To address the need for training data, we build a large-scale dataset of (question, answer, relevant passage) tuples derived from the Wikipedias of 19 Indian languages using state-of-the-art LLMs. Additionally, we include translated versions of the original MS MARCO dataset to further enrich the training data and ensure alignment with real-world information-seeking tasks. Resources are available here: https://huggingface.co/collections/ai4bharat/indicragsuite-683e7273cb2337208c8c0fcb