IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languages

TL;DR

IndicParam is a comprehensive benchmark evaluating 20 large language models on over 13,000 questions across 11 low-resource Indic languages, revealing significant performance gaps and diverse question handling capabilities.

Contribution

The paper introduces IndicParam, a new benchmark dataset for low-resource Indic languages, including diverse question formats and linguistic annotations, to evaluate LLMs' multilingual and cross-lingual abilities.

Findings

Top model achieves 58% accuracy on average

LLMs struggle with low-resource Indic languages

Benchmark reveals limitations in cross-lingual transfer

Abstract

While large language models excel on high-resource multilingual tasks, low- and extremely low-resource Indic languages remain severely under-evaluated. We present IndicParam, a human-curated benchmark of over 13,000 multiple-choice questions covering 11 such languages (Nepali, Gujarati, Marathi, Odia as low-resource; Dogri, Maithili, Rajasthani, Sanskrit, Bodo, Santali, Konkani as extremely low-resource) plus Sanskrit-English code-mixed set. We evaluated 20 LLMs, both proprietary and open-weights, which reveals that even the top-performing \texttt{Gemini-2.5} reaches 58\% average accuracy, followed by \texttt{GPT-5} (45) and \texttt{DeepSeek-3.2} (43.1). We additionally label each question as knowledge-oriented or purely linguistic to discriminate factual recall from grammatical proficiency. Further, we assess the ability of LLMs to handle diverse question formats-such as list-based matching, assertion-reason pairs, and sequence ordering-alongside conventional multiple-choice questions. \benchmark\ provides insights into limitations of cross-lingual transfer and establishes a challenging benchmark for Indic languages. The dataset is available at https://huggingface.co/datasets/bharatgenai/IndicParam. Scripts to run benchmark are present at https://github.com/ayushbits/IndicParam.