IndicEval: A Bilingual Indian Educational Evaluation Framework for Large Language Models

TL;DR

IndicEval is a comprehensive, real-world benchmarking platform for evaluating large language models on authentic Indian high-stakes exams in English and Hindi, revealing strengths and gaps in reasoning, domain knowledge, and bilingual performance.

Contribution

This paper introduces IndicEval, a novel evaluation framework grounded in real examination questions, supporting multilingual assessment and advanced prompting strategies for LLMs.

Findings

Chain-of-Thought prompting enhances reasoning accuracy.

Performance disparities exist across models and subjects.

Multilingual accuracy drops are significant, especially in Hindi.

Abstract

The rapid advancement of large language models (LLMs) necessitates evaluation frameworks that reflect real-world academic rigor and multilingual complexity. This paper introduces IndicEval, a scalable benchmarking platform designed to assess LLM performance using authentic high-stakes examination questions from UPSC, JEE, and NEET across STEM and humanities domains in both English and Hindi. Unlike synthetic benchmarks, IndicEval grounds evaluation in real examination standards, enabling realistic measurement of reasoning, domain knowledge, and bilingual adaptability. The framework automates assessment using Zero-Shot, Few-Shot, and Chain-of-Thought (CoT) prompting strategies and supports modular integration of new models and languages. Experiments conducted on Gemini 2.0 Flash, GPT-4, Claude, and LLaMA 3-70B reveal three major findings. First, CoT prompting consistently improves reasoning accuracy, with substantial gains across subjects and languages. Second, significant cross-model performance disparities persist, particularly in high-complexity examinations. Third, multilingual degradation remains a critical challenge, with marked accuracy drops in Hindi compared to English, especially under Zero-Shot conditions. These results highlight persistent gaps in bilingual reasoning and domain transfer. Overall, IndicEval provides a practice-oriented, extensible foundation for rigorous, equitable evaluation of LLMs in multilingual educational settings and offers actionable insights for improving reasoning robustness and language adaptability.