BnMMLU: Measuring Massive Multitask Language Understanding in Bengali

TL;DR

This paper introduces BnMMLU, a large-scale benchmark for evaluating Bengali language understanding across diverse domains and model types, highlighting current gaps and challenges in multilingual NLP.

Contribution

It provides the most extensive Bengali evaluation dataset, benchmarks 24 models, and analyzes performance gaps, fostering progress in Bengali and multilingual NLP.

Findings

Models show persistent reasoning gaps.

Sublinear performance gains with larger models.

Benchmark reveals significant room for improvement.

Abstract

Large-scale multitask benchmarks have driven rapid progress in language modeling, yet most emphasize high-resource languages such as English, leaving Bengali underrepresented. We present BnMMLU, a comprehensive benchmark for measuring massive multitask language understanding in Bengali. BnMMLU spans 41 domains across STEM, humanities, social sciences, and general knowledge, and contains 134,375 multiple-choice question-option pairs--the most extensive Bengali evaluation suite to date. The dataset preserves mathematical content via MathML, and includes BnMMLU-HARD, a compact subset constructed from questions most frequently missed by top systems to stress difficult cases. We benchmark 24 model variants across 11 LLM families, spanning open-weights general/multilingual, Bengali-centric open-weights, and proprietary models, covering multiple parameter scales and instruction-tuned settings. We evaluate models under standardized protocols covering two prompting styles (Direct vs. Chain-of-Thought) and two context regimes (0-shot vs. 5-shot), reporting accuracy consistently across families. Our analysis highlights persistent gaps in reasoning and application skills and indicates sublinear returns to scale across model sizes. We release the dataset and evaluation templates to support rigorous, reproducible assessment of Bengali language understanding and to catalyze progress in multilingual NLP.