IslamicMMLU: A Benchmark for Evaluating LLMs on Islamic Knowledge

TL;DR

IslamicMMLU is a comprehensive benchmark with over 10,000 questions across Islamic disciplines, used to evaluate and compare the performance of 26 large language models on Islamic knowledge tasks.

Contribution

The paper introduces IslamicMMLU, the first extensive benchmark for assessing LLMs on Islamic knowledge, including a public leaderboard and evaluation of multiple models.

Findings

Model accuracy ranged from 39.8% to 93.8%.

The Quran track showed the widest performance variation.

Arabic-specific models underperformed compared to frontier models.

Abstract

Large language models are increasingly consulted for Islamic knowledge, yet no comprehensive benchmark evaluates their performance across core Islamic disciplines. We introduce IslamicMMLU, a benchmark of 10,013 multiple-choice questions spanning three tracks: Quran (2,013 questions), Hadith (4,000 questions), and Fiqh (jurisprudence, 4,000 questions). Each track is formed of multiple types of questions to examine LLMs capabilities handling different aspects of Islamic knowledge. The benchmark is used to create the IslamicMMLU public leaderboard for evaluating LLMs, and we initially evaluate 26 LLMs, where their averaged accuracy across the three tracks varied between 39.8% to 93.8% (by Gemini 3 Flash). The Quran track shows the widest span (99.3% to 32.4%), while the Fiqh track includes a novel madhab (Islamic school of jurisprudence) bias detection task revealing variable school-of-thought preferences across models. Arabic-specific models show mixed results, but they all underperform compared to frontier models. The evaluation code and leaderboard are made publicly available.