Benchmarking the Legal Reasoning of LLMs in Arabic Islamic Inheritance Cases

TL;DR

This paper evaluates the reasoning capabilities of state-of-the-art LLMs in interpreting and applying Islamic inheritance laws in Arabic, demonstrating that a majority voting ensemble achieves high accuracy in complex legal scenarios.

Contribution

It introduces a benchmark for assessing LLMs on Islamic inheritance cases and shows that an ensemble of models significantly improves reasoning accuracy.

Findings

Majority voting ensemble achieves up to 92.7% accuracy.

Ensemble outperforms individual models across all difficulty levels.

Achieved third place in the Qias 2025 challenge.

Abstract

Islamic inheritance domain holds significant importance for Muslims to ensure fair distribution of shares between heirs. Manual calculation of shares under numerous scenarios is complex, time-consuming, and error-prone. Recent advancements in Large Language Models (LLMs) have sparked interest in their potential to assist with complex legal reasoning tasks. This study evaluates the reasoning capabilities of state-of-the-art LLMs to interpret and apply Islamic inheritance laws. We utilized the dataset proposed in the ArabicNLP QIAS 2025 challenge, which includes inheritance case scenarios given in Arabic and derived from Islamic legal sources. Various base and fine-tuned models, are assessed on their ability to accurately identify heirs, compute shares, and justify their reasoning in alignment with Islamic legal principles. Our analysis reveals that the proposed majority voting solution, leveraging three base models (Gemini Flash 2.5, Gemini Pro 2.5, and GPT o3), outperforms all other models that we utilized across every difficulty level. It achieves up to 92.7% accuracy and secures the third place overall in Task 1 of the Qias 2025 challenge.