Sacred or Synthetic? Evaluating LLM Reliability and Abstention for Religious Questions

TL;DR

This paper introduces FiqhQA, a benchmark for evaluating LLMs on Islamic legal questions across different schools and languages, focusing on accuracy and abstention to ensure reliable religious guidance.

Contribution

It presents the first benchmark for Islamic jurisprudence questions assessing LLM accuracy and abstention, highlighting model variations and language limitations.

Findings

GPT-4o achieves highest accuracy

Fanar and Gemini excel in abstention behavior

Models perform worse in Arabic language

Abstract

Despite the increasing usage of Large Language Models (LLMs) in answering questions in a variety of domains, their reliability and accuracy remain unexamined for a plethora of domains including the religious domains. In this paper, we introduce a novel benchmark FiqhQA focused on the LLM generated Islamic rulings explicitly categorized by the four major Sunni schools of thought, in both Arabic and English. Unlike prior work, which either overlooks the distinctions between religious school of thought or fails to evaluate abstention behavior, we assess LLMs not only on their accuracy but also on their ability to recognize when not to answer. Our zero-shot and abstention experiments reveal significant variation across LLMs, languages, and legal schools of thought. While GPT-4o outperforms all other models in accuracy, Gemini and Fanar demonstrate superior abstention behavior critical for minimizing confident incorrect answers. Notably, all models exhibit a performance drop in Arabic, highlighting the limitations in religious reasoning for languages other than English. To the best of our knowledge, this is the first study to benchmark the efficacy of LLMs for fine-grained Islamic school of thought specific ruling generation and to evaluate abstention for Islamic jurisprudence queries. Our findings underscore the need for task-specific evaluation and cautious deployment of LLMs in religious applications.