From RAG to Agentic RAG for Faithful Islamic Question Answering

TL;DR

This paper introduces a new benchmark and an agentic RAG framework for faithful Islamic question answering, emphasizing hallucination detection, abstention, and grounded evidence retrieval to improve accuracy and robustness.

Contribution

It presents ISLAMICFAITHQA, a bilingual benchmark for Islamic QA, and develops an agentic RAG approach with structured tool calls for iterative evidence seeking and answer revision.

Findings

Retrieval improves answer correctness.

Agentic RAG outperforms standard RAG in accuracy.

Framework achieves state-of-the-art performance with small models.

Abstract

LLMs are increasingly used for Islamic question answering, where ungrounded responses may carry serious religious consequences. Yet standard MCQ/MRC-style evaluations do not capture key real-world failure modes, notably free-form hallucinations and whether models appropriately abstain when evidence is lacking. To shed a light on this aspect we introduce ISLAMICFAITHQA, a 3,810-item bilingual (Arabic/English) generative benchmark with atomic single-gold answers, which enables direct measurement of hallucination and abstention. We additionally developed an end-to-end grounded Islamic modelling suite consisting of (i) 25K Arabic text-grounded SFT reasoning pairs, (ii) 5K bilingual preference samples for reward-guided alignment, and (iii) a verse-level Qur'an retrieval corpus of $\sim$6k atomic verses (ayat). Building on these resources, we develop an agentic Quran-grounding framework (agentic RAG) that uses structured tool calls for iterative evidence seeking and answer revision. Experiments across Arabic-centric and multilingual LLMs show that retrieval improves correctness and that agentic RAG yields the largest gains beyond standard RAG, achieving state-of-the-art performance and stronger Arabic-English robustness even with a small model (i.e., Qwen3 4B). We will make the experimental resources and datasets publicly available for the community.