FARSIQA: Faithful and Advanced RAG System for Islamic Question Answering

TL;DR

FARSIQA introduces an innovative, iterative RAG system tailored for Persian Islamic question answering, significantly improving accuracy and faithfulness by decomposing complex queries and leveraging a large authoritative knowledge base.

Contribution

The paper presents FAIR-RAG, a novel adaptive, iterative framework for faithful Islamic QA, addressing limitations of existing RAG systems in complex, high-stakes domains.

Findings

Achieves 97.0% Negative Rejection on IslamicPCQA benchmark.

Outperforms baselines with a 40-point improvement in rejection rate.

Attains 74.3% Answer Correctness, setting new standards for Persian Islamic QA.

Abstract

The advent of Large Language Models (LLMs) has revolutionized Natural Language Processing, yet their application in high-stakes, specialized domains like religious question answering is hindered by challenges like hallucination and unfaithfulness to authoritative sources. This issue is particularly critical for the Persian-speaking Muslim community, where accuracy and trustworthiness are paramount. Existing Retrieval-Augmented Generation (RAG) systems, relying on simplistic single-pass pipelines, fall short on complex, multi-hop queries requiring multi-step reasoning and evidence aggregation. To address this gap, we introduce FARSIQA, a novel, end-to-end system for Faithful Advanced Question Answering in the Persian Islamic domain. FARSIQA is built upon our innovative FAIR-RAG architecture: a Faithful, Adaptive, Iterative Refinement framework for RAG. FAIR-RAG employs a dynamic, self-correcting process: it adaptively decomposes complex queries, assesses evidence sufficiency, and enters an iterative loop to generate sub-queries, progressively filling information gaps. Operating on a curated knowledge base of over one million authoritative Islamic documents, FARSIQA demonstrates superior performance. Rigorous evaluation on the challenging IslamicPCQA benchmark shows state-of-the-art performance: the system achieves a remarkable 97.0% in Negative Rejection - a 40-point improvement over baselines - and a high Answer Correctness score of 74.3%. Our work establishes a new standard for Persian Islamic QA and validates that our iterative, adaptive architecture is crucial for building faithful, reliable AI systems in sensitive domains.