CGRA-DeBERTa Concept Guided Residual Augmentation Transformer for Theologically Islamic Understanding

TL;DR

This paper introduces CGRA DeBERTa, a specialized transformer model that incorporates theological priors and concept-guided mechanisms to significantly improve question-answering accuracy over Islamic texts, with efficient and interpretable results.

Contribution

The paper presents a novel concept-guided residual augmentation transformer tailored for theological QA, integrating Islamic domain knowledge into a customized DeBERTa framework with lightweight adaptations.

Findings

Achieved an EM score of 97.85, surpassing BERT and DeBERTa by over 8 points.

Enhanced semantic discrimination and theological precision in QA tasks.

Maintained computational efficiency with only 8% additional inference overhead.

Abstract

Accurate QA over classical Islamic texts remains challenging due to domain specific semantics, long context dependencies, and concept sensitive reasoning. Therefore, a new CGRA DeBERTa, a concept guided residual domain augmentation transformer framework, is proposed that enhances theological QA over Hadith corpora. The CGRA DeBERTa builds on a customized DeBERTa transformer backbone with lightweight LoRA based adaptations and a residual concept aware gating mechanism. The customized DeBERTa embedding block learns global and positional context, while Concept Guided Residual Blocks incorporate theological priors from a curated Islamic Concept Dictionary of 12 core terms. Moreover, the Concept Gating Mechanism selectively amplifies semantically critical tokens via importance weighted attention, applying differential scaling from 1.04 to 3.00. This design preserves contextual integrity, strengthens domain-specific semantic representations, and enables accurate, efficient span extraction while maintaining computational efficiency. This paper reports the results of training CGRA using a specially constructed dataset of 42591 QA pairs from the text of Sahih alBukhari and Sahih Muslim. While BERT achieved an EM score of 75.87 and DeBERTa one of 89.77, our model scored 97.85 and thus surpassed them by 8.08 on an absolute scale, all while adding approximately 8 inference overhead due to parameter efficient gating. The qualitative evaluation noted better extraction and discrimination and theological precision. This study presents Hadith QA systems that are efficient, interpretable, and accurate and that scale provide educational materials with necessary theological nuance.