An automated framework for qur’anic education of the hearing-impaired using body pose classification and Arabic sign language integration

TL;DR

This paper proposes an automated system to teach the Quran to hearing-impaired students using Arabic Sign Language and body pose classification.

Contribution

The novel contribution is an accessible Quranic education framework integrating Arabic Sign Language and body pose classification for hearing-impaired learners.

Findings

A dataset of 2,054 labeled images was created with input from local institutions working with deaf users.

The ResNet50-based model achieved near-perfect performance in classifying Arabic Sign Language postures.

Keypoint-based models using MLP, SVM, and RF also showed high performance but were limited by dataset and evaluation constraints.

Abstract

In this paper, an accessible pipeline of automated teaching of the Quran to deaf and hard-of-hearing students is proposed based on the identification of Arabic Sign Language (ArSL) postures that match the words of S The piping includes an instructional method that is accessible to both deaf and hard-of-hearing students, necessitating the use of Arabic Sign Language postures to correspond to the words. A designed list of 2,054 labeled images was obtained with local institutions working with deaf users as guidance. In order to be linguistic and unambiguously semantically designated with Qur’anic terms, DIN 31,635 transliterations is used as a canonical internal representation of all class annotations, and Arabic forms are used to present them. Two complementary approaches are evaluated: (i) a pose-keypoint classification approach using MediaPipe features, trained with multilayer perceptron (MLP), support vector machine (SVM), and random forest (RF) classifiers; and (ii) an image-based model utilizing a ResNet50 backbone. Performance evaluation is conducted using an internal train/validation/test split and assessed based on accuracy, precision, recall, F1-score, confusion matrices, and ROC-AUC metrics. Within this evaluation framework, the ResNet50-based model achieves near-perfect performance, while the keypoint-based models also demonstrate high performance. Such results are limited to the dataset and the evaluation protocol that were adopted and cannot be generalized as performance guarantees. To this end, we explicitly note the following constraints: a single sutra, a small dataset size, the use of fixed frames instead of continuous signing (and co-articulation not modeled), the possibility of bias of the signers and regional variants, and low input of the lower-body landmarks due to framing. Future directions will include more generalization by signer-independent evaluation, extension to a wider range of sūrahs and ArSL variants, continuous sign-language recognition, and on-device real-time implementation of an inclusive Qur’anic education.

Figures