Revolutionizing Communication with Deep Learning and XAI for Enhanced Arabic Sign Language Recognition

TL;DR

This paper presents an advanced Arabic Sign Language recognition system leveraging deep learning models enhanced with explainable AI techniques, achieving high accuracy and transparency for practical applications.

Contribution

It introduces a novel integrated approach combining deep learning and XAI for Arabic Sign Language recognition, with innovative data augmentation and validation methods.

Findings

EfficientNet-B2 achieved 99.48% accuracy on ArSL datasets.

The system enhances interpretability using Grad-CAM.

The approach sets new benchmarks in recognition accuracy.

Abstract

This study introduces an integrated approach to recognizing Arabic Sign Language (ArSL) using state-of-the-art deep learning models such as MobileNetV3, ResNet50, and EfficientNet-B2. These models are further enhanced by explainable AI (XAI) techniques to boost interpretability. The ArSL2018 and RGB Arabic Alphabets Sign Language (AASL) datasets are employed, with EfficientNet-B2 achieving peak accuracies of 99.48\% and 98.99\%, respectively. Key innovations include sophisticated data augmentation methods to mitigate class imbalance, implementation of stratified 5-fold cross-validation for better generalization, and the use of Grad-CAM for clear model decision transparency. The proposed system not only sets new benchmarks in recognition accuracy but also emphasizes interpretability, making it suitable for applications in healthcare, education, and inclusive communication technologies.