Advanced Arabic Alphabet Sign Language Recognition Using Transfer Learning and Transformer Models

TL;DR

This paper introduces a highly accurate Arabic Sign Language recognition system utilizing transfer learning and transformer models, significantly outperforming previous methods and enhancing communication accessibility for Arabic-speaking deaf communities.

Contribution

It demonstrates the effectiveness of combining CNN and transformer models with transfer learning for Arabic sign language recognition, achieving state-of-the-art accuracy.

Findings

Achieved up to 99.6% accuracy on ArSL2018 dataset.

Achieved up to 99.43% accuracy on AASL dataset.

Outperformed previous state-of-the-art approaches.

Abstract

This paper presents an Arabic Alphabet Sign Language recognition approach, using deep learning methods in conjunction with transfer learning and transformer-based models. We study the performance of the different variants on two publicly available datasets, namely ArSL2018 and AASL. This task will make full use of state-of-the-art CNN architectures like ResNet50, MobileNetV2, and EfficientNetB7, and the latest transformer models such as Google ViT and Microsoft Swin Transformer. These pre-trained models have been fine-tuned on the above datasets in an attempt to capture some unique features of Arabic sign language motions. Experimental results present evidence that the suggested methodology can receive a high recognition accuracy, by up to 99.6\% and 99.43\% on ArSL2018 and AASL, respectively. That is far beyond the previously reported state-of-the-art approaches. This performance opens up even more avenues for communication that may be more accessible to Arabic-speaking deaf and hard-of-hearing, and thus encourages an inclusive society.