Arabic Sign Language Recognition using Multimodal Approach

TL;DR

This paper explores a multimodal system combining Leap Motion and RGB camera data for recognizing Arabic Sign Language, achieving 78% accuracy on a custom dataset, and demonstrating the potential of multimodal fusion in sign language recognition.

Contribution

It introduces a novel multimodal recognition system using Leap Motion and RGB data with deep learning, improving recognition capabilities for Arabic Sign Language.

Findings

Achieved 78% accuracy on a custom dataset of 18 words.

Demonstrated the feasibility of multimodal fusion for sign language recognition.

Provided insights into combining sensor data for improved gesture analysis.

Abstract

Arabic Sign Language (ArSL) is an essential communication method for individuals in the Deaf and Hard-of-Hearing community. However, existing recognition systems face significant challenges due to their reliance on single sensor approaches like Leap Motion or RGB cameras. These systems struggle with limitations such as inadequate tracking of complex hand orientations and imprecise recognition of 3D hand movements. This research paper aims to investigate the potential of a multimodal approach that combines Leap Motion and RGB camera data to explore the feasibility of recognition of ArSL. The system architecture includes two parallel subnetworks: a custom dense neural network for Leap Motion data, incorporating dropout and L2 regularization, and an image subnetwork based on a fine-tuned VGG16 model enhanced with data augmentation techniques. Feature representations from both modalities are concatenated in a fusion model and passed through fully connected layers, with final classification performed via SoftMax activation to analyze spatial and temporal features of hand gestures. The system was evaluated on a custom dataset comprising 18 ArSL words, of which 13 were correctly recognized, yielding an overall accuracy of 78%. These results offer preliminary insights into the viability of multimodal fusion for sign language recognition and highlight areas for further optimization and dataset expansion.