Recognizing American Sign Language Manual Signs from RGB-D Videos

TL;DR

This paper introduces a multi-stream 3D CNN framework for real-time recognition of American Sign Language signs from RGB-D videos, leveraging multimodal features and a new dataset, achieving high accuracy.

Contribution

It presents a novel multi-stream 3DCNN approach for ASL recognition and introduces the ASL-100-RGBD dataset with comprehensive annotations.

Findings

Achieved 92.88% accuracy on ASL-100-RGBD dataset.

Outperformed state-of-the-art on Chalearn IsoGD with 76% accuracy.

Demonstrated effectiveness of multimodal fusion with fewer channels.

Abstract

In this paper, we propose a 3D Convolutional Neural Network (3DCNN) based multi-stream framework to recognize American Sign Language (ASL) manual signs (consisting of movements of the hands, as well as non-manual face movements in some cases) in real-time from RGB-D videos, by fusing multimodality features including hand gestures, facial expressions, and body poses from multi-channels (RGB, depth, motion, and skeleton joints). To learn the overall temporal dynamics in a video, a proxy video is generated by selecting a subset of frames for each video which are then used to train the proposed 3DCNN model. We collect a new ASL dataset, ASL-100-RGBD, which contains 42 RGB-D videos captured by a Microsoft Kinect V2 camera, each of 100 ASL manual signs, including RGB channel, depth maps, skeleton joints, face features, and HDface. The dataset is fully annotated for each semantic region (i.e. the time duration of each word that the human signer performs). Our proposed method achieves 92.88 accuracy for recognizing 100 ASL words in our newly collected ASL-100-RGBD dataset. The effectiveness of our framework for recognizing hand gestures from RGB-D videos is further demonstrated on the Chalearn IsoGD dataset and achieves 76 accuracy which is 5.51 higher than the state-of-the-art work in terms of average fusion by using only 5 channels instead of 12 channels in the previous work.