Exploiting Recurrent Neural Networks and Leap Motion Controller for Sign Language and Semaphoric Gesture Recognition

TL;DR

This paper presents a method using Recurrent Neural Networks and Leap Motion Controller data to recognize sign language and gestures, achieving high accuracy and outperforming existing methods on benchmark datasets.

Contribution

It introduces a novel approach combining RNNs with finger joint angles from Leap Motion data for improved gesture recognition.

Findings

High recognition accuracy on ASL gestures

Superior performance compared to state-of-the-art methods

Effective use of finger joint angles as features

Abstract

In human interactions, hands are a powerful way of expressing information that, in some cases, can be used as a valid substitute for voice, as it happens in Sign Language. Hand gesture recognition has always been an interesting topic in the areas of computer vision and multimedia. These gestures can be represented as sets of feature vectors that change over time. Recurrent Neural Networks (RNNs) are suited to analyse this type of sets thanks to their ability to model the long term contextual information of temporal sequences. In this paper, a RNN is trained by using as features the angles formed by the finger bones of human hands. The selected features, acquired by a Leap Motion Controller (LMC) sensor, have been chosen because the majority of human gestures produce joint movements that generate truly characteristic corners. A challenging subset composed by a large number of gestures defined by the American Sign Language (ASL) is used to test the proposed solution and the effectiveness of the selected angles. Moreover, the proposed method has been compared to other state of the art works on the SHREC dataset, thus demonstrating its superiority in hand gesture recognition accuracy.