Position and Rotation Invariant Sign Language Recognition from 3D Kinect Data with Recurrent Neural Networks

TL;DR

This paper presents a method for recognizing Indian sign language gestures using 3D Kinect data and recurrent neural networks, achieving over 84% accuracy while accounting for position and rotation invariance.

Contribution

It introduces a sign language recognition approach using 3D joint data and RNNs with geometric alignment correction, enhancing recognition robustness.

Findings

Achieved 84.81% recognition accuracy.

Implemented geometric transformations for alignment correction.

Demonstrated effectiveness on 30 Indian sign gestures.

Abstract

Sign language is a gesture-based symbolic communication medium among speech and hearing impaired people. It also serves as a communication bridge between non-impaired and impaired populations. Unfortunately, in most situations, a non-impaired person is not well conversant in such symbolic languages restricting the natural information flow between these two categories. Therefore, an automated translation mechanism that seamlessly translates sign language into natural language can be highly advantageous. In this paper, we attempt to perform recognition of 30 basic Indian sign gestures. Gestures are represented as temporal sequences of 3D maps (RGB + depth), each consisting of 3D coordinates of 20 body joints captured by the Kinect sensor. A recurrent neural network (RNN) is employed as the classifier. To improve the classifier's performance, we use geometric transformation for the alignment correction of depth frames. In our experiments, the model achieves 84.81% accuracy.