Reasoning about Body-Parts Relations for Sign Language Recognition

TL;DR

This paper presents a sign language recognition method that models hand movements in the context of other body parts using 3D data, improving accuracy over trajectory-only approaches.

Contribution

It introduces a pipeline that combines body-part context and hand postures for enhanced sign language recognition from 3D Kinect data.

Findings

Considering body parts improves recognition accuracy.

Combining hand postures with gestures enhances prediction.

Method outperforms trajectory-only approaches.

Abstract

Over the years, hand gesture recognition has been mostly addressed considering hand trajectories in isolation. However, in most sign languages, hand gestures are defined on a particular context (body region). We propose a pipeline to perform sign language recognition which models hand movements in the context of other parts of the body captured in the 3D space using the MS Kinect sensor. In addition, we perform sign recognition based on the different hand postures that occur during a sign. Our experiments show that considering different body parts brings improved performance when compared to other methods which only consider global hand trajectories. Finally, we demonstrate that the combination of hand postures features with hand gestures features helps to improve the prediction of a given sign.