Deep self-supervised learning with visualisation for automatic gesture recognition

TL;DR

This paper explores deep learning techniques, including supervised, self-supervised, and visualization methods, for automatic gesture recognition using 3D skeleton data, demonstrating improved accuracy and interpretability.

Contribution

It introduces a combined approach of supervised, self-supervised, and visualization techniques for gesture recognition with 3D skeleton data, highlighting the effectiveness of self-supervised learning.

Findings

Supervised learning achieves high gesture recognition accuracy.

Self-supervised learning improves accuracy in simulated settings.

Grad-CAM visualizations focus on relevant skeleton joints.

Abstract

Gesture is an important mean of non-verbal communication, with visual modality allows human to convey information during interaction, facilitating peoples and human-machine interactions. However, it is considered difficult to automatically recognise gestures. In this work, we explore three different means to recognise hand signs using deep learning: supervised learning based methods, self-supervised methods and visualisation based techniques applied to 3D moving skeleton data. Self-supervised learning used to train fully connected, CNN and LSTM method. Then, reconstruction method is applied to unlabelled data in simulated settings using CNN as a backbone where we use the learnt features to perform the prediction in the remaining labelled data. Lastly, Grad-CAM is applied to discover the focus of the models. Our experiments results show that supervised learning method is capable to recognise gesture accurately, with self-supervised learning increasing the accuracy in simulated settings. Finally, Grad-CAM visualisation shows that indeed the models focus on relevant skeleton joints on the associated gesture.