Accurate online action and gesture recognition system using detectors and Deep SPD Siamese Networks

TL;DR

This paper introduces an online skeleton-based action recognition system combining SPD matrix representations and Siamese networks, enabling real-time detection and classification of continuous motions with high accuracy.

Contribution

It presents a novel online recognition framework with a detector and classifier using SPD matrices and Siamese networks, suitable for unsegmented streaming data.

Findings

Outperforms state-of-the-art in gesture and action recognition benchmarks

Effective continuous detection of kinetic states in streaming data

High accuracy in online recognition tasks

Abstract

Online continuous motion recognition is a hot topic of research since it is more practical in real life application cases. Recently, Skeleton-based approaches have become increasingly popular, demonstrating the power of using such 3D temporal data. However, most of these works have focused on segment-based recognition and are not suitable for the online scenarios. In this paper, we propose an online recognition system for skeleton sequence streaming composed from two main components: a detector and a classifier, which use a Semi-Positive Definite (SPD) matrix representation and a Siamese network. The powerful statistical representations for the skeletal data given by the SPD matrices and the learning of their semantic similarity by the Siamese network enable the detector to predict time intervals of the motions throughout an unsegmented sequence. In addition, they ensure the classifier capability to recognize the motion in each predicted interval. The proposed detector is flexible and able to identify the kinetic state continuously. We conduct extensive experiments on both hand gesture and body action recognition benchmarks to prove the accuracy of our online recognition system which in most cases outperforms state-of-the-art performances.