Online hand gesture recognition using Continual Graph Transformers

TL;DR

This paper introduces a real-time online hand gesture recognition system using a hybrid architecture of spatial graph convolutions and Transformer-based encoding, enhanced with continual learning for adaptability, achieving state-of-the-art results on benchmark data.

Contribution

A novel online recognition framework combining S-GCN and Transformer-based Graph Encoder with continual learning for improved real-time skeleton-based gesture recognition.

Findings

Achieves state-of-the-art accuracy on SHREC'21 dataset

Significantly reduces false positive rates in online recognition

Demonstrates robustness in dynamic environments

Abstract

Online continuous action recognition has emerged as a critical research area due to its practical implications in real-world applications, such as human-computer interaction, healthcare, and robotics. Among various modalities, skeleton-based approaches have gained significant popularity, demonstrating their effectiveness in capturing 3D temporal data while ensuring robustness to environmental variations. However, most existing works focus on segment-based recognition, making them unsuitable for real-time, continuous recognition scenarios. In this paper, we propose a novel online recognition system designed for real-time skeleton sequence streaming. Our approach leverages a hybrid architecture combining Spatial Graph Convolutional Networks (S-GCN) for spatial feature extraction and a Transformer-based Graph Encoder (TGE) for capturing temporal dependencies across frames. Additionally, we introduce a continual learning mechanism to enhance model adaptability to evolving data distributions, ensuring robust recognition in dynamic environments. We evaluate our method on the SHREC'21 benchmark dataset, demonstrating its superior performance in online hand gesture recognition. Our approach not only achieves state-of-the-art accuracy but also significantly reduces false positive rates, making it a compelling solution for real-time applications. The proposed system can be seamlessly integrated into various domains, including human-robot collaboration and assistive technologies, where natural and intuitive interaction is crucial.