MVTN: A Multiscale Video Transformer Network for Hand Gesture Recognition

TL;DR

This paper presents MVTN, a multiscale video transformer that effectively captures diverse hand gesture features using a hierarchical attention mechanism, multimodal data, and achieves state-of-the-art results with reduced complexity.

Contribution

The paper introduces a novel multiscale hierarchical transformer architecture for hand gesture recognition that integrates multimodal data and demonstrates superior performance.

Findings

Achieves state-of-the-art accuracy on NVGesture and Briareo datasets.

Reduces computational complexity compared to existing methods.

Effectively captures multiscale features for dynamic hand gestures.

Abstract

In this paper, we introduce a novel Multiscale Video Transformer Network (MVTN) for dynamic hand gesture recognition, since multiscale features can extract features with variable size, pose, and shape of hand which is a challenge in hand gesture recognition. The proposed model incorporates a multiscale feature hierarchy to capture diverse levels of detail and context within hand gestures which enhances the model's ability. This multiscale hierarchy is obtained by extracting different dimensions of attention in different transformer stages with initial stages to model high-resolution features and later stages to model low-resolution features. Our approach also leverages multimodal data, utilizing depth maps, infrared data, and surface normals along with RGB images from NVGesture and Briareo datasets. Experiments show that the proposed MVTN achieves state-of-the-art results with less computational complexity and parameters. The source code is available at https://github.com/mallikagarg/MVTN.