Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition

TL;DR

This paper introduces a neural architecture search-based approach for multi-modal gesture recognition, leveraging enhanced temporal features and optimized multi-rate, multi-modal networks to achieve state-of-the-art results on benchmark datasets.

Contribution

It presents the first NAS-based method for RGB-D gesture recognition, integrating 3D-CDC for temporal enhancement and optimized backbones for multi-rate, multi-modal learning.

Findings

Achieves state-of-the-art performance on IsoGD, NvGesture, and EgoGesture datasets.

Demonstrates effective multi-modal and multi-rate integration for gesture recognition.

Provides a new perspective on RGB and depth modality relationships.

Abstract

Gesture recognition has attracted considerable attention owing to its great potential in applications. Although the great progress has been made recently in multi-modal learning methods, existing methods still lack effective integration to fully explore synergies among spatio-temporal modalities effectively for gesture recognition. The problems are partially due to the fact that the existing manually designed network architectures have low efficiency in the joint learning of multi-modalities. In this paper, we propose the first neural architecture search (NAS)-based method for RGB-D gesture recognition. The proposed method includes two key components: 1) enhanced temporal representation via the proposed 3D Central Difference Convolution (3D-CDC) family, which is able to capture rich temporal context via aggregating temporal difference information; and 2) optimized backbones for multi-sampling-rate branches and lateral connections among varied modalities. The resultant multi-modal multi-rate network provides a new perspective to understand the relationship between RGB and depth modalities and their temporal dynamics. Comprehensive experiments are performed on three benchmark datasets (IsoGD, NvGesture, and EgoGesture), demonstrating the state-of-the-art performance in both single- and multi-modality settings.The code is available at https://github.com/ZitongYu/3DCDC-NAS