Action Recognition with Multi-stream Motion Modeling and Mutual Information Maximization

TL;DR

This paper introduces a multi-stream motion modeling approach with mutual information maximization for action recognition, incorporating acceleration features and channel attention to improve accuracy on benchmark datasets.

Contribution

It proposes a novel Stream-GCN network with multi-stream inputs, channel attention, and mutual information loss for enhanced action recognition.

Findings

Achieved state-of-the-art results on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets.

Demonstrated the effectiveness of acceleration features in motion modeling.

Showed that mutual information maximization improves task-relevant feature extraction.

Abstract

Action recognition has long been a fundamental and intriguing problem in artificial intelligence. The task is challenging due to the high dimensionality nature of an action, as well as the subtle motion details to be considered. Current state-of-the-art approaches typically learn from articulated motion sequences in the straightforward 3D Euclidean space. However, the vanilla Euclidean space is not efficient for modeling important motion characteristics such as the joint-wise angular acceleration, which reveals the driving force behind the motion. Moreover, current methods typically attend to each channel equally and lack theoretical constrains on extracting task-relevant features from the input. In this paper, we seek to tackle these challenges from three aspects: (1) We propose to incorporate an acceleration representation, explicitly modeling the higher-order variations in motion. (2) We introduce a novel Stream-GCN network equipped with multi-stream components and channel attention, where different representations (i.e., streams) supplement each other towards a more precise action recognition while attention capitalizes on those important channels. (3) We explore feature-level supervision for maximizing the extraction of task-relevant information and formulate this into a mutual information loss. Empirically, our approach sets the new state-of-the-art performance on three benchmark datasets, NTU RGB+D, NTU RGB+D 120, and NW-UCLA. Our code is anonymously released at https://github.com/ActionR-Group/Stream-GCN, hoping to inspire the community.