MV-GMN: State Space Model for Multi-View Action Recognition

TL;DR

The paper introduces MV-GMN, a state-space model that efficiently combines multi-view, multi-modal, and temporal data for action recognition, outperforming Transformer-based models with lower computational complexity.

Contribution

It proposes a novel Multi-View Graph Mamba network with Bidirectional State Space Blocks and GCN modules, reducing computational costs while improving accuracy in multi-view action recognition.

Findings

Achieves 97.3% accuracy on NTU RGB+D 120 cross-subject

Outperforms Transformer-based baselines in accuracy

Requires only linear inference complexity

Abstract

Recent advancements in multi-view action recognition have largely relied on Transformer-based models. While effective and adaptable, these models often require substantial computational resources, especially in scenarios with multiple views and multiple temporal sequences. Addressing this limitation, this paper introduces the MV-GMN model, a state-space model specifically designed to efficiently aggregate multi-modal data (RGB and skeleton), multi-view perspectives, and multi-temporal information for action recognition with reduced computational complexity. The MV-GMN model employs an innovative Multi-View Graph Mamba network comprising a series of MV-GMN blocks. Each block includes a proposed Bidirectional State Space Block and a GCN module. The Bidirectional State Space Block introduces four scanning strategies, including view-prioritized and time-prioritized approaches. The GCN module leverages rule-based and KNN-based methods to construct the graph network, effectively integrating features from different viewpoints and temporal instances. Demonstrating its efficacy, MV-GMN outperforms the state-of-the-arts on several datasets, achieving notable accuracies of 97.3\% and 96.7\% on the NTU RGB+D 120 dataset in cross-subject and cross-view scenarios, respectively. MV-GMN also surpasses Transformer-based baselines while requiring only linear inference complexity, underscoring the model's ability to reduce computational load and enhance the scalability and applicability of multi-view action recognition technologies.