Parts-Mamba: Augmenting Joint Context with Part-Level Scanning for Occluded Human Skeleton

TL;DR

Parts-Mamba is a novel hybrid GCN model that improves occluded human skeleton action recognition by capturing part-specific and distant joint context, significantly enhancing accuracy in real-world scenarios with missing data.

Contribution

The paper introduces Parts-Mamba, a hybrid GCN-Mamba model that effectively captures part-specific and distant joint context to handle occlusions in skeleton-based action recognition.

Findings

Achieves up to 12.9% accuracy improvement on NTU datasets.

Effectively handles occlusions and missing data in skeletons.

Outperforms existing GCN models in non-ideal conditions.

Abstract

Skeleton action recognition involves recognizing human action from human skeletons. The use of graph convolutional networks (GCNs) has driven major advances in this recognition task. In real-world scenarios, the captured skeletons are not always perfect or complete because of occlusions of parts of the human body or poor communication quality, leading to missing parts in skeletons or videos with missing frames. In the presence of such non-idealities, existing GCN models perform poorly due to missing local context. To address this limitation, we propose Parts-Mamba, a hybrid GCN-Mamba model designed to enhance the ability to capture and maintain contextual information from distant joints. The proposed Parts-Mamba model effectively captures part-specific information through its parts-specific scanning feature and preserves non-neighboring joint context via a parts-body fusion module. Our proposed model is evaluated on the NTU RGB+D 60 and NTU RGB+D 120 datasets under different occlusion settings, achieving up to 12.9% improvement in accuracy.