HDBN: A Novel Hybrid Dual-branch Network for Robust Skeleton-based Action Recognition

TL;DR

The paper introduces HDBN, a hybrid dual-branch network combining GCNs and Transformers for improved skeleton-based action recognition, demonstrating superior performance on UAV-Human benchmarks.

Contribution

It proposes a novel dual-branch architecture leveraging GCNs and Transformers to model different skeletal modalities simultaneously, enhancing recognition robustness.

Findings

Achieved 47.95% and 75.36% accuracy on UAV-Human benchmarks.

Outperformed most existing methods in the ICME Grand Challenge 2024.

Demonstrated the effectiveness of combining GCNs and Transformers for skeleton data.

Abstract

Skeleton-based action recognition has gained considerable traction thanks to its utilization of succinct and robust skeletal representations. Nonetheless, current methodologies often lean towards utilizing a solitary backbone to model skeleton modality, which can be limited by inherent flaws in the network backbone. To address this and fully leverage the complementary characteristics of various network architectures, we propose a novel Hybrid Dual-Branch Network (HDBN) for robust skeleton-based action recognition, which benefits from the graph convolutional network's proficiency in handling graph-structured data and the powerful modeling capabilities of Transformers for global information. In detail, our proposed HDBN is divided into two trunk branches: MixGCN and MixFormer. The two branches utilize GCNs and Transformers to model both 2D and 3D skeletal modalities respectively. Our proposed HDBN emerged as one of the top solutions in the Multi-Modal Video Reasoning and Analyzing Competition (MMVRAC) of 2024 ICME Grand Challenge, achieving accuracies of 47.95% and 75.36% on two benchmarks of the UAV-Human dataset by outperforming most existing methods. Our code will be publicly available at: https://github.com/liujf69/ICMEW2024-Track10.