Pose-Guided Graph Convolutional Networks for Skeleton-Based Action Recognition

TL;DR

This paper introduces PG-GCN, a multi-modal graph convolutional network that fuses pose and skeleton data early in the process, significantly improving human action recognition performance.

Contribution

The paper proposes a novel pose-guided GCN framework with a dynamic attention module for early feature fusion, enhancing action recognition accuracy.

Findings

Achieves state-of-the-art results on NTU RGB+D 60 and 120 datasets.

Demonstrates improved robustness through multi-modal feature fusion.

Validates effectiveness of early-stage pose and skeleton data integration.

Abstract

Graph convolutional networks (GCNs), which can model the human body skeletons as spatial and temporal graphs, have shown remarkable potential in skeleton-based action recognition. However, in the existing GCN-based methods, graph-structured representation of the human skeleton makes it difficult to be fused with other modalities, especially in the early stages. This may limit their scalability and performance in action recognition tasks. In addition, the pose information, which naturally contains informative and discriminative clues for action recognition, is rarely explored together with skeleton data in existing methods. In this work, we propose pose-guided GCN (PG-GCN), a multi-modal framework for high-performance human action recognition. In particular, a multi-stream network is constructed to simultaneously explore the robust features from both the pose and skeleton data, while a dynamic attention module is designed for early-stage feature fusion. The core idea of this module is to utilize a trainable graph to aggregate features from the skeleton stream with that of the pose stream, which leads to a network with more robust feature representation ability. Extensive experiments show that the proposed PG-GCN can achieve state-of-the-art performance on the NTU RGB+D 60 and NTU RGB+D 120 datasets.