DG-STGCN: Dynamic Spatial-Temporal Modeling for Skeleton-based Action Recognition

TL;DR

This paper introduces DG-STGCN, a dynamic graph convolutional network that adaptively models spatial and temporal features for skeleton-based action recognition, surpassing previous methods by learning flexible joint relationships.

Contribution

The paper proposes a novel framework with learned affinity matrices for dynamic spatial modeling and group-wise temporal convolutions, improving flexibility and accuracy in skeleton-based action recognition.

Findings

Outperforms state-of-the-art on NTURGB+D, Kinetics-Skeleton, BABEL, and Toyota SmartHome datasets.

Demonstrates significant accuracy improvements over fixed-structure GCN methods.

Validates the effectiveness of dynamic structure learning in capturing complex joint correlations.

Abstract

Graph convolution networks (GCN) have been widely used in skeleton-based action recognition. We note that existing GCN-based approaches primarily rely on prescribed graphical structures (ie., a manually defined topology of skeleton joints), which limits their flexibility to capture complicated correlations between joints. To move beyond this limitation, we propose a new framework for skeleton-based action recognition, namely Dynamic Group Spatio-Temporal GCN (DG-STGCN). It consists of two modules, DG-GCN and DG-TCN, respectively, for spatial and temporal modeling. In particular, DG-GCN uses learned affinity matrices to capture dynamic graphical structures instead of relying on a prescribed one, while DG-TCN performs group-wise temporal convolutions with varying receptive fields and incorporates a dynamic joint-skeleton fusion module for adaptive multi-level temporal modeling. On a wide range of benchmarks, including NTURGB+D, Kinetics-Skeleton, BABEL, and Toyota SmartHome, DG-STGCN consistently outperforms state-of-the-art methods, often by a notable margin.