Hierarchical Graph Convolutional Skeleton Transformer for Action Recognition

TL;DR

This paper introduces HGCT, a hierarchical model combining GCNs and Transformers with a novel DSTT block to improve skeleton-based action recognition by disentangling features and capturing global and local spatiotemporal information.

Contribution

It proposes a new architecture, HGCT, integrating GCN and Transformer benefits with a DSTT block for enhanced action recognition performance.

Findings

HGCT outperforms existing methods in accuracy.

The DSTT block effectively disentangles spatiotemporal features.

The model is lightweight and computationally efficient.

Abstract

Graph convolutional networks (GCNs) have emerged as dominant methods for skeleton-based action recognition. However, they still suffer from two problems, namely, neighborhood constraints and entangled spatiotemporal feature representations. Most studies have focused on improving the design of graph topology to solve the first problem but they have yet to fully explore the latter. In this work, we design a disentangled spatiotemporal transformer (DSTT) block to overcome the above limitations of GCNs in three steps: (i) feature disentanglement for spatiotemporal decomposition;(ii) global spatiotemporal attention for capturing correlations in the global context; and (iii) local information enhancement for utilizing more local information. Thereon, we propose a novel architecture, named Hierarchical Graph Convolutional skeleton Transformer (HGCT), to employ the complementary advantages of GCN (i.e., local topology, temporal dynamics and hierarchy) and Transformer (i.e., global context and dynamic attention). HGCT is lightweight and computationally efficient. Quantitative analysis demonstrates the superiority and good interpretability of HGCT.