Are Spatial-Temporal Graph Convolution Networks for Human Action Recognition Over-Parameterized?

TL;DR

This paper investigates the over-parameterization of spatial-temporal graph convolutional networks for human action recognition and introduces a sparse architecture that maintains performance with significantly fewer parameters.

Contribution

The authors propose a novel sparse ST-GCN generator and multi-level sparsity models that reduce parameters while preserving or improving recognition accuracy.

Findings

Sparse ST-GCNs achieve comparable performance with 95% fewer parameters.

Multi-level sparsity ST-GCNs outperform dense models with only 66% of the parameters.

Experiments on four datasets validate the effectiveness of the sparse architectures.

Abstract

Spatial-temporal graph convolutional networks (ST-GCNs) showcase impressive performance in skeleton-based human action recognition (HAR). However, despite the development of numerous models, their recognition performance does not differ significantly after aligning the input settings. With this observation, we hypothesize that ST-GCNs are over-parameterized for HAR, a conjecture subsequently confirmed through experiments employing the lottery ticket hypothesis. Additionally, a novel sparse ST-GCNs generator is proposed, which trains a sparse architecture from a randomly initialized dense network while maintaining comparable performance levels to the dense components. Moreover, we generate multi-level sparsity ST-GCNs by integrating sparse structures at various sparsity levels and demonstrate that the assembled model yields a significant enhancement in HAR performance. Thorough experiments on four datasets, including NTU-RGB+D 60(120), Kinetics-400, and FineGYM, demonstrate that the proposed sparse ST-GCNs can achieve comparable performance to their dense components. Even with 95% fewer parameters, the sparse ST-GCNs exhibit a degradation of <1% in top-1 accuracy. Meanwhile, the multi-level sparsity ST-GCNs, which require only 66% of the parameters of the dense ST-GCNs, demonstrate an improvement of >1% in top-1 accuracy. The code is available at https://github.com/davelailai/Sparse-ST-GCN.