Topology-aware Convolutional Neural Network for Efficient Skeleton-based Action Recognition

TL;DR

This paper introduces Ta-CNN, a pure CNN architecture for skeleton-based action recognition that effectively models topology, achieving competitive results with less complexity than graph convolutional networks.

Contribution

The paper proposes a novel topology-aware CNN architecture with a feature augmentation module and a SkeletonMix strategy, demonstrating that CNNs can effectively model skeleton topology.

Findings

Outperforms existing CNN-based methods on multiple datasets.

Achieves comparable performance to GCN-based methods with less complexity.

Theoretically shows graph convolution as a special case of normal convolution.

Abstract

In the context of skeleton-based action recognition, graph convolutional networks (GCNs) have been rapidly developed, whereas convolutional neural networks (CNNs) have received less attention. One reason is that CNNs are considered poor in modeling the irregular skeleton topology. To alleviate this limitation, we propose a pure CNN architecture named Topology-aware CNN (Ta-CNN) in this paper. In particular, we develop a novel cross-channel feature augmentation module, which is a combo of map-attend-group-map operations. By applying the module to the coordinate level and the joint level subsequently, the topology feature is effectively enhanced. Notably, we theoretically prove that graph convolution is a special case of normal convolution when the joint dimension is treated as channels. This confirms that the topology modeling power of GCNs can also be implemented by using a CNN. Moreover, we creatively design a SkeletonMix strategy which mixes two persons in a unique manner and further boosts the performance. Extensive experiments are conducted on four widely used datasets, i.e. N-UCLA, SBU, NTU RGB+D and NTU RGB+D 120 to verify the effectiveness of Ta-CNN. We surpass existing CNN-based methods significantly. Compared with leading GCN-based methods, we achieve comparable performance with much less complexity in terms of the required GFLOPs and parameters.