Leveraging Spatio-Temporal Dependency for Skeleton-Based Action Recognition

TL;DR

This paper introduces STC-Net, a novel approach for skeleton-based action recognition that effectively models spatio-temporal dependencies using dynamic curves and dilated kernels, achieving state-of-the-art results.

Contribution

The paper proposes the STC-Net with the Spatio-Temporal Curve module and Dilated Kernels for Graph Convolution, which jointly capture adaptive spatio-temporal dependencies.

Findings

Achieves state-of-the-art performance on four benchmarks.

Effectively models long-range spatio-temporal dependencies.

Introduces adaptive node connection identification for dynamic coverage.

Abstract

Skeleton-based action recognition has attracted considerable attention due to its compact representation of the human body's skeletal sructure. Many recent methods have achieved remarkable performance using graph convolutional networks (GCNs) and convolutional neural networks (CNNs), which extract spatial and temporal features, respectively. Although spatial and temporal dependencies in the human skeleton have been explored separately, spatio-temporal dependency is rarely considered. In this paper, we propose the Spatio-Temporal Curve Network (STC-Net) to effectively leverage the spatio-temporal dependency of the human skeleton. Our proposed network consists of two novel elements: 1) The Spatio-Temporal Curve (STC) module; and 2) Dilated Kernels for Graph Convolution (DK-GC). The STC module dynamically adjusts the receptive field by identifying meaningful node connections between every adjacent frame and generating spatio-temporal curves based on the identified node connections, providing an adaptive spatio-temporal coverage. In addition, we propose DK-GC to consider long-range dependencies, which results in a large receptive field without any additional parameters by applying an extended kernel to the given adjacency matrices of the graph. Our STC-Net combines these two modules and achieves state-of-the-art performance on four skeleton-based action recognition benchmarks.