Revisiting Skeleton-based Action Recognition

TL;DR

This paper introduces PoseC3D, a novel 3D heatmap-based approach for skeleton-based action recognition that outperforms GCN methods in robustness, scalability, and multi-person scenarios.

Contribution

PoseC3D replaces graph convolutional networks with 3D heatmaps, improving feature learning, robustness, and multi-modal integration in skeleton-based action recognition.

Findings

PoseC3D outperforms GCN-based methods on four datasets.

PoseC3D is robust to pose estimation noise.

PoseC3D effectively handles multi-person scenarios.

Abstract

Human skeleton, as a compact representation of human action, has received increasing attention in recent years. Many skeleton-based action recognition methods adopt graph convolutional networks (GCN) to extract features on top of human skeletons. Despite the positive results shown in previous works, GCN-based methods are subject to limitations in robustness, interoperability, and scalability. In this work, we propose PoseC3D, a new approach to skeleton-based action recognition, which relies on a 3D heatmap stack instead of a graph sequence as the base representation of human skeletons. Compared to GCN-based methods, PoseC3D is more effective in learning spatiotemporal features, more robust against pose estimation noises, and generalizes better in cross-dataset settings. Also, PoseC3D can handle multiple-person scenarios without additional computation cost, and its features can be easily integrated with other modalities at early fusion stages, which provides a great design space to further boost the performance. On four challenging datasets, PoseC3D consistently obtains superior performance, when used alone on skeletons and in combination with the RGB modality.