A Survey on 3D Skeleton-Based Action Recognition Using Learning Method

TL;DR

This survey reviews deep learning methods for 3D skeleton-based action recognition, covering architectures like RNNs, CNNs, GCNs, and Transformers, and discusses key datasets and top algorithms in the field.

Contribution

It provides the first comprehensive review of deep learning architectures applied to 3D skeleton-based action recognition, filling a gap in existing surveys.

Findings

Deep architectures like GCNs and Transformers are prominent in current methods.

NTU-RGB+D datasets are central benchmarks for evaluating algorithms.

Top algorithms achieve high accuracy on these datasets.

Abstract

3D skeleton-based action recognition (3D SAR) has gained significant attention within the computer vision community, owing to the inherent advantages offered by skeleton data. As a result, a plethora of impressive works, including those based on conventional handcrafted features and learned feature extraction methods, have been conducted over the years. However, prior surveys on action recognition have primarily focused on video or RGB data-dominated approaches, with limited coverage of reviews related to skeleton data. Furthermore, despite the extensive application of deep learning methods in this field, there has been a notable absence of research that provides an introductory or comprehensive review from the perspective of deep learning architectures. To address these limitations, this survey first underscores the importance of action recognition and emphasizes the significance of 3D skeleton data as a valuable modality. Subsequently, we provide a comprehensive introduction to mainstream action recognition techniques based on four fundamental deep architectures, i.e., Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), Graph Convolutional Network (GCN), and Transformers. All methods with the corresponding architectures are then presented in a data-driven manner with detailed discussion. Finally, we offer insights into the current largest 3D skeleton dataset, NTU-RGB+D, and its new edition, NTU-RGB+D 120, along with an overview of several top-performing algorithms on these datasets. To the best of our knowledge, this research represents the first comprehensive discussion of deep learning-based action recognition using 3D skeleton data.