RGB-D-based Human Motion Recognition with Deep Learning: A Survey

TL;DR

This survey reviews recent deep learning methods for RGB-D-based human motion recognition, categorizing approaches by modality and discussing their advantages, limitations, and future research directions.

Contribution

It provides a comprehensive overview of deep learning techniques applied to RGB-D motion recognition, highlighting recent advances and categorizing methods by modality.

Findings

Deep learning methods have significantly improved RGB-D motion recognition.

Different modalities (RGB, depth, skeleton, RGB+D) offer unique advantages.

The paper discusses current limitations and future research directions.

Abstract

Human motion recognition is one of the most important branches of human-centered research activities. In recent years, motion recognition based on RGB-D data has attracted much attention. Along with the development in artificial intelligence, deep learning techniques have gained remarkable success in computer vision. In particular, convolutional neural networks (CNN) have achieved great success for image-based tasks, and recurrent neural networks (RNN) are renowned for sequence-based problems. Specifically, deep learning methods based on the CNN and RNN architectures have been adopted for motion recognition using RGB-D data. In this paper, a detailed overview of recent advances in RGB-D-based motion recognition is presented. The reviewed methods are broadly categorized into four groups, depending on the modality adopted for recognition: RGB-based, depth-based, skeleton-based and RGB+D-based. As a survey focused on the application of deep learning to RGB-D-based motion recognition, we explicitly discuss the advantages and limitations of existing techniques. Particularly, we highlighted the methods of encoding spatial-temporal-structural information inherent in video sequence, and discuss potential directions for future research.