Learning to Recognize 3D Human Action from A New Skeleton-based Representation Using Deep Convolutional Neural Networks

TL;DR

This paper introduces a novel skeleton-based 3D human action recognition method that transforms joint coordinates into RGB images for deep learning, achieving superior accuracy with less computation.

Contribution

The paper proposes a new RGB image encoding of skeleton sequences and applies deep CNNs, improving recognition accuracy and efficiency over previous methods.

Findings

Outperforms state-of-the-art on MSR Action3D and NTU-RGB+D datasets.

Requires less computation for training and prediction.

Effective representation of complex 3D motions.

Abstract

Recognizing human actions in untrimmed videos is an important challenging task. An effective 3D motion representation and a powerful learning model are two key factors influencing recognition performance. In this paper we introduce a new skeleton-based representation for 3D action recognition in videos. The key idea of the proposed representation is to transform 3D joint coordinates of the human body carried in skeleton sequences into RGB images via a color encoding process. By normalizing the 3D joint coordinates and dividing each skeleton frame into five parts, where the joints are concatenated according to the order of their physical connections, the color-coded representation is able to represent spatio-temporal evolutions of complex 3D motions, independently of the length of each sequence. We then design and train different Deep Convolutional Neural Networks (D-CNNs) based on the Residual Network architecture (ResNet) on the obtained image-based representations to learn 3D motion features and classify them into classes. Our method is evaluated on two widely used action recognition benchmarks: MSR Action3D and NTU-RGB+D, a very large-scale dataset for 3D human action recognition. The experimental results demonstrate that the proposed method outperforms previous state-of-the-art approaches whilst requiring less computation for training and prediction.

Tables

Table 1. Table 1: Three subsets AS1 , AS2 , and AS3 of the MSR Action3D dataset Li2010ActionRB .

AS1	AS2	AS3
Horizontal arm wave	High arm wave	High throw
Hammer	Hand catch	Forward kick
Forward punch	Draw x	Side kick
High throw	Draw tick	Jogging
Hand clap	Draw circle	Tennis swing
Bend	Two hand wave	Tennis serve
Tennis serve	Forward kick	Golf swing
Pickup & Throw	Side-boxing	Pickup & Throw

Table 2. Table 2: Test accuracies (%) of our proposed ResNets on AS1 , AS2 , and AS3 subsets. The best results and configuration are highlighted in bold .

Network	AS1	AS2	AS3	Aver.
ResNet-20	99.60%	98.90%	100.0%	99.50%
ResNet-32	99.80%	99.00%	99.80%	99.53%
ResNet-44	99.50%	98.90%	100.0%	99.48%
ResNet-56	99.40%	98.70%	99.50%	98.20%
ResNet-110	99.00%	96.50%	99.30%	99.28%

Table 3. Table 3: Comparison with the state-of-the-art approaches on MSR Action3D dataset Li2010ActionRB . The best performances are in bold .

Method	AS1	AS2	AS3	Aver.
Bag of 3D Points Li2010ActionRB	72.90%	71.90%	79.20%	74.67%
Motion Trail Model Liang2013ThreeDM	73.70%	81.50%	81.60%	78.93%
Histograms of 3D Joints xia2012view	87.98%	85.48%	63.46%	78.97%
Motion and Shape Features qin2013gesture	81.00%	79.00%	82.00%	80.66%
Spatial-temporal Features hbali2017skeleton	77.36%	73.45%	91.96%	80.92%
Space-time Occupancy vieira2012stop	84.70%	81.30%	88.40%	84.80%
Improved Space-time Occ. Vieira2014OnTI	91.70%	72.20%	98.60%	87.50%
HON4D Oreifej2013HON4DHO	N/A	N/A	N/A	88.89%
Skeletal Quads Evangelidis2014SkeletalQH	88.39%	86.61%	94.59%	89.86%
Multi-modality Information Gao2014HumanAR	92.00%	85.00%	93.00%	90.00%
Depth Motion Maps Chen2013RealtimeHA	96.20%	83.20%	92.00%	90.47%
Covariance Descriptors Hussein:2013:HAR:2540128.2540483	88.04%	89.29%	94.29%	90.53%
HOD Gowayyed2013HistogramOO	92.39%	90.18%	91.43%	91.26%
Moving Pose zanfir2013moving	N/A	N/A	N/A	91.70%
Skeletal and Silhouette Fusion Chaaraoui2013FusionOS	92.38%	86.61%	96.40%	91.80%
Improved Key Poses 10.1007/978-3-319-77383-4_35	91.53%	90.23%	97.06%	92.94%
Pose-based Representation Theodorakopoulos2014PosebasedHA	91.23%	90.09%	99.50%	93.61%
H-RNN 7298714	93.33%	94.64%	95.50%	94.49%
Local Binary Patterns Chen2015ActionRF	98.10%	92.00%	94.60%	94.90%
Spatio-temporal Pyramid Xu2015SpatioTemporalPM	99.10%	92.90%	96.40%	96.10%
Depth Motion Maps jin2017action	99.10%	92.30%	98.20%	96.50%
Features Combination luvizon2017learning	N/A	N/A	N/A	97.10%
Group Sparsity Luo2013GroupSA	97.20%	95.50%	99.10%	97.26%
Our best configuration	99.80%	99.00%	99.80%	99.53%

Table 4. Table 4: Test accuracies (%) of our proposed networks on Cross-Subject and Cross-View settings Shahroudy2016NTURA . The best results and configuration are in bold .

Network	Cross-Subject	Cross-View
ResNet-20	73.40%	80.10%
ResNet-32	73.10%	80.40%
ResNet-44	72.40%	79.80%
ResNet-56	73.00%	79.80%
ResNet-110	67.40%	78.60%

Table 5. Table 5: Comparison with state-of-the-art methods on NTU-RGB+D dataset Shahroudy2016NTURA . The best results and configuration are marked in bold .

Method	Cross-Subject	Cross-View
HON4D Oreifej2013HON4DHO	30.56%	7.26%
Super Normal Vector Yang2014SuperNV	31.82%	13.61%
Joint Angles + HOG2 OhnBar2013JointAS	32.24%	22.27%
Skeletal Quads Evangelidis2014SkeletalQH	38.62%	41.36%
Shuffle and Learn Misra2016ShuffleAL	47.50%	N/A
Histograms of Key Poses Cippitelli2016EvaluationOA	48.90%	57.70%
Lie Group Vemulapalli2014HumanAR	50.08%	52.76%
Rolling Rotations Vemulapalli2016RollingRF	52.10%	53.40%
H-RNN 7298714 (reported in Shahroudy2016NTURA )	59.07%	63.79%
P-LSTM Shahroudy2016NTURA	62.93%	70.27%
Long-Term Motion Luo2017UnsupervisedLO	66.22%	N/A
Spatio-temporal LSTM Liu2016SpatioTemporalLW	69.20%	77.70%
Our best configuration	73.40%	80.40%

Table 6. Table 6: Execution time of each component of the proposed learning framework.

Stage	Average processing time
Stage A	$\text{20.8}\times {\text{10}}^{-3}$s per sequence (CPU time)
Stage B	$\text{4.15}\times {\text{10}}^{-3}$s per sequence (GPU time)
Stage C	$\text{21.84}\times {\text{10}}^{-3}$s per sequence (CPU time)

Equations

$$(x_i^{\prime },y_i^{\prime },z_i^{\prime })=\mathcal{F}(x_i,y_i,z_i)$$

$$x_i^{\prime }=255\times \frac{(x_i-\min \{\mathcal{C}\})}{\max \{\mathcal{C}\}-\min \{\mathcal{C}\}}$$

$$y_i^{\prime }=255\times \frac{(y_i-\min \{\mathcal{C}\})}{\max \{\mathcal{C}\}-\min \{\mathcal{C}\}}$$

$$z_i^{\prime }=255\times \frac{(z_i-\min \{\mathcal{C}\})}{\max \{\mathcal{C}\}-\min \{\mathcal{C}\}}$$

$$I_{\text{RGB}}=[{\text{F}}_1^{{}^{\prime }},{\text{F}}_2^{{}^{\prime }},...,{\text{F}}_N^{{}^{\prime }}],$$

$$\text{Arg}\underset{\mathcal{W}}{\min }({\mathcal{L}}_{\mathcal{X}}(\text{y},\hat{\text{y}}))=\text{Arg}\underset{\mathcal{W}}{\min }\left(-\frac{1}{M}\sum _{i=1}^M\sum _{j=1}^C{\text{y}}_{ij}\log {\hat{\text{y}}}_{ij}\right),$$

Full text

1 Introduction

Human Action Recognition (HAR) Ranasinghe2016ARO is an important topic in machine vision. HAR methods have been widely applied in building electronic systems based on machine intelligence such as intelligent surveillance Niu2004HumanAD ; Lin2008HumanAR , human-machine interaction Pickering2007ARS ; Sonwalkar2015HandGR , health care pansiot2007ambient , and so on. Although significant progress has been made in the last few years, HAR is still a challenging task due to many obstacles, e.g. camera viewpoint, occlusions, or large intra-class variations Poppe2010ASO .

    There are two main types of intelligence that are artificial intelligence and machine intelligence. HAR can be considered as an electronic system based on machine intelligence. The first stage of any HAR system is data acquisition. Nowadays, a variety of electronic imaging systems can be used for this task such as traditional optical cameras, RGB-D sensors, Thermal Infrared (IR) sensors, and Synthetic Aperture Radar (SAR) sensors. For instance, IR sensors have been exploited in HAR akula2018deep . These sensors are able to generate images based on the heat radiated by the human body and to work independently of lighting conditions. SAR imaging has also been widely used for detecting and analyzing human activities gurbuz2007detection . This type of sensor is able to operate far away from potential targets and functions during the daytime as well as nighttime, under all weather conditions. In particular, SAR Image Segmentation (SIS) techniques akbarizadeh2012new ; tirandaz2016unsupervised ; farbod2018optimized can segment humans from other components (e.g. objects, background) in the scene. As a result, the use of SAR sensors offers big advantages for human action recognition, in particular for human target detection and identification in a complex environment such as in military applications.

    Traditional studies on HAR mainly focus on the use of RGB sequences provided by 2D cameras. These approaches typically recognize actions based on hand-crafted local features such as Cuboids Dollr2005BehaviorRV , HOG/HOF Laptev2008LearningRH , HOG-3D Klser2008ASD , which are extracted from the appearance and movements of human body parts. However, it is very hard to fully capture and model the spatial-temporal information of human action due to the absence of 3D structure from the scene. Therefore, RGB-D cameras are becoming one of the most commonly-used sensors for HAR zhang2016rgb ; ye2013survey ; ijjina2017human . Recently, the rapid development of cost-effective and easy-to-use depth cameras, e.g. Microsoft Kinect ${}^{\text{TM}}$sensor Cruz2012KinectAR ; Han2013EnhancedCV , ASUS Xtion PRO ASUS , or Intel*®* RealSense${}^{\text{TM}}$ Intel has helped dealing with problems related to 3D action recognition. In general, depth sensors are able to provide detailed information about the 3D structure of human body. Therefore, many approaches have been proposed for 3D HAR based on data provided by depth sensors such as RGB sequences, depth, or combining these two data types (RGB-D). Furthermore, these devices have integrated real-time skeleton tracking algorithms Shotton2011RealtimeHP that provide high-level information about human motions in a 3D space. Thus, exploiting skeletal data for 3D HAR is an active research topic in computer vision Wang2012MiningAE ; Xia2012ViewIH ; Chaudhry2013BioinspiredD3 ; Vemulapalli2014HumanAR ; Ding2016ProfileHF .

    In recent years, approaches based on Convolutional Neural Networks (CNNs) have achieved outstanding results in many image recognition tasks Krizhevsky2012ImageNetCW ; Karpathy2014LargeScaleVC ; Szegedy2016RethinkingTI . After the success of AlexNet Krizhevsky2012ImageNetCW , a new direction of research has been opened for designing and optimizing higher performing CNN architectures. As a result, there are substantial evidences that seem to show that the learning performance of CNNs may be significantly improved by increasing the number of hidden layers Simonyan2014VeryDC; Szegedy2015GoingDW ; Telgarsky2016BenefitsOD . In the literature of HAR, many studies have indicated that CNNs have the ability to learn complex motion features better than hand-crafted approaches. However, most previous works have just focused on exploring simple CNNs such as AlexNet Krizhevsky2012ImageNetCW and have not exploited the potential of recent state-of-the-art very deep CNN architectures (D-CNNs), e.g. Residual Networks (ResNet) He2016DeepRL . In addition, most existing CNN-based approaches limit themselves to using RGB-D sequences as the input to learning models. Although RGB-D images are informative for understanding human action, the computation complexity of learning models increases rapidly when the dimension of the input features is large. Therefore, representation learning models based on RGB-D modality become more complex, slower and less practical for solving large-scale problems as well as real-time applications.

    Different from previous works, to take full advantages of 3D skeletal data and the learning capacity of D-CNNs, this paper proposes an end-to-end deep learning framework for 3D HAR from skeleton sequences. We focus on solving two main issues: First, using a simple skeleton-to-image encoding method to transform the 3D coordinates of the skeletal joints into RGB images. The encoding method needs to ensure that the image-based representation of skeleton sequences is able to effectively represent the spatial structure and temporal dynamics of the human action. Second, we design and train different D-CNNs based on Residual Networks (ResNets) He2016DeepRL – a recent state-of-the-art CNN for image recognition, to learn and classify actions from the obtained image-coded representations.

    This paper is an extended version of our work published in the 8th International Conference of Pattern Recognition Systems (ICPRS 2017) iet:/content/conferences/10.1049/cp.2017.0154 . It is part of our research project about investigating and developing a low-cost system based on RGB-D data provided by depth sensors for understanding human actions and analyzing their behaviors in indoor environments such as inside home, offices, residential buildings, buses, trains, etc. In general, three observations motivate our exploration of using ResNet He2016DeepRL for 3D HAR from skeleton sequences, including: (1) human actions can be correctly represented as the movements of skeletons Vemulapalli2014HumanAR and the spatial-temporal dynamics of skeletons can be transformed into 2D image structure, which can be effectively learned by D-CNNs; (2) skeletal data is high-level information with much less complexity than RGB-D sequences, this advantage makes our action learning model much simpler and faster; (3) many evidences Szegedy2015GoingDW ; Telgarsky2016BenefitsOD show that deeper convolutional model, especially ResNet He2016DeepRL can boost the learning accuracy in image recognition tasks.

    We evaluate the proposed method on two benchmark skeleton datasets (i.e. MSR Action3D Li2010ActionRB and NTU-RGB+D Shahroudy2016NTURA ). Experimental results confirmed the above statements since our method achieveds state-of-the-art performance compared with the existing results using the same evaluation protocols. Furthermore, we also indicate the effectiveness of our learning framework in terms of computational complexity.

    In summary, two main contributions of this paper include:

$\bullet$ First, we introduce a new skeleton-based representation (Sec. 3.1) and an end-to-end learning framework111Codes and pre-trained models will be shared to the community at https://github.com/huyhieupham/ after publication. (Sec. 3.2) based on D-CNNs to learn the spatio-temporal evolutions of 3D motions and then to recognize human actions;

$\bullet$ Second, we show the effectiveness of our method on HAR tasks by achieving state-of-the-art performance on two benchmark datasets, including the most challenging skeleton benchmark currently available for 3D HAR Shahroudy2016NTURA (Sec. 5).

The advantage of the proposed method is that it has high computational efficiency (Sec. 5.4). Moreover, this approach is general and can be easily applied to other time-series problem, e.g. recognizing human actions with mobile devices with integrated inertial sensors.

    The rest of the paper is organized as follows: Section 2 discusses related works. In Section 3, we present the details of our proposed method, including the color-encoding process from skeletons to RGB images and deep learning networks. Datasets and experiments are described in Section 4. Experimental results are reported in Section 5 with a detailed analysis of computational efficiency. Finally, Section 6 concludes the paper with a discussion on the future work.

2 Related Work

Skeleton-based Action Recognition (SBAR) using depth sensors has been widely studied in recent years. In this section, we briefly review existing SBAR methods, including two main categories that are directly related to our work. The first category is approaches based on hand-crafted local features of skeletons. The second is approaches based on deep learning networks, especially Recurrent Neural Networks with Long Short Term Memory (RNN-LSTMs).

Approaches based on hand-crafted local features: Human motion can be considered as a spatio-temporal pattern. Many researchers have built hand-crafted feature representations for SBAR and then used temporal models for modeling 3D human motion. For instance, Wang et al. Wang2012MiningAE represented the human motion from skeletal data by means of the pairwise relative positions of the key joints. The authors then employed Hidden Markov Model (HMM) Yoon2009HiddenMM for modeling temporal dynamics of actions. A similar approach has been presented by Lv et al. Lv2006 where the 3D joint position trajectories have been mapped into feature spaces. The dynamics of actions was then learned by one continuous HMM. Xia et al. Xia2012ViewIH proposed the use of histogram-based representation of human pose, and then actions were classified into classes by a discrete HMM. Wu and Shao Wu2014LeveragingHP also exploited HMM for recognizing actions from high-level features of skeletons. Vemulapalli et al. Vemulapalli2014HumanAR represented the 3D geometric interactions of the body parts in a Lie Group. These elements were processed with a Fourier Temporal Pyramid (FTP) transformation for modeling temporal evolutions of the original motions. Different from the studies above, Luo et al. Luo2013GroupSA proposed a new dictionary learning model to learn the spatio-temporal information from skeleton sequences. Although promising results have been achieved from approaches based on hand-crafted local features and probabilistic graphical models, they have some limitations that are very difficult to overcome. E.g., most of these approaches are data-dependent and require a lot of hand-designing features. HMM-based methods require preprocessing input data in which the skeleton sequences need to be segmented or aligned. Meanwhile, FTP-based approaches can only utilize limited contextual information of actions.

Approaches based on deep learning: Recurrent Neural Networks with Long Short-Term Memory Network (RNN-LSTMs) hochreiter1997long ; Graves2008SupervisedSL are able to model the contextual information of the temporal sequences as skeleton data. Thus, many authors have explored RNN-LSTMs for SAR. For instance, Du et al. 7298714 proposed an end-to-end hierarchical RNN-LSTM for modeling local motions of a body part in which all skeleton frames were divided into five parts according to the human physical structure. Each part of a skeleton was then fed into an independent RNN and then fused to be the inputs of higher layers. The final representation was used for the classification task. Zhu et al. zhu2016co proposed a LSTM network with a mixed-norm regularization term to cost functions in order to learn the co-occurrence of discriminative features from skeletal data. Liu et al. Liu2016SpatioTemporalLW introduced a new gating mechanism with LSTM network to analyze the reliability of input skeleton sequences. In another study, Shahroudy et al. Shahroudy2016NTURA used a part-aware LSTM network in which the memory-cell was divided into sub-cells such that each sub-cell can model long-term contextual representation of a body part. Finally, all these sub-cells were concatenated to the final output. Although RNN-LSTMs are able to model the long-term temporal of motion and experimental results provided that RNN-LSTMs have outperformed many other approaches, RNN-based approaches just consider skeleton sequences as a kind of low-level feature by feeding directly the raw 3D joint coordinates carried in skeletons into the network input. The huge number of input parameters may make RNNs become very complex and easily lead to overfitting due to insufficient training data. Moreover, many RNN-LSTMs act as a classifier and cannot extract high-level features Sainath2015ConvolutionalLS for recognition tasks.

    Several authors have exploited the feature learning ability of CNNs on skeletal data wang2016action ; hou2016skeleton . However, such studies mainly focus on the use of complex encoding methods for finding good skeletal representations and learning geometric features carried in skeleton sequences with simple CNN architectures. In contrast, in this paper we concentrate on proposing a new and simple skeleton-based representation and exploiting the power of D-CNNs for action recognition. Our experiments on two public datasets, including the MSR Action3D dataset Li2010ActionRB and the NTU-RGB+D dataset Shahroudy2016NTURA show state-of-the-art performance on both datasets. Moreover, in terms of computational cost, our measurement and analyses show that the proposed model is fast enough for many real-time applications.

3 Proposed Method

This section presents the proposed method. We first describe the color-encoding process that allows us to represent the spatial structure and temporal dynamics of skeleton sequences as the static structure of a 2D color image. We then review the main idea of ResNet He2016DeepRL and propose five different ResNet configurations, including 20-layer, 32-layer, 44-layer, 56-layer, and 110-layer networks to learn motion features carried in the image-based representations and perform action classification.

3.1 Building image-based representation from skeleton sequences

Currently, skeletal data that contains 3D joint coordinates can be obtained from depth cameras via real-time skeleton estimation algorithms Davison2003RealTimeSL ; Shotton:2013:RHP:2398356.2398381 . This technology allows to extract the position of the key joints in the body, which is suitable for 3D HAR problems. E.g., the latest version of the Microsoft Kinect ${}^{\text{TM}}$sensor Cruz2012KinectAR ; Han2013EnhancedCV (Kinect v2 sensor) can track the main 25 joints of human body in real-time speed as shown in Figure 1.

In this paper, instead of feeding raw 3D joint coordinates directly to RNN-LSTMs as in many previous works 7298714 ; zhu2016co ; Liu2016SpatioTemporalLW ; Shahroudy2016NTURA , we propose the use of D-CNNs for learning motion features from skeleton sequences. The first step is to find a skeleton-based representation that can effectively capture the spatio-temporal evolutions of skeletons and that can also be easily learned by learning method as D-CNNs. One solution for this problem is to transform each skeleton sequence into a single color image (e.g., RGB image) in which the pixel values must have the ability to represent the movement of skeletons.

    To this end, we transform the 3D joint coordinates of each skeleton into a new space by normalizing their coordinates using a transformation function. Specifically, given a skeleton sequence $\mathcal{S}$ with N frames $[\textbf{F}_{1},\textbf{F}_{2},...,\textbf{F}_{N}]$, let $(x_{i},y_{i},z_{i})$ be the 3D coordinates of each joint in frame $\{\textbf{F}_{n}\}\in\mathcal{S},n\in[1,N]$. A normalization function $\mathcal{F}(\cdot)$ is used to transform all 3D joint coordinates to the range of [0, 255] as follows:

[TABLE]

[TABLE]

[TABLE]

[TABLE]

where $(x^{\prime}_{i},y^{\prime}_{i},z^{\prime}_{i})$ is 3D joint coordinate in the normalized space $\mathcal{S}^{{}^{\prime}}$. The $\max\{\mathcal{C}\}$ and $\min\{\mathcal{C}\}$ are the maximum and minimum values of all coordinates, respectively. To preserve the spatio-temporal information of skeleton movements, we stack all normalized frames according to the temporal order $\mathcal{S}^{{}^{\prime}}=[\textbf{F}_{1}^{{}^{\prime}},\textbf{F}_{2}^{{}^{\prime}},...,\textbf{F}_{N}^{{}^{\prime}}]$ to represent the whole action sequence. These elements are quantified to RGB color space and can be stored as RGB images. In this way, we convert the skeletal data to 3D tensors that will can then be fed into deep learning networks as the input for feature leaning and classification. Figure 2 illustrates the skeleton-to-image transformation process.

Naturally, the human body is structured as four limbs and one trunk (see Figure 1). Simple actions can be performed through the movement of a limb, e.g., hand-waving, kicking forward, etc. More complex actions combine the movements of a group of limbs or the whole body, e.g., running or swimming. To keep the local characteristics of the human action yacoob1999parameterized ; chaudhry2013bio , we divide each skeleton into five parts, including two arms ($\textbf{P}_{1}$, $\textbf{P}_{2}$), two legs ($\textbf{P}_{4}$, $\textbf{P}_{5}$), and one trunk ($\textbf{P}_{3}$). In each part, the joints are concatenated according their physical connections as shown in Figure 3. These parts are then connected in a sequential order: $\textbf{P}_{1}\to\textbf{P}_{2}\to\textbf{P}_{3}\to\textbf{P}_{4}\to\textbf{P}_{5}$. The whole process of rearranging all frames in a sequence can be done by rearranging the order of rows of pixels in the color-based representation. This process is illustrated in Figure 4.

Like that, we have encoded skeleton sequences into RGB images. Figure 5 shows some examples of RGB images obtained from input sequences of MSR Action3D dataset Li2010ActionRB . These images will be learned and classified by D-CNN models. This way, the original skeleton sequence will be recognized via the corresponding image.

3.2 Deep residual networks for skeleton-based human action recognition

In this section, we introduce our network designs for recognizing human action from image-based representation obtained in Section 3.1. Before that, in order to put our method into context, we will briefly review the key idea of ResNet He2016DeepRL .

3.2.1 Residual learning method

Very deep neural networks, especially D-CNNs have demonstrated to have a high performance on many visual-related tasks DBLP:journals/corr/SimonyanZ14a ; Szegedy2015GoingDW . However, D-CNNs are very difficult to optimize due to the vanishing gradients problem. If the network is deep enough, the error signal from the output layer can be completely attenuated on its way back to the input layer. In addition, another problem called “the degradation phenomenon He2015ConvolutionalNN ” also impedes the convergence of deeper networks. More specially, adding more layers to a deep network can lead to higher training/testing error. To overcome these challenges, ResNet He2016DeepRL has been introduced. The key idea behind ResNet architectures is the presence of shortcut connections between input and output of each ResNet building block. These shortcut connections are implemented by identity mappings and are able to provide a path for gradients to back propagate to early layers in the network. This idea improves the information flow in ResNets and helps them to learn faster. Not only that, experimental results on standard datasets such as CIFAR-10 Krizhevsky2009LearningML , and ImageNet ILSVRC15 for image classification tasks showed that the use of the shortcut connections in ResNet architectures can boost recognition performance. Figure 6 shows information flow in a ResNet building block and their implementation details.

Mathematically, a layer or a series of layers in a typical CNN model will try to learn a mapping function $y=\mathcal{F}(x)$ from input feature $x$. However, a ResNet building block will learn the function $y=\mathcal{F}(x)+id(x)$ where $id(x)$ is a identity function $id(x)=x$. The additional element $id(x)$ is the key factor that improves the information flow through layers during training ResNets.

3.2.2 Network design

A deep ResNet can be constructed from multiple ResNet building blocks that are serially connected to each other. To search for the best recognition performance, we suggest different configurations of ResNet with 20, 32, 44, 56, and 110 layers. For further details, see Appendix. These networks are denoted as ResNet-20, ResNet-32, ResNet-44, ResNet-56, and ResNet-110, respectively. In our implementations, a ResNet building block performs the learning of a function $y=\mathcal{F}(x)+id(x)$ where $id(x)=x$ and $\mathcal{F}(x)$ is implemented by a sequence of layers: Conv-BN-ReLU-Dropout-Conv-BN. Specifically, each ResNet block uses the convolutional layers (Conv) with $\textbf{3}\times\textbf{3}$ filters. The Batch Normalization (BN) Ioffe2015BatchNA and non-linear activation function ReLU Nair2010RectifiedLU are applied after each Conv. To reduce possible overfitting, we add a Dropout layer Srivastava2014DropoutAS with a rate of 0.5 into each ResNet block, located between two Conv layers and after ReLU. Finally, another ReLU layer Nair2010RectifiedLU is used after the element-wise addition. The proposed ResNet unit is shown in Figure 7. We refer the interested reader to the Supplementary Materials to see details of the proposed network architectures.

In this learning framework, all networks are designed for accepting images with size $\textbf{32}\times\textbf{32}$ pixels and trained in an end-to-end manner using Stochastic Gradient Descent (SGD) algorithm Bottou2010 from scratch. The last fully-connected layer of the network represents the action class scores and its size can be changed corresponding to the number of action classes.

3.2.3 Learning to recognize actions from skeleton-based representations

In order to recognize an action from a given skeleton sequence $\mathcal{S}$ with N frames $[\textbf{F}_{1},\textbf{F}_{2},...,\textbf{F}_{N}]$, we firstly encode all frames of $\mathcal{S}$ into a single RGB image $I_{\textbf{RGB}}$ as mentioned in section 3.1

[TABLE]

where all elements of $\textbf{F}_{i|i\in[1,N]}^{{}^{\prime}}$ were normalized to the range of $[\textbf{0},\textbf{255}]$ as color pixels and rearranged according to their physical structure (see Figure 4). We then propose the use of ResNets to learn and classify the obtained color images. During the training phase, we minimize the cross-entropy loss function $\mathcal{L}_{\mathcal{X}}(\textbf{y},\hat{\textbf{y}})$ between the action labels y of $I_{\textbf{RGB}}$ and the predicted action labels $\hat{\textbf{y}}$ by the network over the training set $\mathcal{X}$. In other words, the network will be trained to solve the following problem

[TABLE]

where $\mathcal{W}$ is the set of weights that will be learned by the network, $M$ denotes the number of samples in training set $\mathcal{X}$ and $C$ is the number of action classes.

4 Experiments

In this section, we evaluate the effectiveness of the proposed method on two benchmark datasets: MSR Action3D Li2010ActionRB and NTU-RGB+D Shahroudy2016NTURA . To have a fair comparison with state-of-the-art approaches in the literature, we follow the evaluation protocols as provided in the original papers Li2010ActionRB ; Shahroudy2016NTURA . Recognition performance will be measured by average classification accuracy.

4.1 Datasets and experimental protocols

4.1.1 MSR Action3D Dataset Li2010ActionRB

This dataset222The MSR Action3D dataset can be obtained at: https://www.uow.edu.au/w̃anqing/#Datasets was collected with the Microsoft Kinect v1 ${}^{\text{TM}}$sensor. It contains 20 different actions. Each action was performed by 10 subjects for two or three times. The experiments were conducted on 557 valid sequences of the dataset after removing the defective sequences. We followed the same experimental protocol as many other authors, in which the whole data is divided into three subsets named as AS1, AS2, and AS3. For each subset, five subjects (with IDs: 1, 3, 5, 7, 9) are selected for training and the rest (with IDs: 2, 4, 6, 8, 10) for test. More details about the action classes in each subset can be found in Table 1.

4.1.2 NTU-RGB+D Dataset Shahroudy2016NTURA

The NTU-RGB+D dataset333Obtained at: http://rose1.ntu.edu.sg/Datasets/actionRecognition.asp is a very large-scale RGB+D dataset. To the best of our knowledge, the NTU-RGB+D is currently the largest dataset that contains skeletal data for HAR. It is a very challenging dataset due to the large intra-class variations and multiple viewpoints. Specifically, the NTU-RGB+D provides more than 56,000 videos, collected from 40 subjects for 60 action classes. The list of action classes in NTU RGB+D dataset includes: drinking, eating, brushing teeth, brushing hair, dropping, picking up, throwing, sitting down, standing up (from sitting position), clapping, reading, writing, tearing up paper, wearing jacket, taking off jacket, wearing a shoe, taking off a shoe, wearing on glasses, taking off glasses, putting on a hat/cap, taking off a hat/cap, cheering up, hand waving, kicking something, reaching into self pocket, hopping, jumping up, making/answering a phone call, playing with phone, typing, pointing to something, taking selfie, checking time (on watch), rubbing two hands together, bowing, shaking head, wiping face, saluting, putting palms together, crossing hands in front, sneezing/coughing, staggering, falling down, touching head (headache), touching chest (stomachache/heart pain), touching back (back-pain), touching neck (neck-ache), vomiting, fanning self, punching/slapping other person, kicking other person, pushing other person, patting other’s back, pointing to the other person, hugging, giving something to other person, touching other person’s pocket, handshaking, walking towards each other, and walking apart from each other.

    The NTU RGB+D data was collected by the Microsoft Kinect v2 ${}^{\text{TM}}$sensor. Each skeleton frame provides the 3D coordinates of 25 body joints. Two different evaluation criteria have been suggested, including Cross-Subject and Cross-View. In the Cross-Subject settings, 20 subjects (with IDs: 1, 2, 4, 5, 8, 9, 13, 14, 15, 16, 17, 18, 19, 25, 27, 28, 31, 34, 35, and 38) are used for training and the rest are used for test. For Cross-View settings, the sequences provided by cameras 2 and 3 are used for training and sequences provided by camera 1 are used for test.

4.2 Data augmentation

Very deep neural networks as ResNets He2016DeepRL require a lot of data to optimize. D-CNNs cannot learn and classify images effectively when training with an insufficient data amount. Unfortunately, we have only 557 skeleton sequences (less than 30 images per class) on MSR Action3D dataset Li2010ActionRB . Therefore, data augmentation has been applied to prevent ResNets from overfitting. We used random cropping, flip horizontally and vertically techniques to generate more training samples. More specifically, 8 $\times$ cropping has been applied on 40 $\times$ 40 images to create 32 $\times$ 32 images. Then, their horizontally and vertically flipped images are also created. Figure 8 illustrates some data augmentation techniques on an image sample obtained from the MSR Action3D dataset Li2010ActionRB . For the NTU-RGB+D dataset Shahroudy2016NTURA , we do not apply any data augmentation method due to its very large-scale.

4.3 Implementation details

The image-based representations are computed directly from the original skeletons without any data preprocessing technique, e.g., transforming the coordinate system, using a fixed number of frames, or removing noise. In this project, the MatConvNet toolbox Vedaldi2015MatConvNetC was used to design and train our deep learning networks. During the training phase, we use mini-batches of 128 images for ResNet-20, ResNet-32, ResNet-44, and ResNet-56 networks. For ResNet-110 network, we use mini-batches of 64 images. We initialize the weights randomly and train all networks for 120 epochs on a computer using Geforce GTX 1080 Ti GPU with 11GB RAM from scratch. The initial learning rate is set to 0.01 and is decreased to 0.001 at epoch 75. The last 45 epochs use a learning rate of 0.0001. The weight decay is set at 0.0001 and the momentum is 0.9.

5 Experimental Results and Analysis

This section reports our experimental results on the MSR Action3D dataset Li2010ActionRB and the NTU-RGB+D dataset Shahroudy2016NTURA . We compare the obtained classification rates with state-of-the-art approaches in the literature Li2010ActionRB ; vieira2012stop ; xia2012view ; Chaaraoui2013FusionOS ; Chen2013RealtimeHA ; Luo2013GroupSA ; Gowayyed2013HistogramOO ; Hussein:2013:HAR:2540128.2540483 ; qin2013gesture ; Liang2013ThreeDM ; Evangelidis2014SkeletalQH ; Theodorakopoulos2014PosebasedHA ; Gao2014HumanAR ; Vieira2014OnTI ; Chen2015ActionRF ; 7298714 ; Xu2015SpatioTemporalPM ; jin2017action ; Oreifej2013HON4DHO ; Yang2014SuperNV ; OhnBar2013JointAS ; Evangelidis2014SkeletalQH ; Misra2016ShuffleAL ; Cippitelli2016EvaluationOA ; Vemulapalli2014HumanAR ; Vemulapalli2016RollingRF ; 7298714 ; Shahroudy2016NTURA ; Luo2017UnsupervisedLO ; Liu2016SpatioTemporalLW . In addition, we also provide a detailed analysis about the computational efficiency of the proposed learning framework.

5.1 Result on MSR Action3D dataset

Experimental results on MSR Action3D dataset Li2010ActionRB are shown in Table 2. We achieved the best classification accuracy with the ResNet-32 model. More specifically, classification accuracies are 99.80% on AS1, 99.00% on AS2, and 99.80% on AS3. We obtained a total average accuracy of 99.53%. Our result outperforms many previous works Li2010ActionRB ; vieira2012stop ; xia2012view ; Chaaraoui2013FusionOS ; Chen2013RealtimeHA ; Luo2013GroupSA ; Gowayyed2013HistogramOO ; Hussein:2013:HAR:2540128.2540483 ; qin2013gesture ; Liang2013ThreeDM ; Evangelidis2014SkeletalQH ; Theodorakopoulos2014PosebasedHA ; Gao2014HumanAR ; Vieira2014OnTI ; Chen2015ActionRF ; 7298714 ; Xu2015SpatioTemporalPM ; jin2017action ; 10.1007/978-3-319-77383-4_35 ; Theodorakopoulos2014PosebasedHA ; 7298714 ; Chen2015ActionRF ; Xu2015SpatioTemporalPM ; jin2017action ; luvizon2017learning ; Luo2013GroupSA (see Table 3). Moreover, this result also indicates that the proposed ResNet architecture learns image features better than the original ResNet architecture iet:/content/conferences/10.1049/cp.2017.0154 . The learning curves of all proposed ResNets on AS1, AS2, AS3 are shown in Figure 9a, Figure 9b, and Figure 9c, respectively.

5.2 Result on NTU-RGB+D dataset

As shown in Table 4, the proposed learning framework reaches competitive results with an accuracy of 73.40% on Cross-Subject evaluation and an accuracy of 80.40% on Cross-View evaluation. The obtained results indicate that our method can deal with large intra-class variations and multiple viewpoints dataset as NTU-RGB+D Shahroudy2016NTURA . Table 5 provides a comparison with published studies Li2010ActionRB ; Liang2013ThreeDM ; xia2012view ; qin2013gesture ; hbali2017skeleton ; vieira2012stop ; Vieira2014OnTI ; Oreifej2013HON4DHO ; Evangelidis2014SkeletalQH ; Gao2014HumanAR ; Chen2013RealtimeHA ; Hussein:2013:HAR:2540128.2540483 ; Gowayyed2013HistogramOO ; Chaaraoui2013FusionOS . It is clear that our method surpasses many previous approaches in the same experimental conditions. The learning curves of all ResNet configurations on two evaluation settings are shown in Figure 9d and Figure 9e.

5.3 Convergence rate analysis

Figure 9 shows the learning curves of the five proposed deep learning networks on the MSR Action3D Li2010ActionRB and the NTU-RGB+D Shahroudy2016NTURA datasets. We can find that the training convergence rate of the networks are different on the two datasets. More specifically, on MSR Action3D dataset Li2010ActionRB , the proposed networks exhibit rapid convergence after 50 epochs at the beginning of the training process. Meanwhile, these networks can only start to converge after near 80 epochs on the NTU-RGB+D dataset Shahroudy2016NTURA . This phenomenon can be explained by the complexity of feature spaces generated by the two datasets. With more than 56,000 videos collected from 40 subjects for 60 action classes, it is clear that the NTU-RGB+D dataset Shahroudy2016NTURA is more complex than the MSR Action3D dataset Li2010ActionRB . This leads to deep learning networks needing more iterations to start convergence.

As shown in Table 2 and Table 4, the best recognition accuracy and configuration are highlighted in bold. More specifically, our experimental result on the MSR Action3D indicated that the baseline ResNet-32 achieved the best overall accuracy (99.53%). On the NTU RGB+D dataset, the ResNet-32 network was the best version on the Cross-View setting (80.40%). However, it worked worse than the ResNet-20 on the Cross-Subject setting. This is a weak aspect of the proposed approach. This limitation could be overcome by using ensemble learning techniques dietterich2000ensemble .

5.4 Analysis of training and prediction time

We take the AS1 subset of the MSR Action3D dataset Li2010ActionRB and the proposed ResNet-32 network for illustrating the computational efficiency of our deep learning framework. As shown in Figure 10, the proposed framework contains three main stages, including the encoding process from skeleton sequences into RGB images (Stage A), the supervised training stage (Stage B), and the prediction stage (Stage C). With the implementation in Matlab using MatconvNet toolbox Vedaldi2015MatConvNetC 444MatconvNet is an open source library and can be downloaded at address: http://www.vlfeat.org/matconvnet/. on a single NVIDIA GeForce GTX 1080 Ti GPU555More details about the GPU specification, please refer to: https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/., without parallel processing, we take on average $\textbf{4.15}\times\textbf{10}^{-3}$s per sequence during training stage. While the prediction time, including the time for encoding skeletons into RGB images and classification by the pre-trained ResNet, takes on average $\textbf{21.84}\times\textbf{10}^{-3}$s per sequence (see Table 6). These results verify the effectiveness of our proposed method, not only in terms of accuracy but also in terms of computational efficiency.

6 Conclusion and future work

In this paper, we have proposed a new 3D motion representation and an end-to-end deep learning framework based on ResNets He2016DeepRL for human action recognition from skeleton sequences. To this end, we transformed the 3D joint coordinates carried in skeleton sequences into RGB images via a color encoding process. We then designed and trained different D-CNNs based on the ResNet architecture for learning the spatial and temporal dynamics of human motion from image-coded representations. The experimental results on two well-established datasets, including the largest RGB-D dataset currently available, demonstrated that our method can achieve state-of-the-art performance whilst requiring a low computation cost for the training and prediction stages. For future work, we aim to extend this research by investigating a new skeleton encoding method, in which the Euclidean distances and orientation relations between joints will be exploited. In addition, we plan to build a better feature learning and classification framework by experimenting some new and potential architectures based on the idea of ResNet such as Inception Residual Networks (Inception-ResNets) article_inception_v4 or Densely Connected Convolutional Networks (DenseNet Huang2017DenselyCC ). In order to overcome the limitations of Kinect sensors when dealing with HAR in which the lighting conditions are available, we will also consider some new approaches for estimating human body key-points such as deep learning based approaches cao2017realtime ; Guler2018DensePose , KWE and KCE methods akbarizadeh2012new . We hope that our study will open a new door for the computer vision community on exploiting the potentials of very deep convolutional neural networks and skeletal data for 3D human action recognition.

Acknowledgment

This research was carried out at the Cerema Research Center (CEREMA) and Toulouse Institute of Computer Science Research (IRIT), Toulouse, France. Sergio A. Velastin is grateful for funding received from the Universidad Carlos III de Madrid, the European Union’s Seventh Framework Programme for Research, Technological Development and demonstration under grant agreement N. 600371, el Ministerio de Economia, Industria y Competitividad (COFUND2013-51509) el Ministerio de Educación, cultura y Deporte (CEI-15-17) and Banco Santander.