Two-Stream RNN/CNN for Action Recognition in 3D Videos

TL;DR

This paper presents a novel two-stream neural network architecture combining RNNs and CNNs for improved action recognition in 3D videos, achieving significant accuracy gains over previous methods.

Contribution

The paper introduces a combined RNN/CNN system with a voting approach that leverages long-term and recent information for better action recognition in 3D videos.

Findings

Achieved 14% higher recognition accuracy than state-of-the-art methods.

Demonstrated effectiveness of combining RNN and CNN in a voting framework.

Improved action recognition in 3D videos using a hybrid neural network approach.

Abstract

The recognition of actions from video sequences has many applications in health monitoring, assisted living, surveillance, and smart homes. Despite advances in sensing, in particular related to 3D video, the methodologies to process the data are still subject to research. We demonstrate superior results by a system which combines recurrent neural networks with convolutional neural networks in a voting approach. The gated-recurrent-unit-based neural networks are particularly well-suited to distinguish actions based on long-term information from optical tracking data; the 3D-CNNs focus more on detailed, recent information from video data. The resulting features are merged in an SVM which then classifies the movement. In this architecture, our method improves recognition rates of state-of-the-art methods by 14% on standard data sets.

Tables

Table 1. TABLE I: Comparison of testing accuracies on the NTU RGB+D dataset. The first 8 rows are the results reported from other papers. Rows 9 to 19 are results of the methods evaluated in this paper. The first 16 rows are skeleton-based methods. Our 2 Layer BI-GRU-BN-DP-H model outperforms other methods on both evaluation protocols. In addition, the feature fusion model further boosts accuracy by more than 13% on both settings.

Nr.	Method	cross subject	cross view
01	Skeleton Quads[9, 2]	38.62%	41.36%
02	Lie Group[10, 2]	50.08%	52.76%
03	FTP Dynamic Skeletons[11, 2]	60.23%	65.22%
04	HBRNN-L[3, 2]	59.07%	63.97%
05	Deep RNN[2]	56.29%	64.09%
06	Deep LSTM[2]	60.69%	67.29%
07	Part-aware LSTM[2]	62.93%	70.27%
08	ST-LSTM (Tree) + Trust Gate[4]	69.2%	77.7%
09	1 Layer RNN	18.74%	20.27%
10	1 Layer LSTM	60.99%	64.68%
11	1 Layer LSTM-BN	64.07%	71.86%
12	1 Layer LSTM-BN-DP	64.69%	73.48%
13	1 Layer GRU-BN-DP	65.21%	70.36%
14	1 Layer BI-GRU-BN-DP	64.78%	73.12%
15	2 Layer BI-GRU-BN-DP	66.21%	72.46%
16	2 Layer BI-GRU-BN-DP-H	70.70%	80.23%
17	3D-CNN[8]	79.75%	83.95%
18	Decision Fusion	82.05%	86.68%
19	Feature Fusion	83.74%	93.65%

Equations

$$\mathbf{h}_{t}=\sigma\left(\mathbf{W}\left(\begin{array}[]{ccc}\mathbf{x}_{t}\\ \mathbf{h}_{t-1}\\ \end{array}\right)\right)$$

$${\mathbf{y}}_t=\sigma \left(\mathbf{V}{\mathbf{h}}_t\right)$$

$$\mathbf{h}_{t}^{(l)}=\sigma\left(\mathbf{W}^{(l)}\left(\begin{array}[]{ccc}\mathbf{h}_{t}^{(l-1)}\\ \mathbf{h}_{t-1}^{(l)}\\ \end{array}\right)\right)$$

$${\mathbf{h}}_t^{(0)}:={\mathbf{x}}_t$$

$${\mathbf{y}}_t=\sigma \left(\mathbf{V}{\mathbf{h}}_t^{(L)}\right)$$

$$\left(\begin{array}[]{ccc}\mathbf{i}\\ \mathbf{f}\\ \mathbf{o}\\ \hat{\mathbf{c}}_{t}\\ \end{array}\right)=\left(\begin{array}[]{ccc}\mathrm{sigm}\\ \mathrm{sigm}\\ \mathrm{sigm}\\ \tanh\\ \end{array}\right)\mathbf{W}\left(\begin{array}[]{ccc}\mathbf{x}_{t}\\ \mathbf{h}_{t-1}\\ \end{array}\right)$$

$${\mathbf{c}}_t=\mathbf{f}\odot {\mathbf{c}}_{t-1}+\mathbf{i}\odot \widehat{{\mathbf{c}}_t}$$

$${\mathbf{h}}_t=\mathbf{o}\odot \tanh ({\mathbf{c}}_t)$$

$$\left(\begin{array}[]{ccc}\mathbf{z}\\ \mathbf{r}\\ \end{array}\right)=\left(\begin{array}[]{ccc}\mathrm{sigm}\\ \mathrm{sigm}\\ \end{array}\right)\left(\mathbf{W}\left(\begin{array}[]{ccc}\mathbf{x}_{t}\\ \mathbf{h}_{t-1}\\ \end{array}\right)\right)$$

$$\hat{\mathbf{h}_{t}}=\tanh\left(\mathbf{W}\left(\begin{array}[]{ccc}\mathbf{x}_{t}\\ r\odot\mathbf{h}_{t-1}\\ \end{array}\right)\right)$$

$${\mathbf{h}}_t=\mathbf{z}\odot {\mathbf{h}}_{t-1}+(1-\mathbf{z})\odot {\hat{\mathbf{h}}}_t$$

$$\mathrm{BN}(\mathbf{x};\gamma ,\beta )=\gamma \frac{\mathbf{x}-\hat{\mathrm{E}}(\mathbf{x})}{\sqrt{\widehat{\mathrm{Var}}(\mathbf{x})+ϵ}}+\beta$$

$$y(x_i)=\{\begin{array}{ll}{y}_r(x_i),& \text{if }{w}_r\times y_r(x_i)>w_c\times y_c(x_i)\\ y_c(x_i),& \text{otherwise}\end{array}$$

Full text

I Introduction

Recognition of human activity in 3D videos has received increasing attention since 2010[1, 2, 3, 4, 5, 6, 7]. Compared to 2D videos, 3D videos provide more spatial information and could be more informative. Action recognition with 3D videos is applied in different fields, such as health monitoring for patients, assisted living for disabled people, and robot perception and cognition.

Following this line of research, this paper proposes and applies novel deep-learning methods on what is currently the largest 3D action recognition dataset. Our results are compared with existing best approaches and are shown to be superior. Our proposed deep-learning methods consist mainly of three parts: a novel skeleton-based recurrent neural network structure, using a 3D-convolutional [8] neural network for RGB videos, and sketching a new two-stream fusion method to combine RNN and CNN. All methods are evaluated on the NTU RGB+D Dataset[2]. The dataset was published in 2016 and contains more than 56k action samples in four different modalities: RGB videos, depth map sequences, 3D skeletal data, and infrared videos. The dataset consists of 60 different action classes including daily, health-related, and mutual actions. In this paper, we use both the 3D skeletal data and RGB videos.

Traditional studies on 3D action recognition use different kinds of methods [1, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52] to compute handcrafted features, while deep-learning approaches [6, 53, 7, 54, 3, 2, 5, 4] are end-to-end trainable and can be applied directly on raw data. Focussing on the latter, for skeleton-based activity analysis, [3, 2, 5, 4] used different kinds of recurrent neural networks to acquire state-of-the-art performances on various of 3D action datasets. Du et al. [3] propose an hierarchical RNN, which is fed with manually divided five groups of the human skeleton, such as two hands, two legs, and one torso. Inspired from this, Shahroudy et al. [2] present a novel long short-term memory (LSTM) [55] cell, called part-aware LSTM, which is also fed with separated five parts of skeleton. Evolved from these two ideas, Zhu et al. [5] provide a novel deep RNN structure, which can automatically learn the co-occurrence, similar to grouping data into five human body parts, from skeleton data. Most recently, Liu et al. [4] propose a skeleton tree traversal algorithm and a new gating mechanism to improve robustness against noise and occlusion.

However, our proposed RNN structure makes a different contribution. Our method is inspired by recent normalization technologies [56] and a novel recurrent neuron mechanism [57]. With these advanced deep learning technologies embedded into our RNN structure, it can be trained with 13 times fewer iterations and for each iteration consumes 20% less computational time, compared to a normal RNN model with LSTM cells. Our contribution focuses more on making the network much easier to train, less inclined to overfitting, and deep enough to represent the data. More importantly, our proposed RNN structure outperforms all other skeleton-based methods on the largest 3D action recognition dataset.

To process RGB videos, our method is inspired by different kinds of convolutional neural networks[8, 58, 59, 60]. We use a 3D-CNN[8] model on the RGB videos of the NTU RGB+D dataset. We compare the results with proposed RNN models, and fuse their output.

To combine the RNN and CNN models, we propose two fusion structures: decision and feature fusion. The first is very simple to use, whereas the second provides a better performance. For decision fusion, we illustrate a voting method based on the confidence of the classifiers. For feature fusion, we propose a novel two-stream RNN/CNN structure, shown in Fig. 4, which combines temporal and spatial features from the RNN and CNN models and boost the performance by a significant margin. Our two-stream RNN/CNN structure outperforms the current state-of-the-art method [4] more than 14% on both cross subject and cross view settings.

To summarize, our contributions in this paper are:

• $\bullet$

A novel RNN structure is proposed, which converges 13 times faster during training and costs 20% less computational power at each forward pass, compared to a normal LSTM;

• $\bullet$

Two fusion methods are proposed to combine the proposed RNN structure and a 3D-CNN structure [8]. The decision fusion is easier to use, while the feature fusion has superior performance.

II Methodology

In this section, we first introduce the concept of recurrent neural networks and batch normalization, and then describe the proposed RNN structure: a deep bidirectional gated recurrent neural network with batch normalization and dropout. Afterwards, the applied 3D-CNN model and two-stream RNN/CNN fusion architectures, i.e., decision fusion and feature fusion, are introduced.

II-A Recurrent Neural Network

II-A1 Vanilla Recurrent Neural Network

Recurrent neural networks can handle sequence information with varied lengths of time steps. This transforms the input $\mathbf{X}$ to a internal hidden state $\mathbf{h}_{t}$ at each time step. The network passes the state $\mathbf{h}_{t}$ along with the next input $\mathbf{x}_{t+1}$ to the neuron, time step after time step. The neuron learns when to remember and forget information with nonlinear activation functions:

[TABLE]

[TABLE]

where $t\in\{1,\dots,T\}$ represents time steps, and $\sigma$ represents a nonlinear activation function such as a standard logistic sigmoid function $\mathrm{sigm}(x)=1/(1+\mathrm{e}^{-x})$ or a hyperbolic tangent function $\tanh(x)$.

Multiple layers of RNN can be stacked to increase the complexity:

[TABLE]

[TABLE]

[TABLE]

where $l\in\{1,...,L\}$ denotes the layer number.

In practice, a vanilla RNN does not remember information over a longer time; a problem which is related to the vanishing gradient problem.

II-A2 Long Short-Term Memory

This problem can be solved by LSTM [55] which stores information in gated cells at the neurons. This allows errors to be backpropagated through hundreds or thousands of time steps:

[TABLE]

[TABLE]

[TABLE]

where $\mathbf{i},\mathbf{f}$ and $\mathbf{o}$ denote input gate, forget gate, and output gate, respectively. $\hat{\mathbf{c}}_{t}$ represents new candidate values, which could be added to the cell state $\mathbf{c}_{t}$. We use $\odot$ for element-wise multiplication.

II-A3 Gated Recurrent Unit

An improvement to LSTM called gated recurrent unit (GRU) was proposed in [57]. GRU has a simpler structure and can be computed faster. The three gates from LSTM are combined into two gates, respectively updating gate $\mathbf{z}$ and resetting gate $\mathbf{r}$ in GRU. GRU also combines cell state $\mathbf{c}_{t}$ and hidden state $\mathbf{h}_{t}$ into one state $\mathbf{h}_{t}$. The mathematical description is as follows:

[TABLE]

[TABLE]

[TABLE]

where $\hat{\mathbf{h}}_{t}$ denotes new candidate state values.

II-A4 Bidirectional Recurrent Neural Network

A bidirectional RNN [61] performs a forward pass and a backward pass, which runs input data from $t=T$ to $t=1$ and from $t=1$ to $t=T$, respectively.

For classification, the output of an RNN $\mathbf{y}_{t}$ can be passed to a fully-connected layer with softmax activation functions; this allows us to interpret the output as a probability.

II-A5 Batch Normalization

To train a deep neural network, the internal covariate shift [56] slows down the training process. The internal covariate shift is the distribution of each layer’s input changes during training, because the parameters in the previous layer are changing. To reduce the internal covariate shift, we could whiten the layer activations, but this takes too much computation power. Batch normalization, a part of the neural network structure, approximates this process by standardizing the activations $\mathbf{x}$ using a statistical estimate of the mean $\widehat{\mathrm{E}}(\mathbf{x})$ and standard deviation $\widehat{\mathrm{Var}}(\mathbf{x})$ for each training mini-batch. It can be shown that

[TABLE]

where $\gamma\in\mathbb{R}^{d}$ and $\beta\in\mathbb{R}^{d}$ are scale and shift parameters for the activation $\mathbf{x}\in\mathbb{R}^{n\times d}$. With these, identity transformation for each activation could be presented. $\epsilon\in\mathbb{R}$ is a constant added as a regularization parameter for numerical stability. The division in Eq. (12) is performed element-wise. $\gamma$ and $\beta$ are learned during training and fixed during inference.

II-A6 Proposed RNN Structure

For skeleton-based action recognition tasks, the data set consists of the 3D coordinates of a number of body joints. We feed this information, together with action labels, to an RNN. This RNN network has two bidirectional layers, each of which consists of 300 GRU cells. After the recurrent layers follows the batch normalization layer, which standardizes the activations from the RNN layer. Then the normalized activations flow to the next fully-connected layer with 600 rectified linear unit (ReLU) [62] activation functions. During training, in each iteration the network randomly drops out 25% of the neurons between the batch normalization layer and the next fully-connected layer to reduce overfitting. Lastly, a softmax layer maps the compressed motion information (features) to 60 action classes. Fig. 1 shows the structure of this RNN network.

To highlight the improvements of this final proposed model, we compare our approach to simpler models. These models are a standard RNN; an LSTM-RNN; LSTM plus batch normalization (“LSTM-BN”), LSTM-BN with dropout (“LSTM-BN-DP”), GRU-BN-DP, and a bidirectional GRU-BN-DP which we call “BI-GRU-BN-DP”. All these models have one recurrent layer. The next complexity is adding an extra layer of hidden units to the last model (“2 layer BI-GRU-BN-DP”). Finally, we add another fully-connected layer on top, before the softmax layer, and call this model “2 layer BI-GRU-BN-DP-H”. Sec. III discusses the results of all models.

II-B Convolutional Neural Network

To process RGB videos, we choose to use the 3D-CNN model from[8], as it shows promising performances on 2D video action recognition tasks. We believe that 3D convolution nets are more suitable for learning features from videos than 2D convolution nets.

2D convolution generates a series of 2D feature maps from images. Inspired by this, a 3D convolution processes frame clips, where the third dimension is time step, which results in a series of 3D feature volumes, as shown in Fig. 2. This compressed representation contains spatiotemporal information from the video clips. To learn a rich amount of features, multiple layers of convolution and max-pooling operations are stacked into one model.

To be specific, the 3D-CNN model [8], which we choose, has five convolutional groups, each group has one or two convolutional layers and one max-pooling layer, two fully-connected layers, and one softmax output layer. The details of this model are presented in Fig. 3.

We finetune this model with pretrained parameters [8] on Sports-1M Dataset, which has approximately one million YouTube videos. This reduces overfitting and demands less training time on the current dataset.

II-C Two-stream RNN/CNN

As having the proposed RNN structure for the skeleton data and the 3D-CNN model for the RGB videos, we want to combine the strengths of RNN and CNN nets. To improve the performance, we propose two fusion models, decision fusion and feature fusion.

II-C1 Decision Fusion

In the case of decision fusion, we use a simple but efficient voting method, inspired by majority voting. As a result of having only two classifiers, we cannot apply majority voting. Instead, the fusion method predicts based on voting confidence.

We first split the dataset into training, validation and testing. The same training set is used to train the RNN and CNN nets. The validation set is then used to find the best parameters, trust weights $w_{r}$ and $w_{c}$ for the voting method. We initialize the trust weights with equal values, which means $w_{r}=1.00$ and $w_{c}=1.00$ for both RNN and CNN classifiers. Afterwards, we compare the confidences, which are the highest probabilities of softmax output from both classifiers for each prediction. The more confident one wins:

[TABLE]

where $y(x_{i})$ is the fused prediction for sample $x_{i}$; $y_{r}$ and $y_{c}$ denote RNN and CNN prediction, respectively.

Based on this concept, we develop a way to fuse the predictions from RNN and CNN. We evaluate the performance with the validation dataset and search for the best trust weights for decision fusion. Having only two parameters $w_{r}$ and $w_{c}$, only little tuning is needed.

II-C2 Feature Fusion

Another way to combine these two neural networks is feature fusion.

We first train the RNN and CNN models on the training dataset. As in training, neural nets can learn discriminant information from raw data. Thus, we use the trained RNN model to extract temporal features from 3D skeleton data and use the trained CNN model to learn spatiotemporal features from RGB videos. Both features come from the first fully-connected layer in each model. The features are concatenated, L2 normalized, and eventually, fed to a linear SVM classifier.

The SVM parameter $C$ is found using the validation dataset. Then the model is tested on the test set. The feature fusion structure for two streams of RNN and CNN features is presented in Fig. 4.

III Experiments

The models introduced in the previous section are evaluated in the experiments. The dataset is first introduced in this section, then the setups and parameter settings for the experiments are illustrated. We compare the results of the proposed models with the current best methods. In the end, we analyze and discuss the problems related to deep learning methods for 3D action recognition.

III-A NTU RGB+D Dataset [2]

The proposed approaches are evaluated on the NTU RGB+D dataset[2], which we know as the current largest publicly available 3D action recognition dataset. The dataset consists of more than 56k action videos and 4 million frames, which were collected by 3 Kinect V2 cameras from 40 distinct subjects, and divided into 60 different action classes including 40 daily (drinking, eating, reading, etc.), 9 health-related (sneezing, staggering, falling down, etc.), and 11 mutual (punching, kicking, hugging, etc.) actions. It has four major data modalities provided by the Kinect sensor: 3D coordinates of 25 joints for each person (skeleton), RGB frames, depth maps, and IR sequences. In this paper, we use the first two modalities, since they are the two most informative modalities.

The large intra-class and view point variations make this dataset challenging. However, the large amount of action samples makes it highly suitable for data-driven methods.

This dataset has two standard evaluation criteria [2]. The first one is a cross-subject test, in which half of the subjects are used for training and the other half are used for testing. The second one is a cross-view test, in which two viewpoints are used for training and one is excluded for evaluation.

III-B Implementation details

In our experiments, the implementation consists of RNN, CNN, and Fusion. For all these models we use the same training, validation and testing splits. The validation set is composed of 10% of the subjects in the training set in [2]. The remaining subjects in the training set [2] make up the training set.

III-B1 RNN Implementation

In the RNN experiments, we have two human skeletons as input, each skeleton has 25 3D coordinates. Since the longest time step is 300, we pad all the action sequences to a length of 300. The dimension of each action sample is 300 (time steps) $\times$ 150 (coordinates).

We use TensorFlow[63] with TFlearn[64] and run the experiments on either one NVIDIA GTX 1080 GPU or one NVIDIA GTX TITAN X GPU. We train the network using RMSprop [65] optimizer and set learning rate as 0.001, decay as 0.9, and momentum as 0. We train the network from scratch using mini-batches of 1000 sequences for one-layer models and use mini-batches of 650 sequences for two-layer models. For all RNN nets, we use 300 neurons for each single-directional layer, double the amount of neurons for bidirectional layers, and we use a 75% keep probability for dropout. For batch normalization, we initialize $\gamma$ as 1.0, $\beta$ as 0.0, and set $\epsilon$ as $1\times 10^{-5}$. The estimated means and variances are fixed during inference.

As a comparison, the mentioned parameters are the same for all proposed RNN models, only the structure changes.

III-B2 CNN Implementation

We use the 3D-CNN model[8] in Caffe[66] and train it on RGB frames from the NTU RGB+D dataset, with pretrained parameters[8] from the Sport1M dataset. From RGB videos, we extract the frames, crop and resize them from $1920\times 1080$ pixels to $320\times 240$ pixels [8]. Videos are split into non-overlapped 16-frame clips.

We refer to the input of CNN model as a size of $c\times t\times h\times w$, where $c$ is the number of channels, $t$ is the number of time steps, $h$ and $w$ are the height and width of the frame, respectively. The network takes video clips as input and predicts the 60 action labels which belong to the 60 different actions. It further resizes the input frames to 128$\times$171 pixel resolution. The input dimensions are 3$\times$16$\times$128$\times$171 pixel. During training we use jittering on the input clips by random cropping them into 3$\times$16$\times$112$\times$112 pixel. We fine-tune the network with stochastic gradient descent optimizer using mini-batches of 44 clips, with initial learning rate of 0.0001. The learning rate is then reduced by half, when no training progress was observed [65]. The training stopped after around 20 epochs.

For video-based prediction, the model averages the predictions over all 16-frame clips split from the same video and provides the final prediction for the input video. A similar idea is applied for extracting features from fc-6 layer, which averages the 4096-dimensional feature vectors over all clips in the same video, resulting in one 4096-dimensional vector for each video.

III-B3 Fusion Implementation

We fuse the best RNN structure, 2 layer BI-GRU-BN-DP-H, with the 3D-CNN model, first using decision fusion, then using feature fusion.

For decision fusion, we first extract the softmax output, then search for the fusion parameters, trust weight $w_{r}$ and $w_{c}$ for RNN and CNN from the validation split. The parameters are $w_{r}=1.00$ and $w_{c}=2.88$ for the cross subject setup, and $w_{r}=1.00$ and $w_{c}=3.02$ for the cross view setup.

For feature fusion, we extract the RNN features (600 dimensions) from the fully-connected layer, and extract CNN features (4096 dimensions) from the fc-6 layer [8]. We then concatenate them into one feature array (4,696 dimensions) and apply L2 normalization. In the end, we have normalized RNN/CNN features from training, validation, and testing splits. We use training and validation splits to find the optimal value of $C$ for linear SVM[67] model. For both cross-subject and cross-view setups, we find that $C=8.0$ gives the best validation accuracy.

Among all the models in this paper, feature fusion model shows the best testing results. We refer to this model as a two-stream RNN/CNN structure as shown in Fig. 4.

III-C Experimental Results and Analysis

The evaluation results are shown in Tab. I. The first 16 rows are skeleton-based methods. The 3D-CNN model (17th row) uses RGB videos as input. The decision fusion (18th row) and feature fusion (19th row) models use the best RNN structure, which is the 2 Layer BI-GRU-BN-DP-H (16th row), and the 3D-CNN (17th row) model.

Tab. I shows that our RNN structure, the 1 Layer LSTM-BN, already outperforms the baseline method part-aware LSTM reported in [2] because batch normalization improves the LSTM model. Adding a dropout procedure reduces overfitting and further improves the results (rows 11, 12). From rows 12 and 13 we can see that the performances of LSTM and GRU cells are similar[68]. GRU is better in the cross-subject test and LSTM is better in the cross-view test. On the other hand, GRU is faster than LSTM both in computational speed and converge rate. As presented in Fig. 5 left, for 1k training steps, the same model performs 5.42% more accurately and takes 20% less computational time when using GRU cells than when using LSTM cells.

The addition of the extra fully-connected layer brings another significant improvement. This increases the complexity of the neural network, which helps the model capture more inherent features from the 3D skeleton data [69]. The recurrent layers before the fully-connected layer can be seen as a temporal feature extractor, which compact input information (dimension $300\times 150$) into 600 dimensions. The latter part of the RNN structure can be considered as a classifier learning to map these 600-dimensional features to 60 different action categories. Altogether, our novel RNN model, 2 Layer BI-GRU-BN-DP-H, outperforms all the other skeleton-based models including ST-LSTM (Tree traversal) + Trust Gate [4].

Then, we use the RGB video data to train the 3D-CNN model. We use the voting method based on confidence to fuse the 2 Layer BI-GRU-BN-DP-H and 3D-CNN model. In the next step, we utilize a linear SVM [8] to fuse the fc-6 features from the CNN and the fc features from the RNN. This further improves results by over 13% in comparison to our best RNN, and by more than 14% compared to literature models [2, 4]. This boosting is due to the features from RNN and CNN model being highly complementary. The RNN model uses 50 3D coordinates for two human bodies over 300 time steps, and learns to find the long-term motion pattern. Whereas the CNN model has 2D RGB frames, which additionally has spatiotemporal information about objects, such as cups, pens, and books. However, the CNN model can only memorize information for 16 time steps long—longer memorization is prohibited by GPU memory limitations. These facts make the features from RNN and CNN model highly complementary, as the testing results show in row 16, 17, and 19 in Tab. I.

III-D Discussion

To better analyze and improve the performance of the model, we take a closer look at actions that are highly confusing to the two-stream RNN/CNN structure. As presented in Fig. 6 and 7, such action pairs include reading vs. writing, putting on a shoe vs. taking off a shoe, and rubbing two hands vs. clapping. These actions are shown in a video at https://www.youtube.com/watch?v=G0PXKCEgIoA. Fig. 8 shows some classified action samples.

There could be several reasons for this observation. First, these actions are sometimes inherent confusing. Secondly, there are flaws in the data. Kinect depth information, from which the NTU skeleton data is created, is quite noisy [70, 71]. Correspondingly, the 3D skeleton data used in our RNNs are also quite noisy [4]. RGB videos data are more accurate and stable, but single frames carry no 3D information. Thirdly, the 3D-CNN model[8] is trained with small video clips, which are 16 time steps long. The CNN model is adapted to find only short-term temporal features in these clips. As GPU memory and computing power increase, the model could also be adapted to find long-term temporal features in each whole video. Lastly, although, the RNN model can memorize the whole action sequences and give final predictions, it has no information about the appearances and movements of surrounding objects, which could be discriminative information for the classification task.

IV Conclusion and Future Work

In this paper, we propose a novel RNN structure for 3D skeletons that achieves state-of-the-art performance on the largest 3D action recognition dataset. The proposed RNN model can also be trained 13 times faster and saves 20% computational power on each training step. Additionally, the RGB videos from the same dataset are used to finetune a 3D-CNN model. In the end, an efficient fusion structure, two-stream RNN/CNN, is introduced to fuse the capabilities of both RNN and CNN models. The results of this method are 13% higher than using the proposed RNN alone, and 14% higher than the best published result in the literature. In the future, we want to consider using the other sensor modalities such as depth maps and IR sequences and see what is the best architecture to fuse all these modalities.