A New Representation of Skeleton Sequences for 3D Action Recognition

TL;DR

This paper introduces a novel representation of skeleton sequences for 3D action recognition, transforming sequences into clips based on cylindrical coordinates and employing deep neural networks to learn spatial and temporal features.

Contribution

The paper proposes a new skeleton sequence representation using cylindrical coordinate-based clips and a multi-task learning framework for improved 3D action recognition.

Findings

Effective representation of skeleton sequences for action recognition

Deep neural networks successfully learn spatial and temporal features

Significant improvement over existing methods

Abstract

This paper presents a new method for 3D action recognition with skeleton sequences (i.e., 3D trajectories of human skeleton joints). The proposed method first transforms each skeleton sequence into three clips each consisting of several frames for spatial temporal feature learning using deep neural networks. Each clip is generated from one channel of the cylindrical coordinates of the skeleton sequence. Each frame of the generated clips represents the temporal information of the entire skeleton sequence, and incorporates one particular spatial relationship between the joints. The entire clips include multiple frames with different spatial relationships, which provide useful spatial structural information of the human skeleton. We propose to use deep convolutional neural networks to learn long-term temporal information of the skeleton sequence from the frames of the generated clips, and then use a Multi-Task Learning Network (MTLN) to jointly process all frames of the generated clips in parallel to incorporate spatial structural information for action recognition. Experimental results clearly show the effectiveness of the proposed new representation and feature learning method for 3D action recognition.

Tables

Table 1. Table 1: Performance on the NTU RGB+D dataset.

Methods	Accuracy
	Cross Subject	Cross View
Lie Group [42]	50.1%	52.8%
Skeletal Quads [7]	38.6%	41.4%
Dynamic Skeletons [16]	60.2%	65.2%
Hierarchical RNN [6]	59.1%	64.0%
Deep RNN [37]	59.3%	64.1%
Deep LSTM [37]	60.7%	67.3%
Part-aware LSTM [37]	62.9%	70.3%
ST-LSTM [26]	65.2%	76.1%
ST-LSTM + Trust Gate [26]	69.2%	77.7%
Coordinates + FTP	61.06%	74.64%
Frames + CNN	75.73%	79.62%
Clips + CNN + Concatenation	77.05%	81.11%
Clips + CNN + Pooling	76.37%	80.46%
Clips + CNN + MTLN	79.57%	84.83%

Table 2. Table 2: Performance on the SBU kinect interaction dataset.

Methods	Accuracy
\pbox3cm Raw Skeleton [53]	49.7%
Joint Feature [18]	86.9%
CHARM [25]	83.9%
Hierarchical RNN [6]	80.35%
Deep LSTM [54]	86.03%
Deep LSTM + Co-occurrence [54]	90.41%
ST-LSTM [26]	88.6%
ST-LSTM + Trust Gate [26]	93.3%
Coordinates + FTP	79.75%
Frames + CNN	90.88%
Clips + CNN + Concatenation	92.86%
Clips + CNN + Pooling	92.26%
Clips + CNN + MTLN	93.57%

Table 3. Table 3: Performance on the CMU dataset.

Methods	Accuracy
	CMU subset	CMU
Hierarchical RNN [6]	83.13%	75.02%
Deep LSTM [54]	86.00%	79.53%
Deep LSTM + Co-occurrence [54]	88.40%	81.04%
Coordinates + FTP	83.44%	73.61%
Frames + CNN	91.53%	85.36%
Clips + CNN + Concatenation	90.97%	85.76%
Clips + CNN + Pooling	90.66%	85.56%
Clips + CNN+ MTLN	93.22%	88.30%

Equations

$$\begin{array}[]{c}\mathbf{y^{k}}=\left[y_{1}^{k},\cdots,y_{j}^{k},\cdots,y_{14}^{k}\right]\\ \\ y_{j}^{k}=\dfrac{1}{14}\sum\limits_{i=1}^{14}\max(0,x_{i,j}^{k})\\ \end{array}$$

$$\begin{array}[]{ll}\ell_{k}(\mathbf{z_{k}},\mathbf{y})&=\sum\limits_{i=1}^{m}y_{i}\left(-log\left(\dfrac{\exp z_{ki}}{\sum\limits_{j=1}^{m}\exp z_{kj}}\right)\right)\\ &=\sum\limits_{i=1}^{m}y_{i}\left(log\left(\sum\limits_{j=1}^{m}\exp z_{kj}\right)-z_{ki}\right)\end{array}$$

$$\mathcal{L}(Z,\mathbf{y})=\sum _{k=1}^4{\ell }_k({\mathbf{z}}_{\mathbf{k}},\mathbf{y})$$

Full text

1 Introduction

3D skeleton data records the trajectories of human skeleton joints and is robust to illumination changes and invariant to camera views [14]. With the prevalence of highly-accurate and affordable devices, action recognition based on 3D skeleton sequence has been attracting increasing attention [49, 42, 6, 37, 54, 26, 22, 46, 19]. In this paper, we focus on skeleton-based 3D action recognition.

To recognize a video action, the temporal information of the sequence needs to be exploited to understand the dynamics of human postures [29, 9, 44, 8, 20]. For skeleton data, the spatial structure of the human skeleton is also an important clue for action recognition [54]. Each skeleton sequence provides only the trajectory of human skeleton joints. The time series of the joints can be used in recurrent neural networks (RNNs) with Long-Short Term Memory (LSTM) neurons [11, 12] to explore the spatial structure and temporal structure of the skeleton sequence for action recognition [6, 41, 54, 37, 26]. Although LSTM networks are designed to explore the long-term temporal dependency problem, it is still difficult for LSTM to memorize the information of the entire sequence with many timesteps [48, 13]. In addition, it is also difficult to construct deep LSTM to extract high-level features [35, 30].

Convolutional neural networks (CNNs) [24] nowadays have achieved great success in image classification [2, 3, 23, 38, 39, 50, 21]. However, for video action recognition, it lacks the capacity to model the long-term temporal dependency of the entire video [45]. In this paper, instead of directly exploring the long-term temporal information from the skeleton sequences, we first represent the skeleton sequences as clips consisting of only a few frames. With the generated clips, the long-term temporal structure of the skeleton sequence can be effectively learned by using deep CNNs to process the frame images of the generated clips. In addition, the spatial structural information of the human skeleton can be exploited from the entire clips.

More specifically, for each skeleton sequence, we generate three clips corresponding to the three channels of the cylindrical coordinates of the skeleton sequence. Each clip consists of four frames, which are generated by computing the relative positions of the joints to four reference joints. Each frame of the clips describes the temporal information of the entire skeleton sequence, and includes one particular spatial relationship between the joints. The entire clips aggregate multiple frames with different spatial relationships, providing important information of the spatial structure of the skeleton joints.

Since the temporal information of a skeleton sequence is incorporated in the frames of the generated clips, the long-term temporal structure of the skeleton sequence can be learned by extracting features from the frames of the generated clips. More specifically, each frame of the generated clips is fed to a deep CNN to extract a CNN feature. Then the three CNN features of the three clips at the same time-step (See Figure 1) are concatenated into one feature vector. Consequently, four feature vectors are extracted from all the time-steps. Each feature vector represents the temporal information of the skeleton sequence and one particular spatial relationship between the joints. The feature vectors of different time-steps represent different spatial relationships with intrinsic relationships among them. This paper proposes to utilize the intrinsic relationships among different feature vectors for action recognition using a Multi-Task Learning Network (MTLN). Multi-task learning aims at improving the generalization performance by jointly training multiple related tasks and utilizing their intrinsic relationships [1]. In the proposed MTLN, the classification of each feature vector is treated as a separate task, and the MTLN jointly learns multiple classifiers each from one feature vector and outputs multiple predictions, each corresponding to one task. All the feature vectors of the same skeleton sequence have the same label as the skeleton sequence. During training, the loss value of each task is individually computed using its own class scores. Then the loss values of all tasks are summed up to define the total loss of the network which is then used to learn the network parameters. During testing, the class scores of all tasks are averaged to form the final prediction of the action class. Multi-task learning simultaneously solves multiple tasks with weight sharing, which can improve the performance of individual tasks [1].

The main contributions of this paper are summarized as follows. (1) We propose to transform each skeleton sequence to a new representation, i.e., three clips, to allow global long-term temporal modelling of the skeleton sequence by using deep CNNs to learn hierarchical features from frame images. (2) We introduce a MTLN to process all the CNN features of the frames in the generated clips, thus to learn the spatial structure and the temporal information of the skeleton sequence. The MTLN improves the performance by utilizing intrinsic relationships among different frames of the generated clips. Our experimental results demonstrate that MTLN performs better than concatenating or pooling the features of the frames (See Section 4.3). (3) The proposed method achieves the state-of-the-art performance on three skeleton datasets, including the large scale NTU RGB+D dataset [37].

2 Related Works

In this section, we cover the relevant literature of skeleton-based action recognition methods using hand-crafted features or using deep learning networks.

Hand-crafted Features   In [17], the covariance matrices of the trajectories of the joint positions are computed over hierarchical temporal levels to model the skeleton sequences. In [43], the pairwise relative positions of each joint with other joints are computed to represent each frame of the skeleton sequences, and Fourier Temporal Pyramid (FTP) is used to model the temporal patterns. In [51], the pairwise relative positions of the joints are also used to characterize posture features, motion features, and offset features of the skeleton sequences. Principal Component Analysis (PCA) is then applied to the normalized features to compute EigenJoints as representations. In [49], histograms of 3D joint locations are computed to represent each frame of the skeleton sequences, and HMMs are used to model the temporal dynamics. In [42], the rotations and translations between various body parts are used as representations, and a skeleton sequence is modelled as a curve in the Lie group. The temporal dynamics are modelled with FTP.

**Deep Learning Methods **   In [6], the skeleton joints are divided into five sets corresponding to five body parts. They are fed into five LSTMs for feature fusion and classification. In [54], the skeleton joints are fed to a deep LSTM at each time slot to learn the inherent co-occurrence features of skeleton joints. In [37], the long-term context representations of the body parts are learned with a part-aware LSTM. In [26], both the spatial and temporal information of skeleton sequences are learned with a spatial temporal LSTM. A Trust Gate is also proposed to remove noisy joints. This method achieves the state-of-the-art performance on the NTU RGB+D dataset [37].

3 Proposed Method

An overall architecture of the proposed method is shown in Figure 1. The proposed method starts by generating clips of skeleton sequences. A skeleton sequence of any length is transformed into three clips each consisting of several gray images. The generated clips are then fed to a deep CNN model to extract CNN features which are used in a MTLN for action recognition.

3.1 Clip Generation

Compared to RGB videos which consist of multiple frame images, skeleton sequences only provide the trajectories of the 3D coordinates. This paper proposes to transform the original skeleton sequence to a collection of clips each consisting of several images, thus to allow spatial temporal feature learning using deep neural networks. Intuitively, one could represent the content of each frame of the skeleton sequence as an image to generate a video. However, if the skeleton sequence has many frames, this method will result in a long video of which the temporal dynamics will be difficult to learn. In addition, each frame of the generated video will also be very sparse as the number of the skeleton joints is small. To overcome this problem, we propose to represent the temporal dynamics of the skeleton sequence in a frame image, and then use multiple frames to incorporate different spatial relationships between the joints. An advantage of this method is that for any skeleton sequence of any length, the generated clips contain the same number of frames and the long-term temporal information of the original skeleton sequence can be effectively captured with the powerful CNN representations of the frame images in the generated clips.

As shown in Figure 2, for a skeleton sequence, the skeleton joints of each frame are first arranged as a chain by concatenating the joints of each body part. Considering that the relative positions between joints provide more useful information than their absolute locations (e.g., the relative location of the hand to the shoulder in “pushing”), four reference joints, namely, the left shoulder, the right shoulder, the left hip and the right hip, are respectively used to compute relative positions of the other joints, thus to incorporate different spatial relationships between joints and provide useful structural information of the skeleton. These four joints are selected as reference joints due to the fact that they are stable in most actions. They can thus reflect the motions of the other joints. Although the base of the spine is also stable, it is close to the left hip and the right hip. It is therefore discarded to avoid information redundancy. By combing the relative joints of all the frames, four 2D arrays with dimension $(m-1)\times t$ are generated ($m$ is the number of skeleton joints in each frame and $t$ is the number of frames of the skeleton sequence). The relative positions of joints in the 2D arrays are originally described with 3D Cartesian coordinates. Considering that the cylindrical coordinates are more useful to analyse the motions as each human body utilizes pivotal joint movements to perform an action, the 3D Cartesian coordinates are transformed to cylindrical coordinates in the proposed representation of skeleton sequences. The cylindrical coordinates have been used to extract view-invariant motion features for action recognition in [47]. The four 2D arrays corresponding to the same channel of the 3D cylindrical coordinates are transformed to four gray images by scaling the coordinate values between 0 to 255 using a linear transformation. A clip is then constructed with the four gray images. Consequently, three clips are generated from the three channels of the 3D coordinates of the four 2D arrays.

3.2 Clip Learning

Each frame of the generated clips describes the temporal dynamics of all frames of the skeleton sequence and one particular spatial relationship between the skeleton joints in one channel of the cylindrical coordinates. Different frames of the generated clip describe different spatial relationships and there exists intrinsic relationships among them. A deep CNN is first leveraged to extract a compact representation from each frame of the generated clips to exploit the long-term temporal information of the skeleton sequence. Then the CNN features of all frames of the generated clips are jointly processed in parallel using multi-task learning, thus to utilize their intrinsic relationships to learn the spatial temporal information for 3D action recognition.

3.2.1 Temporal Pooling of CNN Feature Maps

To learn the features of the generated clips, a deep CNN is firstly employed to extract a compact representation of each frame of the clips. Since each frame describes the temporal dynamics of the skeleton sequence, the spatial invariant CNN feature of each frame could thus represent the robust temporal information of the skeleton sequence.

Given the generated clips, the CNN feature of each frame is extracted with the pre-trained VGG19 [38] model. The pre-trained CNN model is leveraged as a feature extractor due to the fact that the CNN features extracted by the models pre-trained with ImageNet [34] are very powerful and have been successfully applied in a number of cross-domain applications [5, 10, 33, 15]. In addition, current skeleton datasets are either too small or too noisy to suitably train a deep network. Although the frames of the generated clips are not natural images, they could still be fed to the CNN model pre-trained with ImageNet [34] for feature extraction. The similarity between a natural image and the generated frames is that both of them are matrices with some patterns. The CNN models trained on the large image dataset can be used as a feature extractor to extract representations of the patterns in matrices. The learned representations are generic and can be transferred to novel tasks from the original tasks [52, 27].

The pre-trained VGG19 [38] model contains 5 sets of convolutional layers conv1, conv2, …, conv5. Each set includes a stack of 2 or 4 convolutional layers with the same kernel size. Totally there are 16 convolutional layers and three fully connected layers in the network. Although deep neural networks are capable of learning powerful and generic features which can be used in other novel domains, the features extracted from the different layers have different transferability. Particularly, the features in earlier layers are more generic, while in later layers, the features are more task-specific, which largely rely on the original classes and dataset. The features of the later layers are thus less suitable than those of the earlier layers to transfer to other domains [52, 27]. Therefore, this paper adopts a compact representation that is derived from the activations of the convolutional layer to exploit the temporal information of a skeleton sequence. The feature maps in the convolutional layer have been successfully applied for action recognition and image retrieval [31, 32]. Specifically, the last 3 convolutional layers and fully connected layers of the network are discarded. Each frame image of the three clips is scaled to $224\times 224$, and is then duplicated three times to formulate a color image, so that it can be fed to the network. The output of the convolutional layer conv5_1 is used as the representation of the input frame, which is a 3D tensor with size $14\times 14\times 512$, i.e., 512 feature maps with size $14\times 14$.

The rows of the generated frame correspond to different frames of a skeleton sequence. The dynamics of the row features of the generated image therefore represents the temporal evolution of the skeleton sequence. Meanwhile, the activations of each feature map in the conv5_1 layer are the local features corresponding to the local regions in the original input image [31]. The temporal information of the sequence can thus be extracted from the row features of the feature maps. More specifically, the feature maps are processed with temporal mean pooling with kernel size $14\times 1$, i.e., the pooling is applied over the temporal, or row dimension, thus to generate a compact fusion representation from all temporal stages of the skeleton sequence. Let the activation at the $i^{th}$ row and the $j^{th}$ column of the $k^{th}$ feature map be $x_{i,j}^{k}$. After temporal mean pooling, the output of the $k^{th}$ feature map is given by:

[TABLE]

The outputs of all feature maps (512) are concatenated to form a 7168D ($14\times 512=7168$) feature vector, which represents the temporal dynamics of the skeleton sequence in one channel of the cylindrical coordinates.

3.2.2 Multi-Task Learning Network (MTLN)

As shown in Figure 1(e), the three 7168D features of the three clips at the same time-step are concatenated to form a feature vector, generating four feature vectors in total. Each feature vector represents the temporal dynamics of the skeleton sequence and includes one particular spatial relationship between the joints in one of three cylindrical coordinates. The four feature vectors have intrinsic relationships between each other. An MTLN is then proposed to jointly process the four feature vectors to utilize their intrinsic relationships for action recognition. The classification of each feature vector is treated as a separate task with the same classification label of the skeleton sequence.

The architecture of the network is shown in Figure 1(f). It includes two fully connected (FC) layers and a Softmax layer. Between the two FC layers there is a rectified linear unit (ReLU) [28] to introduce an additional non-linearity. Given the four features as inputs, the MTLN generates four frame-level predictions, each corresponding to one task. During training, the class scores of each task are used to compute a loss value. Then the loss values of all tasks are summed up to generate the final loss of the network used to learn the network parameters. During testing, the class scores of all tasks are averaged to form the final prediction of the action class. The loss value of the $k^{th}$ task ($k=1,\cdots,4$) is given by Equation 2.

[TABLE]

where $\mathbf{z_{k}}$ is the vector fed to the Softmax layer generated from the $k^{th}$ input feature, $m$ is the number of action classes and $y_{i}$ is the ground-truth label for class $i$. The final loss value of the network is computed as the sum of the four individual losses, as shown below in Equation 3:

[TABLE]

where $Z=[\mathbf{z_{1}},\cdots,\mathbf{z_{4}}]$.

4 Experiments and Analysis

The proposed method is tested on three skeleton action datasets: NTU RGB+D dataset [37], SBU kinect interaction dataset [53] and CMU dataset [4].

The main ideas of the proposed method **Clips + CNN + MTLN ** are 1) generating three clips (each clip consists of four frames) from a skeleton sequence, 2) using CNNs to learn global long-term temporal information of the skeleton sequence from each frame of the generated clips, and 3) using MTLN to jointly train the CNN features of the four frames of the clips to incorporate the spatial structural information for action recognition.

We also conducted the following baselines to demonstrate the advantages of the proposed method:

Coordinates + FTP  In this baseline, the Fourier Temporal Pyramid (FTP) [43] is applied to the 3D coordinates of the skeleton sequences to extract temporal features for action recognition. This baseline is used to show the benefits of using CNNs for long-term temporal modelling of the skeleton sequences.

Frames + CNN  In this baseline, the CNN features of single frames instead of the entire generated clips are used for action recognition. In other words, only one feature vector shown in Figure 1(e) is used to train a neural network for classification. Thus the loss value of the network is given by Equation 2. The average accuracy of the four features is provided. This baseline is used to show the benefits of using the entire generated clips to incorporate the spatial structural information for action recognition.

Clips + CNN + Concatenation  In this baseline, the CNN features of all frames of the generated clips are concatenated before performing action recognition. In other words, the four feature vectors shown in Figure 1(e) are concatenated and then fed to a neural network for classification. This baseline is used to show the benefits of using MTLN to process the features of the entire clips in parallel.

Clips + CNN + Pooling  In this baseline, max pooling is applied to the CNN features of all frames of the generate clips before performing action recognition. Same as Clips + CNN + Concatenation, this baseline is also used to show the benefits of using MTLN.

4.1 Datasets

NTU RGB+D Dataset [37]   To the best of our knowledge, this dataset is so far the largest skeleton-based human action dataset, with more than 56000 sequences and 4 million frames. There are 60 classes of actions performed by 40 distinct subjects, including both one-person daily actions (e.g., clapping, reading, writing) and two-person interactions (e.g., handshaking, hug, pointing). These actions are captured by three cameras, which are placed at different locations and view points. In total, there are 80 views for this dataset. In this dataset, each skeleton has 25 joints. The 3D coordinates of the joints are provided. Due to the large view point, intra-class and sequence length variations, the dataset is very challenging.

SBU Kinect Interaction Dataset [53]   This dataset was collected using the Microsoft Kinect sensor. It contains 282 skeleton sequences and 6822 frames. In this dataset, each frame contains two persons performing an interaction. The interactions include approaching, departing, kicking, punching, pushing, hugging, shaking hands and exchanging. There are 15 joints for each skeleton. This dataset is challenging due to the fact that the joint coordinates exhibit low accuracy [53].

CMU Dataset [4]   This dataset contains 2235 sequences and about 1 million frames. For each skeleton, the 3D coordinates of 31 joints are provided. The dataset has been categorized into 45 classes [54]. All of the actions are performed by only one person. The dataset is very challenging due to the large sequence length variations and intra-class diversity.

4.2 Implementation Details

For all datasets, the clips are generated with all frames of the original skeleton sequence without any pre-processing such as normalization, temporal down-sampling or noise filtering. The proposed method was implemented using the MatConvNet toolbox [40]. The number of the hidden unit of the first FC layer is set to 512. For the second FC layer (i.e., the output layer), the number of the unit is the same as the number of the action classes in each dataset. The network is trained using the stochastic gradient descent algorithm. The learning rate is set to 0.001 and batch size is set to 100. The training is stopped after 35 epochs. The performance of the proposed method on each dataset is compared with existing methods using the same testing protocol.

4.3 Results

NTU RGB+D Dataset  As in [37], the evaluation on this dataset is performed with two standard protocols, i.e., cross-subject evaluation and cross-view evaluation. In cross-subject evaluation, the sequences of 20 subjects are used for training and the data from 20 other subjects are used for testing. In cross-view evaluation, the sequences captured by two cameras are used for training and the rest are used for testing.

The results are shown in Table 1. It can be seen that the proposed method performs significantly better than others in both cross-subject and cross-view protocols. The accuracy of the proposed method is 79.57% when tested with the cross-subject protocol. Compared to the previous state-of-the-art method (ST-LSTM + Trust Gate [26]), the performance is improved by 10.37%. When tested with the cross-view protocol, the accuracy is improved from 77.7% to 84.83%.

The improved performance of the proposed method is due to the novel clip representation and feature learning method. As shown in Table 1, Frames + CNN achieves an accuracy of about 75.73% and 79.62% for the two testing protocols, respectively. The performances are much better than Coordinates + FTP. Compared to extracting temporal features of skeleton sequences with FTP and native 3D coordinates, using CNN to learn the temporal information of skeleton sequences from the generated frames is more robust to noise and temporal variations due to the convolution and pooling operators, resulting in better performances. From Table 1, it can also be seen that Frames + CNN also performs better than the previous state-of-the-art method. It clearly shows the effectiveness of the CNN features of the proposed clip representation. The performances are improved by learning entire clips with CNN and MTLN (i.e., Clips + CNN + MTLN). The improvements are about 4% and 5% for the two testing protocols, respectively. It can also be seen that the proposed MTLN (i.e., Clips + CNN + MTLN) performs better than feature concatenation (i.e., Clips + CNN + concatenation) and pooling (i.e., Clips + CNN + pooling). Frames + CNN, Clips + CNN + concatenation and Clips + CNN + pooling can be viewed as a single-task method, while using MTLN to process multiple frames of the generated clips in parallel utilizes their intrinsic relationships and incorporates the spatial structural information, which improves the performance of the single-task method for action recognition.

SBU Kinect Interaction Dataset   As in [53], the evaluation of this dataset is a 5-fold cross validation, with the provided training/testing splits. Each frame of the skeleton sequences contains two separate human skeletons. In this case, the two skeletons are considered as two data samples and the clip generation and feature extraction are conducted separately for the two skeletons. For testing, the prediction of actions is obtained by averaging the classification scores of the two samples.

Considering that the number of samples in this dataset is too small, data augmentation is performed to increase the number of samples. More specifically, each frame image of the generated clips are resized to $250\times 250$, and then random patches with size of $224\times 224$ are cropped from the original image for feature learning using CNN. For this dataset, 20 sub-images are cropped and the total data samples are extended to 11320.

The comparisons of the proposed method with other methods are shown in Table 2. Similar to the NTU RGB+D dataset, CNN features perform better than FTP to learn the temporal information. It can be seen that when using CNN features of individual frames, the accuracy is 90.88%, which is similar to the Deep LSTM + Co-occurrence method [54]. When incorporating the CNN features of the entire clips using concatenation and pooling methods, the performance is improved by about 2%. The performance is improved to 93.57% when learning the entire clips with MTLN. It clearly shows the benefit of using MTLN to learn the CNN features entire clips.

Since the joint positions of this dataset are not very accurate [53], existing methods including HBRNN [6] and Co-occurrence LSTM [54] remove the joint noise by smoothing the position of each joint using the Svaitzky-Golay filter [36]. In [26], a Trust Gate is introduced to remove the noisy joints and this improves the accuracy from 88.6% to 93.3%. Our method does not perform any pre-processing to handle the noisy joints, but still performs better than all the others. It clearly shows that the features learned from the generated clips are robust to noise due to the convolution and pooling operators of the deep network.

CMU Dataset  As in [54], for this dataset, the evaluation is conducted on both the entire dataset with 2235 sequences, and a selected subset of 664 sequences. The subset includes 8 classes of actions, , i.e., basketball, cartwheel, getup, jump, pickup, run, sit and walk back. For the entire dataset, the testing protocol is 4-fold cross validation, and for the subset, it is evaluated with 3-fold cross validation. The training/tesing splits of the different folds are provided by [54].

Similar to the SBU kinect interaction dataset, data augmentation is also conducted on CMU dataset. For the entire dataset, each frame image is used to generate 5 more images and the total data samples are extended to 11175, and for the subset, the total samples are extended to 13280, which is 20 times of the original number.

The results are shown in Table 3. It can be seen that the performance of the proposed method is much better than previous state-of-the-art methods on both the subset and the entire set. When tested on the subset, the accuracy of the proposed method was about 93.22%, which is about 5% better than the previous method [54]. The performance on the entire dataset is improved from 81.04% to 88.3%.

4.4 Discussions

Three gray clips or one color clip? As shown in Figure 1, the frames of the three generated clips are gray images, each corresponding to only one channel of the cylindrical coordinates. Each frame is duplicated three times to formulate a color image for CNN feature learning. The output CNN features of the three channels are concatenated in a feature vector for action recognition. A simple alternative is to generate a color clip with three channels of the cylindrical coordinates, and then extract a single CNN feature from the color frame for action recognition. When this was tested on CMU dataset, the performance is 84.67%, which is about 4% worse than the proposed method. This is perhaps due to the fact that the relationship of the three generated channels is different from that of the RGB channels of natural color images. The RGB channels are arranged in sequence and there is no matching order between 3D coordinates and RGB channels.

The more frames, the better performance?   This paper uses only four reference joints to generate clips, each having four frames. When 6 more joints are selected to generate more frames, i.e., the head, the left hand, the right hand, the left foot, the right foot and the hip, the performance does not improve. When tested on CMU data, the performance is 86.01%, which is about 2% worse than the proposed method. This is due to the fact that the other joints are not as stable as the selected four joints, which can introduce noise.

Cartesian coordinates or cylindrical coordinates?   As mentioned in Section 3.1, the 3D Cartesian coordinates of the vectors between the reference joints and the other joints are transformed to cylindrical coordinates to generate clips. We found that when using the original Cartesian coordinates for clip generation and action recognition, the performance drops. When tested on CMU dataset, the accuracy is 86.21%, which is about 2% worse than the proposed method. The cylindrical coordinates are more useful than the Cartesian coordinates to analyse the motions as each human skeleton utilizes pivotal joint movements to perform an action.

Features in different layers   As mentioned in Section 3.2.1, the feature maps in conv5_1 layer of the pre-trained CNN model is adopted as the representation of each input image. We found that using the features in the earlier layers decreased the performance. When using the features of the conv4_1 layer, the accuracy on CMU dataset is 84.59%, which is about 4% worse than the proposed method. This is perhaps due to the fact that the features in the earlier layers are not deep enough to capture the salient information of the input image. We also found that using the features in the later layers made the performance worse. When using the features of the fc6 layer, the accuracy on CMU dataset is 83.52%, which is about 5% worse than the proposed method. This is because the features in the later layers are more task-specific, which largely rely on the original classes and dataset. The features of the later layers are thus less suitable than those of the earlier layers to transfer to other domains [52, 27].

5 Conclusion

In this paper, we have proposed to transform a skeleton sequence to three video clips for robust feature learning and action recognition. We proposed to use a pre-trained CNN model followed by a temporal pooling layer to extract a compact representation of each frame. The CNN features of the three clips at the same time-step are concatenated in a single feature vector, which describes the temporal information of the entire skeleton sequence and one particular spatial relationship between the joints. We then propose an MTLN to jointly learn the feature vectors at all the time-steps in parallel, which utilizes their intrinsic relationships and improves the performance for action recognition. We have tested the proposed method on three datasets, including NTU RGB+D dataset, SBU kinect interaction dataset and CMU dataset. Experimental results have shown the effectiveness of the proposed new representation and feature learning method.

6 Acknowledgment

This work was partially supported by Australian Research Council grants DP150100294, DP150104251, and DE120102960. This paper used the NTU RGB+D Action Recognition Dataset made available by the ROSE Lab at the Nanyang Technological University, Singapore.