On human motion prediction using recurrent neural networks

TL;DR

This paper critically examines the use of recurrent neural networks for human motion prediction, revealing that simple baselines can outperform complex models, and proposes improved RNN architectures for better performance.

Contribution

The paper analyzes current RNN methods for human motion prediction and introduces three modifications that lead to a simple, scalable, and state-of-the-art RNN model.

Findings

Simple baselines can outperform complex RNN models.

Proposed modifications improve RNN performance.

New architecture achieves state-of-the-art results.

Abstract

Human motion modelling is a classical problem at the intersection of graphics and computer vision, with applications spanning human-computer interaction, motion synthesis, and motion prediction for virtual and augmented reality. Following the success of deep learning methods in several computer vision tasks, recent work has focused on using deep recurrent neural networks (RNNs) to model human motion, with the goal of learning time-dependent representations that perform tasks such as short-term motion prediction and long-term human motion synthesis. We examine recent work, with a focus on the evaluation methodologies commonly used in the literature, and show that, surprisingly, state-of-the-art performance can be achieved by a simple baseline that does not attempt to model motion at all. We investigate this result, and analyze recent RNN methods by looking at the architectures, loss functions, and training procedures used in state-of-the-art approaches. We propose three changes to the standard RNN models typically used for human motion, which result in a simple and scalable RNN architecture that obtains state-of-the-art performance on human motion prediction.

Tables

Table 1. Table 1 : Detailed results for motion prediction, measured in mean angle error for walking, eating, smoking and discussion activities of the Human 3.6M dataset. The top section corresponds to previous work based on deep recurrent neural networks. ‘‘Zero noise’’ is a model trained by feeding ground truth at each time step. ‘‘Sampling-based loss’’ is trained by letting the decoder feed its own output. SA stands for ‘‘Single action’’, and MA stands for ‘‘Multi-action’’. Finally ‘‘Untied’’ is the same model as Residual sup (MA), but with untied weights between encoder and decoder.

	Walking				Eating				Smoking				Discussion
milliseconds	80	160	320	400	80	160	320	400	80	160	320	400	80	160	320	400
ERD [10]	0.93	1.18	1.59	1.78	1.27	1.45	1.66	1.80	1.66	1.95	2.35	2.42	2.27	2.47	2.68	2.76
LSTM-3LR [10]	0.77	1.00	1.29	1.47	0.89	1.09	1.35	1.46	1.34	1.65	2.04	2.16	1.88	2.12	2.25	2.23
SRNN [18]	0.81	0.94	1.16	1.30	0.97	1.14	1.35	1.46	1.45	1.68	1.94	2.08	1.22	1.49	1.83	1.93
Running avg. 4	0.64	0.87	1.07	1.20	0.40	0.59	0.77	0.88	0.37	0.58	1.03	1.02	0.60	0.90	1.11	1.15
Running avg. 2	0.48	0.74	1.02	1.17	0.32	0.52	0.74	0.87	0.30	0.52	0.99	0.97	0.41	0.74	0.99	1.09
Zero-velocity	0.39	0.68	0.99	1.15	0.27	0.48	0.73	0.86	0.26	0.48	0.97	0.95	0.31	0.67	0.94	1.04
Zero noise (SA)	0.44	0.71	1.16	1.34	0.39	0.65	1.13	1.36	0.51	0.83	1.48	1.62	0.57	1.47	2.08	2.30
Sampling-based loss (SA)	0.92	0.98	1.02	1.20	0.98	0.99	1.18	1.31	1.38	1.39	1.56	1.65	1.78	1.80	1.83	1.90
Residual (SA)	0.34	0.60	0.95	1.09	0.30	0.53	0.92	1.13	0.36	0.66	1.17	1.27	0.44	0.93	1.45	1.60
Residual unsup. (MA)	0.27	0.47	0.70	0.78	0.25	0.43	0.71	0.87	0.33	0.61	1.04	1.19	0.31	0.69	1.03	1.12
Residual sup. (MA)	0.28	0.49	0.72	0.81	0.23	0.39	0.62	0.76	0.33	0.61	1.05	1.15	0.31	0.68	1.01	1.09
Untied (MA)	0.33	0.54	0.78	0.91	0.28	0.45	0.65	0.83	0.35	0.62	1.03	1.14	0.35	0.71	1.01	1.09

Table 2. Table 2 : Prediction results for our zero-velocity baseline and our main prediction methods on the remainder 11 actions of the H3.6m dataset.

	Directions				Greeting				Phoning				Posing				Purchases				Sitting
milliseconds	80	160	320	400	80	160	320	400	80	160	320	400	80	160	320	400	80	160	320	400	80	160	320	400
Zero-velocity	0.25	0.44	0.61	0.68	0.80	1.23	1.81	1.87	0.80	1.23	1.81	1.87	0.32	0.63	1.16	1.45	0.72	1.03	1.46	1.49	0.43	1.12	1.41	1.58
Res. (SA)	0.44	0.95	1.27	1.55	0.87	1.40	2.19	2.26	0.31	0.57	0.88	1.04	0.50	0.96	1.64	1.96	0.74	1.60	1.57	1.72	0.44	1.05	1.51	1.69
Res. unsup. (MA)	0.27	0.47	0.73	0.87	0.77	1.18	1.74	1.84	0.24	0.43	0.68	0.83	0.40	0.77	1.32	1.62	0.62	1.10	1.07	1.14	0.68	1.04	1.43	1.65
Res. sup. (MA)	0.26	0.47	0.72	0.84	0.75	1.17	1.74	1.83	0.23	0.43	0.69	0.82	0.36	0.71	1.22	1.48	0.51	0.97	1.07	1.16	0.41	1.05	1.49	1.63
Untied (MA)	0.31	0.52	0.77	0.89	0.79	1.19	1.72	1.83	0.27	0.46	0.68	0.85	0.42	0.77	1.29	1.58	0.52	1.01	1.07	1.16	0.51	1.13	1.56	1.74
	Sitting down				Taking photo				Waiting				Walking Dog				Walking together				Average
milliseconds	80	160	320	400	80	160	320	400	80	160	320	400	80	160	320	400	80	160	320	400	80	160	320	400
Zero-velocity	0.27	0.54	0.93	1.05	0.22	0.47	0.78	0.89	0.27	0.49	0.96	1.12	0.60	0.96	1.27	1.33	0.33	0.60	0.96	1.03	0.42	0.74	1.12	1.20
Res. (SA)	0.38	0.77	1.36	1.59	0.37	0.66	1.30	1.70	0.36	0.73	1.31	1.51	0.62	1.02	1.55	1.65	0.44	0.81	1.25	1.36	0.46	0.88	1.35	1.54
Res. unsup. (MA)	0.41	0.80	1.43	1.63	0.27	0.56	0.98	1.16	0.32	0.62	1.13	1.30	0.58	0.95	1.37	1.45	0.35	0.62	0.87	0.87	0.39	0.72	1.08	1.22
Res. sup. (MA)	0.39	0.81	1.40	1.62	0.24	0.51	0.90	1.05	0.28	0.53	1.02	1.14	0.56	0.91	1.26	1.40	0.31	0.58	0.87	0.91	0.36	0.67	1.02	1.15
Untied (MA)	0.47	0.89	1.57	1.72	0.30	0.56	0.95	1.12	0.38	0.64	1.18	1.41	0.61	0.98	1.42	1.54	0.40	0.69	0.98	1.03	0.42	0.74	1.11	1.26

Full text

1 Introduction

An important component of our capacity to interact with the world resides in the ability to predict its evolution over time. Handing over an object to another person, playing sports, or simply walking in a crowded street would be extremely challenging tasks without our understanding of how people move, and our ability to predict what they are likely to do in the following instants. Similarly, machines that are able to perceive and interact with moving people, either in physical or virtual environments, must have a notion of how people move. Since human motion is the result of both physical limitations (e.g. torque exerted by muscles, gravity, moment preservation) and the intentions of subjects (how to perform an intentional motion), motion modeling is a complex task that should be ideally learned from observations.

Our focus in this paper is to learn models of human motion from motion capture (mocap) data. More specifically, we are interested in human motion prediction, where we forecast the most likely future 3D poses of a person given their past motion. This problem has received interest in a wide variety of fields, such as action prediction for socially-aware robotics [21], 3D people tracking within computer vision [43, 13], motion generation for computer graphics [22] or modeling biological motion in psychology [42].

Traditional approaches have typically imposed expert knowledge about motion in their systems in the form of Markovian assumptions [32, 25], smoothness, or low dimensional embeddings [48]. Recently, a family of methods based on deep recurrent neural networks (RNNs) have shown good performance on this task while trying to be more agnostic in their assumptions. For example, [10] uses curriculum learning and incorporates representation learning in the architecture, and [18] manually encodes the semantic similarity between different body parts. These approaches benefit from large, publicly available collections of motion capture data [16], as well as recent advances in the optimization of time-series modelling [9].

Recent work has validated its performance via two complementary methods: (1) quantitative prediction error in the short-term, typically measured as a mean-squared loss in angle-space, and (2) qualitative motion synthesis for longer time horizons, where the goal is to generate feasible motion. The first evaluation metric is particularly interesting for computer vision applications such as people tracking, where predictions are continually matched and corrected with new visual evidence. The second criterion, most relevant for open-loop motion generation in graphics, is hard to evaluate quantitatively, because human motion is a highly non-deterministic process over long time horizons. This problem is similar to the one found in recent research on deep generative networks [41], where the numerical evaluation based on the negative log-likelihood and Parzen window estimates are known to be far from perfect.

We have empirically observed that current deep RNN-based methods have difficulty obtaining good performance on both tasks. Current algorithms are often trained to minimize a quantitative loss for short-term prediction, while striving to achieve long-term plausible motion by tweaking the architectures or learning procedures. As a result, their long-term results suffer from occasional unrealistic artifacts such as foot sliding, while their short-term results are not practical for tracking due to clear discontinuities in the first prediction. In fact, the discontinuity problem is so severe, that we have found that state-of-the-art methods are quantitatively outperformed by a range of simple baselines, including a constant pose predictor. While this baseline does not produce interesting motion in the long-run, it highlights both a poor short-term performance, as well as a severe discontinuity problem in current deep RNN approaches. In this work, we argue that (a) the results achieved by recent work are not fully satisfactory for either of these problems, and (b) trying to address both problems at once is very challenging, especially in the absence of a proper quantitative evaluation for long-term plausibility.

We focus on short-term prediction, which is the most relevant task for a visual tracking scenario. We investigate the reasons for the poor performance of recent methods on this task by analyzing several factors such as the network architectures and the training procedures used in state-of-the-art RNN methods. First, we consider the training schedule used in [10, 18]. It is a known problem in RNNs [5] and reinforcement learning [36] that networks cannot learn to recover from their own mistakes if they are fed only ground-truth during training. The authors of [10, 18] introduced increasing amounts of random noise during training to compensate for this effect. However, this noise is difficult to tune, makes it harder to choose the best model based on validation error, and has the effect of degrading the quality of the prediction in the first frame. Instead, we propose a simple approach that introduces realistic error in training time without any scheduling; we simply feed the predictions of the net, as it is done in test time. This increases the robustness of the predictions compared to a network trained only on ground truth, while avoiding the need of a difficult-to-tune schedule.

Unfortunately, this new architecture is still unable to accurately represent the conditioning poses in its hidden representation, which still results in a discontinuity in the first frame of the prediction. We borrow ideas from research on the statistics of hand motion [15], and model velocities instead of absolute joint angles, while keeping the loss in the original angle representation to avoid drift. Therefore, we propose a residual architecture that models first-order motion derivatives, which results in smooth and much more accurate short-term predictions.

Both of our contributions can be implemented using an architecture that is significantly simpler than those in previous work. In particular, we move from the usual multi-layer LSTM architectures (long short-term memory) to a single GRU (Gated Recurrent Unit), and do not require a spatial encoding layer. This allows us to train a single model on the entire Human 3.6M dataset [16] in a few hours. This differs from previous approaches [10, 18], which trained only action-specific models from that dataset. Our approach sets the new state of the art on short-term motion prediction, and overall gives insights into the challenges of motion modelling using RNNs. Our code is publicly available at https://github.com/una-dinosauria/human-motion-prediction.

2 Related work

Our main task of interest is human motion prediction, with a focus on recent deep RNN architectures [10, 18]. One of our findings is that, similar to [17, 3], a family of simple baselines outperform recent deep learning approaches. We briefly review the literature on these topics below.

Modelling of human motion.

Learning statistical models of human motion is a difficult task due to the high-dimensionality, non-linear dynamics and stochastic nature of human movement. Over the last decade, and exploiting the latent low-dimensionality of action-specific human motion, most work has focused on extensions to latent-variable models that follow state-space equations such as hidden Markov models (HMMs) [26], exploring the trade-offs between model capacity and inference complexity. For example, Wang et al. [48] use Gaussian-Processes to perform non-linear motion prediction, and learn temporal dynamics using expectation maximization and Markov-chain Monte Carlo. Taylor et al. [40] assume a binary latent space and model motion using a conditional restricted Boltzman machine (CRBM), which requires sampling for inference. Finally, Lehrmann et al. [26] use a random forest to non-linearly choose a linear system that predicts the next frame based on the last few observations.

Applications of human motion models.

Motion is a key part of actions; therefore, the field of action recognition has paid special attention to models and representations of human motion. In their seminal work, Yacoob and Black [51] model motion with time-scale and time-shifted activity bases from a linear manifold of visual features computed with Principal Component Analysis (PCA). More complex models like mixtures of HMMs [34, 33], latent topic models of visual words [49] or LSTMs [29] are used in recent methods. Although their purpose (action classification from a sequence of poses) is different from ours, this field contains interesting insights for motion prediction, such as the importance of a mathematically sound orientation representation [46, 47] or how learned, compact motion representations improve action recognition accuracy [30].

Another popular use of motion models, specially short-term ones, is pose tracking. The use of simple linear Markovian models [37] or PCA models [44] has evolved to locally linear ones like factor analizers [28], non-linear embeddings like Laplacian Eigenmaps [38], Isomap [19], dynamic variants of Gaussian Process Latent Variable Models (GPLVM) [48, 52], or physics-based models [8].

In animation, similar methods have been used for the generation of human pose sequences. Spaces of HMMs parameterised by style were used by Brand et al. [7] to generate complex motions. Arikan and Forsyth [2] collapse full sequences into nodes in a directed graph, connected with possible transitions between them, and in [24] cluster trees improve the path availability. More recently, motion models based on GPLVM have been used for controlling virtual characters in a physical simulator [27]. An overview of motion generation for virtual characters can be found in  [45].

Deep RNNs for human motion.

Our work focuses on recent approaches to motion modelling that are based on deep RNNs. Fragkiadaki et al. [10] propose two architectures: LSTM-3LR (3 layers of Long Short-Term Memory cells) and ERD (Encoder-Recurrent-Decoder). Both are based on concatenated LSTM units, but the latter adds non-linear space encoders for data pre-processing. The authors also note that, during inference, the network is prone to accumulate errors, and quickly produces unrealistic human motion. Therefore, they propose to gradually add noise to the input during training (as is common in curriculum learning [6]), which forces the network to be more robust to prediction errors. This noise scheduling makes the network able to generate plausible motion for longer time horizons, specially on cyclic walking sequences. However, tuning the noise schedule is hard in practice.

More recently, Jain et al. [18] introduced structural RNNs (SRNNs), an approach that takes a manually designed graph that encodes semantic knowledge about the RNN as input, and creates a bi-layer architecture that assigns individual RNN units to semantically similar parts of the data. The authors also employ the noise scheduling technique introduced by Fragkiadaki et al., and demonstrate that their network outperforms previous work both quantitatively in short-term prediction, as well as qualitatively. Interestingly, SRNNs produce plausible long-term motion for more challenging, locally-periodic actions such as eating and smoking, and does not collapse to unrealistic poses in aperiodic ‘‘discussion’’ sequences.

Revisiting baselines amid deep learning.

The rise and impressive performance of deep learning methods in classical problems such as object recognition [23] has encouraged researchers to attack both new and historically challenging problems using variations of deep neural networks. For example, there is a now a large body of work on visual question answering (VQA), i.e. the task of answering natural-language questions by looking at images, based almost exclusively on end-to-end trainable systems with deep CNNs for visual processing and deep RNNs for language modelling [50, 35, 31]. Recently, however, Zhou et al. [53] have shown that a simple baseline that concatenates features from questions’ words and CNN image features performs comparably to approaches based on deep RNNs. Moreover, Jabri et al. [17] have shown competitive performance on VQA with a simple baseline that does not take images into account, and state-of-the-art performance with a baseline that is trained to exploit the correlations between questions, images and answers.

Our work is somewhat similar to that of Jabri et al. [17], in that we have found a very simple baseline that outperforms sophisticated state-of-the-art methods based on deep RNNs for short-term motion prediction. In particular, our baseline outperforms the ERD and LSTM-3LR models by Fragkiadi et al. [10], as well as the structural RNN (SRNN) method of Jain et al. [18]. Another example of baselines outperforming recent work in the field of pose models can be found in [25], where a Gaussian pose prior outperforms the more complicated GPLVM.

3 Method

Recent deep learning methods for human pose prediction [10, 18] offer an agnostic learning framework that could potentially be integrated with video data [10] or used for other forecasting applications [18]. However, for the specific task of motion forecasting, we note that they have a few common pitfalls that we would like to improve.

3.1 Problems

First frame discontinuity.

While both methods generate continuous motion, a noticeable jump between the conditioning ground truth and the first predicted frame is present in their results (see Figure 1). This jump is particularly harmful for tracking applications, where short-term predictions are continuously updated with new visual evidence.

Hyper-parameter tuning.

These methods add to the typical set of network hyper-parameters an additional one, particularly hard to tune: the noise schedule.

In time series modelling, it is often necessary to model noise as part of the input, in order to improve robustness against noisy observations. For example, in Kalman filtering, a small amount of Gaussian noise is modelled explicitly as part of the standard state-space equations. In applications such as motion synthesis, exposing the method to the errors that the network will make at test time is crucial to prevent the predicted poses from leaving the manifold of plausible human motion. Algorithms like DAGGER [36], used in reinforcement learning, use queries to an ‘‘expert’’ during training so that the predictor learns how to correct its own errors. It is, however, not straightforward how one would use this approach for pose prediction.

The basic architectures that we use, RNNs, typically do not consider this mismatch between train and test input, which makes them prone to accumulate errors at inference time. To alleviate this problem, Fragkiadaki et al. propose to use noise scheduling; that is, to inject noise of gradually increasing magnitude to the input in training time (see Fig. 2, left), which corresponds to a type of curriculum learning. Jain et al. [18] similarly adopt this idea, and have found that it helps stabilizing long-term motion synthesis. The downsides are, (1) that both noise distribution and magnitude scheduling are hard to tune, (2) that while this noise improves long-term predictions, it tends to hurt performance in short-term predictions, as they become discontinuous from previous observations, and (3) that the common rule for choosing the best model, based on lowest validation error, is not valid anymore, since lowest validation error typically corresponds to the validation epoch without injected noise.

Depth and complexity of the models.

LSTM-3LR, ERD and SRNN use more than one RNN in their architectures, stacking two or three layers for increased model capacity. While deeper models have empirically shown the best performance on a series of tasks such as machine translation [39], deep networks are known to be hard to train when data is scarce (which is the data regime for action-specific motion models). Moreover, recent work has shown that shallow RNNs with minimal representation processing can achieve very competitive results in tasks such as the learning of sentence-level embeddings [20], as long as a large corpus of data is available. Finally, deeper models are computationally expensive, which is an important factor to consider in the context of large-scale training datasets.

Action-specific networks.

Although the vision community has recently benefited from large-scale, publicly available datasets of motion capture data [16], motion modelling systems have been typically trained on small action-specific subsets. While restricting the training data to coherent subsets makes modelling easier, it is also well-known that deep networks work best when exposed to large and diverse training datasets [23, 20]. This should specially apply to datasets like Human3.6M, where different actions contain large portions of very similar data (e.g. sitting or walking).

3.2 Solutions

Sequence-to-sequence [39] architecture.

We address short-term motion prediction as the search for a function that maps an input sequence (conditioning ground truth) to an output (predicted) sequence. In this sense, the problem is analogous to machine translation, were sequence-to-sequence (seq2seq) architectures are popular. In seq2seq, two networks are trained; (a) an encoder that receives the inputs and generates an internal representation, and (b), a decoder network, that takes the internal state and produces a maximum likelihood estimate for prediction. Unlike the common practice in machine translation, we enforce the encoder and the decoder to share weights, which we found to accelerate convergence. A benefit of this architecture is that the encoding-decoding procedure during training is more similar to the protocol used at test-time. Moreover, there are multiple variations of seq2seq architectures (e.g., with attention mechanisms [4], or bi-directional encoders [35]), that could potentially improve motion prediction.

Sampling-based loss.

While it is often common in RNNs to feed the ground truth at each training time-step to the network, this approach has the downside of the network not being able to recover from its own mistakes. Previous work has addressed this problem by scheduling the rate at which the network sees either the ground truth or its own predictions [5], or by co-training and adversarial network to force the internal states of the RNN to be similar during train and test time [11]. These approaches, however, rely heavily on hyper-parameter tuning, which we want to avoid. Striving for simplicity, during training we let the decoder produce a sequence by always taking as input its own samples. This approach requires absolutely no parameter tuning. Another benefit of this approach is that we can directly control the length of the sequences that we train on. As we will see, training to minimize the error on long-term motions results in networks that produce plausible motion in the long run, while training to minimize error the short-term reduces the error rate in the first few predicted frames.

Residual architecture.

While using a seq2seq architecture trained with a sampling-based loss can produce plausible long-term motion, we have observed that there is still a strong discontinuity between the conditioning sequence and prediction. Our main insight is that motion continuity, a known property of human motion, is easier to express in terms of velocities than in poses. While it takes considerable modelling effort to represent all possible conditioning poses so that the first frame prediction is continuous, it only requires modeling one particular velocity (zero, or close to zero velocity) to achieve the same effect. This idea is simple to implement in current deep learning architectures since it translates into adding a residual connection between the input and the output of each RNN cell (see Fig. 2, right). We note that, although residual connections have been shown to improve performance on very deep convolutional networks [14], in our case they help us model prior knowledge about the statistics of human motion.

Multi-action models.

We also explore training a single model to predict motion for multiple actions, in contrast to previous work [10, 18], which has focused on building action-specific models. While modelling multiple actions is a more difficult task than modelling single-action sets, it is now a common practice to train a single, conditional model, on multiple data modalities, as this allows the network to exploit regularities in large datasets [12]. Semantic knowledge about each activity can be easily incorporated using one-hot vectors; i.e., concatenating, in the input, a 15-dimensional vector that has zeros everywhere, but a value of one in the index of the indicated action.

4 Experimental setup

We consider three main sets of experiments to quantify the impact of our contributions:

1. Seq2seq architecture and sampling-based loss.

First, we train action-specific models using our proposed sequence-to-sequence architecture with sampling-based loss, and compare it to previous work, which uses noise scheduling, and to a baseline that feeds the ground truth at each time-step. The goal of these experiments is to verify that using a sampling-based loss, which does not require parameter tuning, performs on par with previous work on short-term motion prediction, while still producing plausible long-term motion. In these experiments, the network is trained to minimize the loss over 1 second of motion.

2. Residual architecture.

The second set of experiments explore the effects of using a residual architecture that models first-order motion derivatives, while keeping the loss in the original angle space. Here, we are interested in learning whether a residual architecture improves short term prediction; therefore, in these experiments, the network is trained to minimize the prediction error over 400 milliseconds.

3. Multi-action models.

Our last round of experiments quantifies the benefits of training our architecture on the entire Human 3.6M dataset, as opposed to building action-specific models. We consider both a supervised and an unsupervised variant. The supervised variant enhances the input to the model by concatenating one-hot vectors with the 15 action classes. In contrast, the unsupervised variant does not use one-hot input during training nor prediction. In these experiments we also train the network to minimize the prediction error over the next 400 milliseconds.

Dataset and data pre-processing.

Following previous work, we use the Human 3.6M (H3.6M) dataset by Ionescu et al. [16], which is currently the largest publicly available dataset of motion capture data. H3.6M includes seven actors performing 15 varied activities such as walking, smoking, engaging in a discussion, taking pictures, and talking on the phone, each in two different trials. For a fair comparison, we adopt the pose representation and evaluation loss from [10, 18]. Pose is represented as an exponential map representation of each joint, with a special pre-processing of global translation and rotation (see  [40] for more details). For evaluation, similar to [10, 18], we measure the Euclidean distance between our prediction and the ground truth in angle-space for increasing time horizons. We report the average error on eight randomly sampled test sequences, and use the sequences of subject five for testing, while the rest of the sequences are used for training.

A scalable seq2seq architecture.

In all our experiments, we use a single gated recurrent unit [9] (GRU) with 1024 units, as a computationally less-expensive alternative to LSTMs, and we do not use any time-independent layers for representation learning. Experimentally, we found that stacking recurrent layers makes the architecture harder to train, while it also makes it slower; we also found that the best performance is obtained without a spatial encoder. We do, however, use a spatial decoder to back-project the 1024-dimensional output of the GRU to 54 dimensions, the number of independent joint angles provided in H3.6M.

We use a learning rate of 0.005 in our multi-action experiments, and a rate of 0.05 in our action-specific experiments; in both cases, the batch size is 16, and we clip the gradients to a maximum L2-norm of 5. During training as well as testing, we feed 2 seconds of motion to the encoder, and predict either 1 second (for long-term experiments) or 400 milliseconds (for short-term prediction) of motion from the decoder. We implemented our architecture using TensorFlow [1], which takes 75ms for forward processing and back-propagation per iteration on an NVIDIA Titan GPU.

Baselines.

We compare against two recent approaches to human motion prediction based on deep RNNs: LSTM-3LR and ERD by Fragkiadaki et al. [10], and SRNN by Jain et al. [18]. To reproduce previous work, we rely on the pre-trained models and implementations of ERD, LSTM-3LR and SRNN publicly available 111https://github.com/asheshjain399/RNNexp. These implementations represent the best efforts of the SRNN authors to reproduce the results of the ERD and LSTM-3LR models reported by Fragkiadaki et al. [10], as there is no official public implementation for that work. We found that, out of the box, these baselines produce results slightly different (most often better) from those reported by Jain et al. [18].

We also consider an agnostic zero-velocity baseline which constantly predicts the last observed frame. For completeness, we also consider running averages of the last two and four observed frames. While these baselines are very simple to implement, they have not been considered in recent work that uses RNNs to model human motion.

5 Results

Figure 3 shows a summary of the results obtained by ERD, LSTM-3LR and SRNN, as well as a zero-velocity baseline and our method, on four actions of the Human 3.6 dataset. Tables 1 and 2 describe these results in more detail, and include results on the rest of the actions. In the remainder of the section we analyze these results.

Zero-velocity baseline.

The first striking result is the comparatively good performance of the baselines, specially the zero-velocity one. They clearly outperform state-of-the-art results, highlighting the severity of the discontinuities between conditioning and prediction in previous work. The good performance of the baseline also means that deterministic losses are not suitable to evaluate motion forecasting with a long time horizon.

Sampling-based loss.

In Table 1, using our sampling-based loss consistently achieves motion prediction error competitive with or better than the state of the art. Moreover, since we have trained our model to minimize the error over a 1-second time horizon, the network retains the ability to generate plausible motion in the long run. Figure 4 shows a few qualitative examples of long-term motion using this approach. Given that our proposed sampling-based loss does not require any hyper-parameter tuning, we would argue that this is a fast-to-train, interesting alternative to previous work for long-term motion generation using RNNs.

Residual architecture and multi-action models.

Finally, we report the performance obtained by our architecture with sampling-based loss, residual connections and trained on single (SA) or multiple actions (MA) in the bottom subgroup of Table 1. We can see that using a residual connection greatly improves performance and pushes our method beyond the state of the art, which highlights the fact that velocity representations are easier to model by our network. Importantly, our method obtains its best performance when trained on multiple actions; this result, together with the simplicity of our approach, uncovers the importance of large amounts of training data when learning short-term motion dynamics. We also note that highly aperiodic classes such as discussion, directions and sitting down remain very hard to model.

Moreover, we observe that adding semantic information to the network in the form of action labels helps in most cases, albeit by a small margin. Likely, this is due to the fact that, for short-term motion prediction, modelling physical constraints (e.g. momentum preservation) is more important than modelling high-level semantic intentions.

When analysing Fig. 4, it becomes obvious that the best numerical results do not correspond to the best qualitative long-term motion -- a result that persists even when trained to minimize loss over long horizons (e.g. 1 second). One can hardly blame the method though, since our network is achieving the lowest loss in an independent validation set. In other words, the network is excelling in the task that has been assigned to it. In order to produce better qualitative results, we argue that a different loss that encourages other similarity measures (e.g. adversarial, entropy-based etc.) should be used instead. Our results suggest that it is inherently hard to produce both accurate short-term predictions -- which are relatively deterministic and seem to be properly optimized with the current loss -- and long-term forecasting using RNNs.

6 Conclusions and future work

We have demonstrated that previous work on human motion modelling using deep RNNs has harshly neglected the important task of short-term motion prediction, as we have shown that a zero-velocity prediction is a simple but hard-to-beat baseline that largely outperforms the state of the art. Based on this observation, we have developed a sequence-to-sequence architecture with residual connections which, when trained on a sample-based loss, outperforms previous work. Our proposed architecture, being simple and scalable, can be trained on large-scale datasets of human motion, which we have found to be crucial to learn the short-term dynamics of human motion. Finally, we have shown that providing high-level supervision to the network in the form of action labels improves performance, but an unsupervised baseline is very competitive nonetheless. We find this last result particularly encouraging, as it departs from previous work in human motion modelling which has typically worked on small, action-specific datasets. Future work may focus on exploring ways to use even larger datasets of motion capture in an unsupervised manner.

Acknowledgements.

The authors thank Laura Sevilla-Lara for proofreading our work. We also thank Nvidia for the donation of some of the GPUs used in this research. This research was supported in part by NSERC.