Language2Pose: Natural Language Grounded Pose Forecasting

TL;DR

Language2Pose introduces a neural model that maps natural language descriptions to 3D human poses, enabling more accurate and human-like animations for applications like virtual characters and robot motion planning.

Contribution

The paper presents JL2P, a joint embedding neural architecture with curriculum learning for language-to-pose translation, improving animation accuracy over previous methods.

Findings

Outperforms existing data-driven approaches in accuracy

Generates animations that are more visually representative

Validated on a public 3D pose and language dataset

Abstract

Generating animations from natural language sentences finds its applications in a a number of domains such as movie script visualization, virtual human animation and, robot motion planning. These sentences can describe different kinds of actions, speeds and direction of these actions, and possibly a target destination. The core modeling challenge in this language-to-pose application is how to map linguistic concepts to motion animations. In this paper, we address this multimodal problem by introducing a neural architecture called Joint Language to Pose (or JL2P), which learns a joint embedding of language and pose. This joint embedding space is learned end-to-end using a curriculum learning approach which emphasizes shorter and easier sequences first before moving to longer and harder ones. We evaluate our proposed model on a publicly available corpus of 3D pose data and human-annotated sentences. Both objective metrics and human judgment evaluation confirm that our proposed approach is able to generate more accurate animations and are deemed visually more representative by humans than other data driven approaches.

Tables

Table 1. Table 1 : Average positional error (APE) for JL2P , JL2P w/o Joint Emb., JL2P w/o L1, JL2P w/o Curriculum and Lin et. al.. Lower is better. Our models (JL2P and variants) show consistent increase in accuracy over Lin et. al. across all joints with the addition of components joint embedding, smooth L1 loss and curriculum learning. Two-tailed pairwise t-test between all models and JL2P where ∗ ⁣ ∗ ⁣ ∗ *** - p < < 0.001, and ∗ ⁣ ∗ ** - p < < 0.01.

Models	Average Positional Error (APE) in mm
	Mean	Mean w/o Root	Root	Torso	Head	LArm	RArm	LHip	RHip	LFoot	RFoot
Lin et. al.[17]	54.9∗∗∗	50.0	151.6	26.6	35.4	61.3	61.6	32.2	32.1	63.3	63.2
JL2P w/o Curriculum	52.2∗∗∗	47.9	139.2	24.2	32.5	57.3	57.2	30.6	30.7	62.9	63.2
JL2P w/o L1	51.7∗∗	47.0	145.0	24.4	32.8	58.0	57.6	29.9	30.7	59.3	59.8
JL2P w/o Joint Emb.	50.4	45.7	143.3	24.0	31.0	55.6	54.5	29.7	29.5	59.0	59.5
JL2P	49.5	45.4	131.1	23.0	31.4	55.3	55.0	28.6	29.0	59.2	58.8

Equations

$${\hat{Y}}_{1:T}=f\left(X_{1:N};\Theta \right)$$

$$z_x=p_e\left(X_{1:N};{\Phi }_e\right)$$

$$z_y=q_e\left(Y_{1:T};{\Psi }_e\right)$$

$${\mathcal{L}}_c=d\left(q_d\left(z_x;{\Psi }_d\right),Y_{1:T}\right)$$

$${\mathcal{L}}_u=d\left(q_d\left(z_y;{\Psi }_d\right),Y_{1:T}\right)$$

$${\mathcal{L}}_j={\mathcal{L}}_c+{\mathcal{L}}_u$$

$$SmoothL1(x,y)=\{\begin{array}{ll}0.5(x-y{)}^2& \text{for }|x-y|<1\\ |x-y|-0.5& \text{otherwise}\end{array}$$

$$\text{mbox}APE(p)=\frac{1}{\mathcal{Y}}\sum _{\mathcal{Y}}\Vert {\hat{y}}_t[j]-y_t[j]{\Vert }_2$$

Full text

In this paper, we address this multimodal problem by introducing a neural architecture called Joint Language-to-Pose (or JL2P), which learns a joint embedding of language and pose. This joint embedding space is learned end-to-end using a curriculum learning approach which emphasizes shorter and easier sequences first before moving to longer and harder ones. We evaluate our proposed model on a publicly available corpus of 3D pose data and human-annotated sentences. Both objective metrics and human judgment evaluation confirm that our proposed approach is able to generate more accurate animations and are deemed visually more representative by humans than other data driven approaches.

http://chahuja.com/language2pose

1 Introduction

Generating animations from natural language descriptions is a first step for movie script visualization [11, 20] which can later be stitched together while maintaining co-references in the story-line [38]. These language grounded animations can also be useful in cases like virtual human animation [30, 7, 6], robot motion and task planning [16, 2].

An animation consists of a sequence of poses, which can be represented by positions of different joints in the body such as Root (base of spine), head, shoulder, wrist, knee and many more.

Pose forecasting conditioned on natural language has 3 major challenges. First, pose and natural language are very different modalities. The model needs a joint space where both natural language sentences and poses can be mapped. The model should also be able to decode animations from this embedding space. Second, different words of a sentence represent different qualities about the animation. Verbs and adverbs describe the action and speed/acceleration of the action; nouns and adjectives describe locations and directions respectively. The model has to map these concepts to small pose sequences and then stitch them to render convincing animations. Third, we want to see if objective metrics correlate with subjective metrics for this task as our models are trained using objective distance metrics, but the quality of generated animations can only be judged by humans.

In this paper, our two main contributions tackle the modeling challenges of pose and natural language. First, we propose a model Joint Language-to-Pose (or JL2P) that learns a joint embedding space of these two modalities. Second, we use a training curriculum to help the model emphasize more on shorter and and easier sequences first and longer and harder sequences later. Additionally, to make the training regimen robust to the outliers in the dataset, we use Smoooth L1 as the distant metric in our loss function. Through multiple objective and subjective experiments, we show that our model can generate more accurate and natural animations from natural language sentences than other data driven models.

2 Related Work

Pose Forecasting: Data driven human pose forecasting attempts to understand the behaviours of the subject from its history of poses and generates the next sequence of poses. Short-term predictions [24] focus on modeling joint angles corresponding to hands, legs, head and torso. Long-term predictions [10, 31, 24] additionally model the positions of the human character to generate animations like walking, running, jumping and crawling.

While some works use different actions (such as running, kicking, and more) as conditioning variables to generate the future pose [31, 18], others rely solely on the history of poses to predict what kind of motion will follow [8]. Pose forecasting for locomotion is a more commonly researched topic, where models decide where and when to run/walk based on low-level control parameters such as trajectory and terrain [13]. Task based locomotion (such as writing on a whiteboard, moving a box, and sitting on a box) add the nuances of transitioning from one task to another, but pose generation is based on task-specific footstep plans that act as motion templates [1].

All these approaches are either action specific, or require a set of low-level control parameters to forecast future pose. In this work, we aim replace low-level control parameters with high-level control parameters (e.g. natural language) to control actions and their speed and direction for the generated pose.

Image or Speech conditioned pose forecasting: Images with a human can act as a context to forecast what comes next. Chao et. al. [5] use one image frame to predict the next few poses. These generated poses can now be used to aid the generation of a video [35] or a sequence of images [19]. An image, a high-level control parameter, has action information for pose generation, but it does not provide a fine-grained control on the speed and acceleration of the motion trajectory.

Speech can also be used to control animations of virtual characters. Taylor et. al.[32] use a data driven approach to model facial animation, while upper body pose forecasting conditioned on speech inputs has been tackled by Takeuchi et. al.[30]. But, these pose sequences model the non-verbal behaviours (such as head nods, pose switches, hand waving and so on) of the character and do not offer fine-grained control over the characters next movements.

Language conditioned pose forecasting: Natural language sentences consists of verbs describing the actions, adverbs describing the speed/acceleration of the action, and nouns with adjectives to describe the direction or target. This information can help provide a more fine-grained control over pose generations compared to image or speech.

Statistical models [29, 28] which use bigram models for natural language have been trained to encode motion sequences from sentences. Ahn et. al. [2] use around 2100 hours of youtube videos with annotated text descriptions to train a pose generation model. Pose sequences extracted from videos have limited translation and occluded lower bodies, hence their model only predicts the upper body with a static Root joint. Some works use 3D motion capture data instead [26, 34].

Human motions generally have translation of the Root joint, hence forecasting trajectory is important to get natural looking animations. Lin et. al [17] generates pose of all the joints of the body by pretraining a pose2pose autoencoder model before mapping language embeddings on the learned pose space. But the embedding space is not learned jointly [23] which may limit the generative powers of the pose decoder. In contrast, our proposed approach learns a joint embedding space of language and pose using a curriculum learning training regime.

3 Problem Statement

As an example, consider a natural language sentence which describes a human’s motion: ”A person walks in a circle”. The goal of this cross-modal language-to-pose translation task is to generate an animation representing the sentence; i.e. an animation that shows a person following a trajectory of a circle with a walking motion (see figure 1).

Formally, given a sentence, represented by an N-sized sequence of words $X_{1:N}=\left[x_{1},x_{2},\ldots x_{N}\right]$, we want to predict a T-sized sequence of 3D poses $Y_{1:T}=\left[y_{1},y_{2},\ldots y_{T}\right]$ that are coherent with the semantics in the sentence. $x_{i}\in\mathcal{R}^{K}$ is the $i^{th}$ word vector with dimension $K$. $y_{t}\in\mathcal{R}^{J\times 3}$ is the pose matrix at time $t$. Rows of $y_{t}$ represent joints of the skeleton and columns are the $xyz$-coordinates of each joint. Tensors $X$ and $Y$ are elements of sets $\mathcal{X}$ and $\mathcal{Y}$ respectively.

Modeling language-to-pose is done by training a model $f:\mathcal{R}^{K\times N}\longrightarrow\mathcal{R}^{J\times 3\times T}$ to predict a pose sequence $\hat{Y}_{1:T}$

[TABLE]

where $\Theta$ are trainable parameters of the model $f$.

4 Joint Language-to-Pose

Language-to-pose models should be able to grasp nuanced concepts like speed, direction of motion and the kind of actions from the language and translate them to pose sequences (or animations). This requires the model to learn a multimodal joint space of language and pose. In doing so, it should also be able to generate sequences that are deemed correlated to the sentence by humans.

To achieve that objective, we propose Joint Language-to-Pose (or JL2P) model to learn the joint embedding space. Given an input sentence, an animation can be sampled from this model at inference stage.

In this section, a joint embedding space of language and pose is formalized. This is followed by an algorithm to train for the joint embedding space and a discussion on the practical edge cases at inference time for our Joint Language-to-Pose model.

4.1 Joint Embedding Space for Language and Pose

To learn a joint embedding space of language and pose, the sentence $X_{1:N}$ and pose $Y_{1:T}$ are first mapped to a latent representation using a sentence encoder $p_{e}(X_{1:N};\Phi_{e})$ and a pose encoder $q_{e}(Y_{1:T};\Psi_{e})$ respectively. These estimate the latent representation or embeddings $z_{x}$ and $z_{y}$ respectively in the embedding space $\mathcal{Z}\subset\mathcal{R}^{h}$,

[TABLE]

$z_{x},z_{y}$ should lie close to each other in $\mathcal{Z}$ as they represent the same concept. To ensure that they do lie close together, a joint translation loss is constructed (refer to Figure 2) and trained end to end with a training curriculum.

4.2 Joint Loss Function

Once we have the embedding $z_{x}$ or $z_{y}$, a pose decoder $q_{d}(;\Psi_{d})$ is used to generate an animation from the joint embedding space $\mathcal{Z}$. The output of the pose decoder must now lie close to the pose sequence $Y_{1:T}$. Hence, using $X_{1:N}$ as inputs and $Y_{1:T}$ as outputs, the cross-modal translation loss is defined as,

[TABLE]

and using $Y_{1:T}$ as inputs and $Y_{1:T}$ as outputs, the uni-modal translation (or autoencoder) loss is defined as,

[TABLE]

where $d(x,y)$ is a function to calculate the distance between the predicted values and ground truth of pose. $\Phi_{e}$, $\Psi_{e}$ and $\Psi_{d}$ are trainable parameters of the sentence encoder, pose encoder and pose decoder respectively.

Combining equations 4 and 5 we get a joint translation loss,

[TABLE]

Jointly optimizing the loss $\mathcal{L}_{j}$ pushes $z_{x}$ and $z_{y}$ closer together improving generalizability and additionally trains the pose decoder which is useful for inference from the joint embedding space.

As $\mathcal{L}_{j}$ is a mutivariate function in $X_{1:N}$ and $Y_{1:T}$, coordinate descent [33] for optimizing the loss function is a natural choice and is described in Algorithm 1.

4.3 Training Curriculum

Cross modal pose forecasting can be a challenging task to train [5]. Starting with simpler examples before moving on to tougher ones can be beneficial to the training process [4, 37, 36].

The curriculum design commonly used for pose forecasting [5] is adapted for our joint model. We first optimize the model to predict 2 time steps conditioned on the complete sentence. This easy task helps the model learn very short pose sequences like leg motions for walking, hand motions for waving and torso motions for bending. Once the loss on the validation set starts increasing, we move on to the next stage in the curriculum. The model is now given twice the amount of poses for prediction. The complexity of the task is increased in every stage till the maximum time-steps ($T$) of prediction is reached. We describe the complete training process in Algorithm 1.

4.4 Optimization

For the distance metric $d(x,y)$ in Equation 4, 5 and 6, Smooth L1 loss (similar to Huber Loss [15]) is used which is defined as,

[TABLE]

In contrast, Lin et. al.[17] uses L2 loss for $d(x,y)$. L2 loss is more sensitive to outliers than L1 loss due to its linearly proportional gradient with respect to the error, while L1 loss has a constant gradient of 1 or -1. But L1 Loss can become unstable when $|x-y|\approx 0$, due to oscillating gradients between 1 and -1. On the other hand, Smooth L1 is continuous and smooth near 0 and more generally for all $x,y\in\mathcal{R}$, hence it is more stable than L1 as a loss function.

5 Experiments

Joint language to pose modeling can be broken down into three core challenges,

Prediction Accuracy by Joint Space: How accurate is pose prediction from the joint embedding ? 2. 2.

Human Judgment: Which of the generated animation is more representative of the input sentence? Does the subjective evaluation correlate with the results from the objective evaluations? 3. 3.

Modeling nuanced language concepts: Is the model able to map nuanced concepts such as speed, direction and action in the generated animations?

Experiments are designed to evaluate these challenges of language grounded pose forecasting.

In the following subsections, the dataset and its pre-processing is briefly discussed which is followed by the evaluation metrics for both objective and subjective evaluations. Finally, design choices of the encoder and decoder models are described which are used to construct the baselines in the final subsection.

5.1 Dataset

Our models are trained an evaluated on KIT Motion-Language Dataset [25] which combines human motion with natural language descriptions. It consists of 3911 recordings (approximately 11.23 hours) which are re-targeted to a kinematic model of a human skeleton with 50 DoFs (6 DoF for the Root joint’s orientation and position, while remaining 44 DoFs for arms legs, head and torso). The dataset also consists of 6278 English sentences (approximately 8 words per sentence) describing the recordings. This is more than the number of recordings as each recording has one or more descriptions which are annotated by human volunteers. We use 20% of the data as a randomly sampled held-out set for evaluating all models.

There is wide variety of motions in this dataset ranging from locomotion (e.g. walking, running, jogging), performing (e.g. playing violin/guitar), and gesticulation (e.g. waving). Many recording have adjectives to further describe the motion like speed (e.g. fast and slow), direction (left, right and forward), and number for periodic motions (e.g. walk 4 steps).

We use the pre-processing steps used in Holden et. al. [14]. All the frames of the motion are transformed such that body always faces the Z-axis. Joint rotation angles are transformed to 3D positions is the skeleton’s local frame of reference with Root as the origin. Root’s position on XZ-plane and orientation along Y-axis is represented as velocity instead of absolute values.

Motion sequences are then sub-sampled to a frequency of 12.5 Hz down from 100Hz. This is low enough to bring enough variance between 2 time-steps for the decoder to train for a regression task, while not compromising on the human’s perception of the animation [5].

5.2 Implementation Details

For pose encoder ($q_{e}$) a network of Gated Recurrent Units (GRUs) [9] is used in our model JL2P. The pose decoder ($q_{d}$) is the same except it has residual connection from the input to the output layer. This is similar to the pose decoder in Lin et. al. [17], except an extra layer to predict the trajectory (or Trajectory Predictor) is discarded in our model.

For langauge encoder ($p_{e}$), a network of Long-Short Term Memory Units (LSTMs) [12] is used. Each token of the sentence is converted into a distributional embedding using a pre-trained Word2Vec model [22]. 111We also train a variant of the model with BERT as the language encoder, but it did not show any significant improvements.

5.3 Baselines

There has been limited work done in the domain of data-driven cross-modal translation from natural language descriptions to pose sequence generation. The closest work to our proposed approach is by **Lin et. al. ** [17]222As we could not find code or pre-trained models for this work, we use our own implementation and training on the same data as all other baselines. As mentioned in Section 2, their model does not follow a training curriculum and uses L2 loss as the loss function. Thier model also maps the language embeddings to an existing embedding space of poses instead of jointly learning it.

We also compare our model **JL2P ** (see Section 4) with three ablations derived from itself. These ablations study the 3 main components of the model, joint embedding space, curriculum learning and Smooth L1 loss:

• •

JL2P w/o Curriculum - Training curriculum in Section 4.3 is dropped.

• •

JL2P w/o L1 - L2 loss is used instead of Smooth L1 loss as the distance metric $d(x,y)$.

• •

JL2P w/o Joint Emb. - Instead of joint training as described in Section 4.2, autoencoder loss $\mathcal{L}_{u}$ minimized first followed by optimization of the cross-translation loss $\mathcal{L}_{c}$.

5.4 Objective Evaluation Metrics

All models are evaluated on the held-out set with a metric Average Position Error (APE). Given a particular joint j, it can be denoted as $\mbox{APE}(j)$,

[TABLE]

where $y_{t}[j]$ is the true location and $\hat{y}_{t}[j]\in\mathcal{Y}$ is the predicted location of joint $j$ at time $t$

Another metric, Probability of Correct Keypoints (PCK) [3, 27], is also used as an evaluation metric.

5.5 User Study: Subjective Evaluation Metric

Joint language to pose generation is a subjective task, hence a human’s subjective judgment on the quality of prediction is an important metric for this task.

To achieve this, we design a user study which asked human annotators to rank two videos generated by 2 different models but with same sentence as the input. One of the videos is generated by Lin et. al. and the other is either ground truth or generated by JL2P , JL2P w/o Curriculum, JL2P w/o Joint Emb., or JL2P L1. The annotators answer the following question for each pair of videos, Which of the 2 generated animations is better described by ”$<$sentence$>$”?. To ensure that annotators spend enough time to decide, any annotations which took less than 20 seconds333each video is 8 seconds long at an average. We set a threshold of 20 seconds to give annotators 4 seconds to make their decision. were rejected. This study subjectively evaluates the preference of humans for generated animations by different models.

6 Results and Discussion

In this section we first use objective measures and then conduct a user study to get a subjective evaluation. Finally, we probe some qualitative examples to understand the effectiveness of the model in tackling the core challenges described in Section 5.

6.1 Prediction Accuracy by Joint Space

JL2P demonstrates at least a 9% improvement over Lin et. al. (see Table 1) for all joints. The maximum improvement around 15% is seen in the Root joint. Errors in Root prediction can lead to a ”sliding-effect” of the feet; when the generation is translating faster than the frequency of the feet. Improvements in APE scores for long-term prediction, especially for Root, can help get rid of these artifacts in the generated animation.

When compared to its variants, JL2P loses maximum APE value when it is trained without curriculum (or JL2P w/o Curriculum). As discussed in Section 4.3, learning to predict shorter sequences before moving on to longer ones proves beneficial for pose generation. APE scores go down by 4%, if L2 loss is used instead of Smooth L1. In an output space as diverse as pose sequences, it becomes important to ignore outliers which may drive model to overfit. APE scores go down only by 1%, if the embedding space is not trained with the joint loss $\mathcal{L}_{j}$

APE values across time for JL2P of different parts of the body (Root, Legs, Arms, Torso and Head) are plotted in Figure 6. Root’s APE scores have the fastest rate of increase, followed by Arms, Legs and then Head, Torso. Two out of three coordinates of Root are represented as velocity which accumulates errors when integrated back to absolute positions; this is probably a contributing factor to the rapid increase of prediction error over time.

Our final objective metric is PCK. PCK values (for 35$\leq\sigma\leq$55) on generated animations are compared among JL2P , its variants and Lin et. al. in Figure 7. JL2P and its ablations show a consistent improvement over Lin et. al. which further strengthen the claim about the prediction accuracy by our model’s joint space.

6.2 Human Judgment

Human judgment is quantified by preference scores in Figure 5. Human preference of all our baseline models and ground truth are compared against Lin et. al. animations. JL2P has a preference of 75% which is shy of ground truth by 10%. Preference scores consistently drop with all the other variants of JL2P .

JL2P w/o Joint Emb. has the lowest preference score of 60% when ranked against Lin et. al. . It is still more preferred than Lin et. al. but far more unlikely to be picked when pitted against JL2P . This is an interesting change in trend, as removing joint loss from JL2P did not affect the objective scores significantly, but have lowered its human preference by a significant fraction. This leads us to conclude that objective metrics are not enough to judge the performance of a model. Instead a combination of human judgment and objective metrics is necessary for evaluating pose generation models.

6.3 Modeling nuanced language concepts

Root joint decides the trajectory of the animation which is crucial for translating concepts like speed (e.g. fast, and slow), direction (e.g. left, right, forward and backward) from natural language to animation. We plot the trajectories generated by JL2P, ground truth and Lin et. al. for different sentences in Figure 3.

Modeling direction: Animations’ trajectory for these sentences for JL2P is similar to that of the ground truth trajectories. In contrast, Lin et. al.’s trajectories tend to be semantically incorrect and have a slightly curved forward motion for these sentences.

Modeling speed: In the sentence, ”A person runs very fast forward”, JL2P is able to understand that the animation has to move faster. It is able to walk approximately the same distance as the ground truth in the same amount of time, hence has the same speed. In contrast, even though Lin et. al.’s motion is in the forward direction, it is not able to maintain the same speed as required by the sentence.

**Modeling actions: ** In figure 4, we plot animations generated by a diverse set of sentences. JL2P is able to understand the action from the sentences, and is able to generate an animation corresponding to the action. JL2P is able to handle many actions ranging from kneeling (with complex leg motions) to jogging (with periodic hand and leg motion).

We show, via qualitative examples, that our model JL2P is able to model nuanced language concepts which are then reproduced in the animations generated at inference time.

7 Conclusions

In this paper, we proposed a neural architecture called Joint Language-to-Pose (or JL2P), which integrates language and pose to learn a joint embedding space in an end-to-end training paradigm. This embedding space can now be used to generate animations conditioned on an input description. We also proposed the use of curriculum learning approach which forces the model to generate shorter sequences before moving on to longer ones. We evaluated our proposed model on a parallel corpus of 3D pose data and human-annotated sentences with objective metrics to measure prediction accuracy, as well as with a user study to measure human judgment. Our results confirm that our approach, to learn a joint embedding in a curriculum learning paradigm by JL2P, was able to generate more accurate animations and are deemed more visually represented by humans than the state-of-the-art model.

8 Acknowledgements

This material is based upon work partially supported by the National Science Foundation (Award #1750439) and Oculus VR. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of National Science Foundation or Oculus VR. No official endorsement should be inferred.