Embracing Aleatoric Uncertainty: Generating Diverse 3D Human Motion

TL;DR

This paper introduces Diverse-T2M, a novel method for text-to-3D human motion generation that incorporates aleatoric uncertainty to produce diverse, semantically consistent motions, advancing the state-of-the-art in diversity and quality.

Contribution

It presents a transformer-based approach that explicitly models uncertainty using noise signals and a stochastic latent space sampler, improving diversity in text-to-motion generation.

Findings

Significantly improves motion diversity on benchmark datasets.

Maintains high text-motion semantic consistency.

Outperforms existing methods in diversity metrics.

Abstract

Generating 3D human motions from text is a challenging yet valuable task. The key aspects of this task are ensuring text-motion consistency and achieving generation diversity. Although recent advancements have enabled the generation of precise and high-quality human motions from text, achieving diversity in the generated motions remains a significant challenge. In this paper, we aim to overcome the above challenge by designing a simple yet effective text-to-motion generation method, \textit{i.e.}, Diverse-T2M. Our method introduces uncertainty into the generation process, enabling the generation of highly diverse motions while preserving the semantic consistency of the text. Specifically, we propose a novel perspective that utilizes noise signals as carriers of diversity information in transformer-based methods, facilitating a explicit modeling of uncertainty. Moreover, we construct a latent space where text is projected into a continuous representation, instead of a rigid one-to-one mapping, and integrate a latent space sampler to introduce stochastic sampling into the generation process, thereby enhancing the diversity and uncertainty of the outputs. Our results on text-to-motion generation benchmark datasets~(HumanML3D and KIT-ML) demonstrate that our method significantly enhances diversity while maintaining state-of-the-art performance in text consistency.

Tables

Table 1. TABLE I: Comparison with the state-of-the-art methods on HumanML3D  [ 47 ] test set. We compute standard metrics following Guo et al.  [ 47 ] . For each metric, we repeat the evaluation 20 times and report the average with 95% confidence interval. § reports results using ground-truth motion length. Bold face indicates the best result, while blue refers to the second best.

Methods	MultiModality$\uparrow$	R Precision$\uparrow$			FID$\downarrow$	MultiModal Dist$\downarrow$
		Top 1	Top 2	Top 3
Seq2Seq [48]	-	${0.180}^{\pm .002}$	${0.300}^{\pm .002}$	${0.396}^{\pm .002}$	${11.75}^{\pm .035}$	${5.592}^{\pm .007}$
Language2Pose [11]	-	${0.246}^{\pm .002}$	${0.387}^{\pm .002}$	${0.486}^{\pm .002}$	${11.02}^{\pm .046}$	${5.296}^{\pm .008}$
Text2Gesture [49]	-	${0.165}^{\pm .001}$	${0.267}^{\pm .002}$	${0.345}^{\pm .002}$	${5.012}^{\pm .030}$	${6.030}^{\pm .008}$
Hier [31]	-	${0.301}^{\pm .002}$	${0.425}^{\pm .002}$	${0.552}^{\pm .004}$	${6.532}^{\pm .024}$	${5.012}^{\pm .018}$
MoCoGAN [50]	${0.019}^{\pm .000}$	${0.037}^{\pm .000}$	${0.072}^{\pm .001}$	${0.106}^{\pm .001}$	${94.41}^{\pm .021}$	${9.643}^{\pm .006}$
Dance2Music [1]	${0.043}^{\pm .001}$	${0.033}^{\pm .000}$	${0.065}^{\pm .001}$	${0.097}^{\pm .001}$	${66.98}^{\pm .016}$	${8.116}^{\pm .006}$
TEMOS§	${0.038}^{\pm .018}$	${0.424}^{\pm .002}$	${0.612}^{\pm .002}$	${0.722}^{\pm .002}$	${3.734}^{\pm .028}$	${3.703}^{\pm .008}$
TM2T [12]	${2.424}^{\pm .093}$	${0.424}^{\pm .003}$	${0.618}^{\pm .003}$	${0.729}^{\pm .002}$	${1.501}^{\pm .017}$	${3.467}^{\pm .011}$
T2M [39]	${2.219}^{\pm .074}$	${0.455}^{\pm .003}$	${0.636}^{\pm .003}$	${0.736}^{\pm .002}$	${1.087}^{\pm .021}$	${3.347}^{\pm .008}$
MDM [51]	${2.799}^{\pm .072}$	-	-	${0.611}^{\pm .007}$	${0.544}^{\pm .044}$	${5.566}^{\pm .027}$
MLD [29]	${2.413}^{\pm .079}$	${0.481}^{\pm .003}$	${0.673}^{\pm .003}$	${0.772}^{\pm .002}$	${0.473}^{\pm .013}$	${3.196}^{\pm .010}$
MotionDiffuse [16]	${1.553}^{\pm .042}$	${0.491}^{\pm .001}$	${0.681}^{\pm .001}$	${0.782}^{\pm .001}$	${0.630}^{\pm .001}$	${3.113}^{\pm .001}$
T2M-GPT [15]	${1.831}^{\pm .048}$	${0.492}^{\pm .003}$	${0.679}^{\pm .002}$	${0.775}^{\pm .002}$	${0.141}^{\pm .005}$	${3.121}^{\pm .009}$
ReMoDiffuse [17]	${1.795}^{\pm .043}$	${0.510}^{\pm .005}$	${0.698}^{\pm .006}$	${0.795}^{\pm .004}$	${0.103}^{\pm .004}$	${2.974}^{\pm .016}$
MoMask [14]	${1.241}^{\pm .040}$	${0.521}^{\pm .002}$	${0.713}^{\pm .002}$	${0.807}^{\pm .002}$	${0.045}^{\pm .002}$	${2.958}^{\pm .008}$
MotionLCM [52]	${2.259}^{\pm .092}$	${0.502}^{\pm .003}$	${0.698}^{\pm .002}$	${0.798}^{\pm .002}$	${0.304}^{\pm .012}$	${3.012}^{\pm .007}$
Motion Mamba [53]	${2.294}^{\pm .058}$	${0.502}^{\pm .003}$	${0.693}^{\pm .002}$	${0.792}^{\pm .002}$	${0.281}^{\pm .009}$	${3.060}^{\pm .058}$
Ours	${3.976}^{\pm .155}$	${0.525}^{\pm .002}$	${0.717}^{\pm .003}$	${0.809}^{\pm .002}$	${0.057}^{\pm .003}$	${2.941}^{\pm .010}$

Table 2. TABLE II: Comparison with the state-of-the-art methods on KIT-ML  [ 54 ] test set. We compute standard metrics following Guo et al.  [ 47 ] . For each metric, we repeat the evaluation 20 times and report the average with 95% confidence interval. § reports results using ground-truth motion length. Bold face indicates the best result, while blue refers to the second best.

		R Precision$\uparrow$
Methods	MultiModality$\uparrow$	Top 1	Top 2	Top 3	FID$\downarrow$	MultiModal Dist$\downarrow$
Seq2Seq [48]	-	${0.103}^{\pm .003}$	${0.178}^{\pm .005}$	${0.241}^{\pm .006}$	${24.86}^{\pm .348}$	${7.960}^{\pm .031}$
Language2Pose [11]	-	${0.221}^{\pm .005}$	${0.373}^{\pm .005}$	${0.483}^{\pm .005}$	${6.545}^{\pm .072}$	${5.147}^{\pm .030}$
Text2Gesture [49]	-	${0.178}^{\pm .005}$	${0.255}^{\pm .004}$	${0.338}^{\pm .005}$	${12.12}^{\pm .183}$	${6.964}^{\pm .029}$
Hier [31]	-	${0.255}^{\pm .006}$	${0.432}^{\pm .007}$	${0.531}^{\pm .007}$	${5.203}^{\pm .107}$	${4.986}^{\pm .027}$
MoCoGAN [50]	${0.250}^{\pm .009}$	${0.022}^{\pm .002}$	${0.042}^{\pm .003}$	${0.063}^{\pm .003}$	${82.69}^{\pm .242}$	${10.47}^{\pm .012}$
Dance2Music [1]	${0.062}^{\pm .002}$	${0.031}^{\pm .002}$	${0.058}^{\pm .002}$	${0.086}^{\pm .003}$	${115.4}^{\pm .240}$	${10.40}^{\pm .016}$
TEMOS§	${0.532}^{\pm .018}$	${0.353}^{\pm .002}$	${0.561}^{\pm .002}$	${0.687}^{\pm .002}$	${3.717}^{\pm .028}$	${3.417}^{\pm .008}$
TM2T [12]	${3.292}^{\pm .081}$	${0.280}^{\pm .005}$	${0.463}^{\pm .006}$	${0.587}^{\pm .005}$	${3.599}^{\pm .153}$	${4.591}^{\pm .026}$
T2M [39]	${2.052}^{\pm .107}$	${0.361}^{\pm .005}$	${0.559}^{\pm .007}$	${0.681}^{\pm .007}$	${3.022}^{\pm .107}$	${3.488}^{\pm 028}$
MDM [51]	${1.907}^{\pm .214}$	-	-	${0.396}^{\pm .004}$	${0.497}^{\pm .021}$	${9.191}^{\pm .022}$
MLD [29]	${2.192}^{\pm .071}$	${0.390}^{\pm .008}$	${0.609}^{\pm .008}$	${0.734}^{\pm .007}$	${0.404}^{\pm .027}$	${3.204}^{\pm .027}$
MotionDiffuse [16]	${0.730}^{\pm .013}$	${0.417}^{\pm .004}$	${0.621}^{\pm .004}$	${0.739}^{\pm .004}$	${1.954}^{\pm .062}$	${2.958}^{\pm .005}$
T2M-GPT [15]	${1.570}^{\pm .039}$	${0.416}^{\pm .006}$	${0.627}^{\pm .006}$	${0.745}^{\pm .006}$	${0.514}^{\pm .029}$	${3.007}^{\pm .023}$
ReMoDiffuse [17]	${1.239}^{\pm .028}$	${0.427}^{\pm .014}$	${0.641}^{\pm .004}$	${0.765}^{\pm .055}$	${0.155}^{\pm .006}$	${2.814}^{\pm .012}$
MoMask [14]	${1.131}^{\pm .043}$	${0.433}^{\pm .007}$	${0.656}^{\pm .005}$	${0.781}^{\pm .005}$	${0.204}^{\pm .011}$	${2.779}^{\pm .022}$
Motion Mamba [53]	${1.678}^{\pm .064}$	${0.419}^{\pm .006}$	${0.645}^{\pm .005}$	${0.765}^{\pm .006}$	${0.307}^{\pm .041}$	${3.021}^{\pm .025}$
Ours	${3.462}^{\pm .122}$	${0.437}^{\pm .005}$	${0.654}^{\pm .005}$	${0.774}^{\pm .005}$	${0.257}^{\pm .013}$	${2.801}^{\pm .024}$

Table 3. TABLE III: Ablation study about variable text-to-codes predictor (VP) and the noise signal (NS) on HumanML3D test set. We compute standard metrics following Guo et al.  [ 47 ] . For each metric, we repeat the evaluation 20 times and report the average with 95% confidence interval.

		R Precision$\uparrow$
Methods	MultiModality$\uparrow$	Top 1	Top 2	Top 3	FID$\downarrow$	MultiModal Dist$\downarrow$
Baseline (VQ)	${1.130}^{\pm .043}$	${0.505}^{\pm .002}$	${0.698}^{\pm .002}$	${0.794}^{\pm .003}$	${0.089}^{\pm .005}$	${3.043}^{\pm .008}$
+ VP	${1.904}^{\pm .060}$	${0.509}^{\pm .002}$	${0.700}^{\pm .003}$	${0.795}^{\pm .002}$	${0.085}^{\pm .003}$	${3.040}^{\pm .009}$
+ NS	${4.047}^{\pm .146}$	${0.510}^{\pm .003}$	${0.702}^{\pm .002}$	${0.797}^{\pm .002}$	${0.083}^{\pm .003}$	${3.031}^{\pm .010}$
Baseline (RVQ)	${1.221}^{\pm .040}$	${0.518}^{\pm .002}$	${0.714}^{\pm .002}$	${0.807}^{\pm .002}$	${0.058}^{\pm .002}$	${2.962}^{\pm .008}$
+ VP	${1.891}^{\pm .055}$	${0.524}^{\pm .002}$	${0.716}^{\pm .002}$	${0.810}^{\pm .002}$	${0.054}^{\pm .002}$	${2.952}^{\pm .008}$
+ NS	${3.976}^{\pm .155}$	${0.525}^{\pm .002}$	${0.717}^{\pm .003}$	${0.809}^{\pm .002}$	${0.057}^{\pm .003}$	${2.941}^{\pm .010}$

Equations

$${\hat{z}}_i=\underset{c_k\in C}{\mathrm{argmin}}{\Vert z_i-c_k\Vert }_2,$$

$${\hat{r}}_i^v=\underset{c_k^v\in C^v}{\mathrm{argmin}}{\Vert r_i^v-c_k^v\Vert }_2,r_i^0=z_i$$

$$r_i^{v+1}=r_i^v-{\hat{r}}_i^v,v=0,1,\dots ,V.$$

$$P(m_t=c_k)=p_k^t,k=1,.,K.$$

$$I=(1+w)I_{\text{text}}-w{I}_{\text{noise}},$$

$$I$$

$$z\sim p(z\mid g).$$

$$q_{\varphi }(z\mid e)$$

$$(\mu ,log{\sigma }^2)=EncoderNet(e),$$

$$q_{\varphi }(z\mid e)=\mathcal{N}\left(z;\mu ,diag\left({\sigma }^2\right)\right),$$

$$z=\mu +\sigma \odot ϵ,ϵ\sim \mathcal{N}(0,I),$$

$${\mathcal{L}}_{mdr}=\Vert M-\hat{M}{\Vert }_1+\beta \sum _{v=1}^V{\Vert R^v-\mathrm{sg}\left[{\hat{R}}^v\right]\Vert }_2^2$$

$$\mathcal{L}={\mathcal{L}}_{\text{tran }}+{\mathcal{L}}_{\text{kl}},$$

$${\mathcal{L}}_{\text{tran }}={\mathbb{E}}_{\hat{M}\sim p(\hat{M})}[-\log p(\hat{M}\mid s)],$$

$$p(\hat{M}\mid s)=\prod _{i=1}^{T/l}\left(p\left({\hat{m}}_t\mid s\right)\cdot \left(1-[mask{]}_t\right)+[mask{]}_t\right),$$

$${\mathcal{L}}_{\text{kl}}=\frac{1}{D}\sum _{i=1}^{D}KL\left(\mathcal{N}({\mu }_i,{\sigma }_i),\mathcal{N}(0,1)\right),$$

Full text

I Introduction

Human motion generation has extensive practical applications, such as gaming, film production, and AR/VR content creation. Generative models for human motion synthesis have been developed to automate the labor-intensive and time-consuming process of character animation, with the aim of reducing costs and enhancing development efficiency. Recently, there have been notable advancements in human motion generation with various conditional inputs, such as music [1, 2, 3], control signals [4, 5, 6], action categories [7, 8], and natural language descriptions [9, 10, 11, 12, 13]. These developments offer a more intuitive and user-friendly method for animating virtual characters. Among all conditional modalities, text-based human motion generation (text-to-motion generation, T2M) has become a dominant focus of research, as linguistic descriptors provide a convenient and natural interface for human-computer interaction.

Existing works in this field mainly follow either the two-stage [14, 15] or diffusion-based [16, 17] paradigm. The former two-stage method involves: (i) conduct motion discrete representation, e.g., T2M-GPT [15] and Att-T2M [18] utilize Vector Quantized Variational Autoencoders (VQ-VAE) [19], while MoMask [14] and T2M-HiFiGPT [20] introduce variants,i.e., residual VQVAE [21, 22];

(ii)) predict text-to-motion sequence, e.g., generative pre-trained transformers (GPT) [23] and mask generative transformers [24] are employed for text-to-code sequence prediction and then decoded into motion sequences based on the codebook and decoder from the first stage. Additionally, given the remarkable success of diffusion-based generative models in other domains [25, 26], the latter paradigm employs conditional diffusion models for human motion generation, including MDM [27], MotionDiffuse [16], ReMoDiffuse [17], FineMoGen [28], and MLD [29]. However, diffusion-based paradigm directly learn text-driven generation in more complex pose space, which is challenging. In complex pose spaces, data distribution becomes sparse, making it difficult for models to capture the underlying structure of the data. This complexity hinders the learning of the correspondence between text and motions and the slight changes may lead to visible defects in generated motion. Considering fidelity, training efficiency, the two-stage methods to discrete representation of motion before learning to generation ability receives more attention.

To effectively apply text-driven motion generation, two key requirements need to be met: text-motion consistency and generation diversity. The text provided by the user does not describe all the details in the motion. For the more loosely controlled parts, there should be diverse and reasonable generation results, while maintaining consistency of textual description. For example, as highlighted by the purple boxes in Figure 2 (b), there are multiple plausible postures that are consistent with the textual instruction “moves arm as if to wipe something”, such as standing, leaning, or bending. On the one hand, this can enhance the visual richness and dynamism of application scenes, preventing homogenization. On the other hand, diverse generations provides a broader array of foundational options for animators to perform secondary processing, thereby expanding creative possibilities and improving the overall quality of the animations.

However, we observe that although recent two-stage methods enable the generation of precise and high-quality human motions controlled by text, they still struggle to achieve the diversity mentioned above. In light of this, we conduct an in-depth analysis. Specifically, we find that the ability of the two-stage approach to produce high-quality motions stems from discretizing real motions into a large set of basis motions. These basis motions can then be selected and permuted to compose a variety of complex motion sequences. Because these basis motions are derived from real motion data, the authenticity of the decoded generated motions is preserved, avoiding visual defects such as gestures that violate common sense or distortions. Additionally, these basis motions can be decoupled into spatial positions and body poses. As shown in Figure 2 (a), each code in the VQ-VAE codebook represents a human pose. Transformers are then used to learn a one-to-one mapping between text inputs and code sequences. By introducing small perturbations to the final generated result, different motion outputs can be obtained. However, while this process ensures strong semantic consistency with the text, its deterministic nature limits the diversity of generated motions. As Heraclitus famously said, “Panta Rhei” (everything flows), different individuals—and even the same individual at different times—exhibit distinct sequences of motions. Therefore, incorporating more uncertainty into the generation process can enhance generation diversity, allowing for more realistic results.

In this paper, we propose a simple yet effective T2M method, i.e., Diverser-T2M, to generate highly diverse motions while maintaining semantic consistency of the text. As illustrated in Figure 1, our Diverser-T2M significantly enhances diversity while maintaining state-of-the-art performance in text consistency. Our Diverser-T2M follows the two-stage paradigm, in which employ RVQ-VAE to conduct the motion discrete representation and learn the text-to-code sequence prediction with mask transformer. To enhance the generative power of the model, we introduce uncertainty to the generation process. Since diversity requires a carrier of additional information, we propose using noise as this carrier. Specifically, instead of only using text description as the condition signal, we incorporate a noise signal to inject uncertainty. Each inference leverages different noise signals to introduce diverse sources of information. Moreover, the text-motion correspondence is not one-to-one, each text description should correspond to a broad solution space and be mapped to a distribution. To address this, we design a variational text-to-codes predictor by constructing a latent space where each text corresponds to a distribution. We also integrate a latent space sampler into our predictor, allowing for sampling from this distribution during inference, which significantly enhances diversity. Extensive experiments on two benchmark datasets (HumanML3D and KIT-ML) demonstrate that our proposed Diverse-T2M outperforms the previous state-of-the-art methods at various metrics. It is worth noting that the Multimodality metric, which reflects diversity, is 30% higher compared to the second place.

The main contribution of this work can be summarized as follows:

• •

We analyze the potential reasons for the lack of diversity in existing methods, and propose a simple yet effective text-to-motion generation method, Diverse-T2M. Our method introduces uncertainty into the generation process, enabling the generation of highly diverse motions while preserving the semantic consistency of the text.

• •

We propose a novel perspective that utilizes noise signals as carriers of diversity information in transformer-based methods, facilitating a explicit modeling of uncertainty.

• •

We construct a latent space that projects text into a continuous representation rather than a rigid one-to-one mapping, and incorporate a latent space sampler to introduce stochasticity into the generation process, thereby enhancing both the diversity and uncertainty of the outputs.

• •

Experimental results show that our proposed Diverser-T2M significantly enhances diversity while maintaining state-of-the-art performance in text consistency. And demonstrated commercial value and potential for non-human character-animated.

The remainder of the paper is organized as follows. Section II briefly reviews related work on human motion generation. Subsequently, we present the technical details of the proposed method in Section III. The experimental results are presented in Section IV. Finally, we conclude this paper in Section V.

II Related Work

II-A Text-driven human motion generation.

Due to the user-friendly and convenient language input, text-to-motion is one of the most significant motion generation tasks. Text-to-motion is a burgeoning field focused on generating 3D human motions driven by textual descriptions. Text2Action [30] utilize an RNN-based model to generate motions conditioned on brief texts. Language2Pose  [11] further improve by learning a joint embedding space of text and pose, which is learned end-to-end using a curriculum learning approach. It emphasizes shorter and easier sequences first before moving to longer and harder ones and eventually enables the generation of motion sequences from text embeddings. [31] propose a hierarchical two-stream sequential model to explore a finer joint-level mapping between natural language sentences and 3D pose sequences corresponding to the given motion. MotionCLIP [27] and ohMG [32] model text-to-motion in an unsupervised manner using the large pretrained CLIP [33] model, which has been widely used in the multimodal field [34, 35, 36, 37, 38]. T2M [39] extended the temporal VAE to learn the probabilistic mapping between texts and motions and it introduce a large-scale dataset HumanML3D, which is marked a significant advancement. ACTOR [33] design a class-agnostic transformer VAE for generating actions from predefined classes in a non-autoregressive fashion. TEMOS [40] added a text encoder and produced more diverse action sequences and TEACH [41] expandeds by generating sequences of actions from multiple natural language descriptions. TM2T [12] combines text-to-motion and motion-to-text tasks, allowing for a more comprehensive understanding of both motion and text modalities. MotionGPT [42] treats human motion as a foreign language and leverages language understanding and zero-shot transfer abilities of pre-trained language models.

Recently, emerging two-stage approaches and diffusion models based have dramatically changed the field of motion generation. Regarding the former two-stage methods, they contain two components: 1) motions are firstly discretized as tokens via vector quantization [19] and 2) then predict the code sequence of motion. For the first stage, recent works such as T2M-GPT [15] and Att-T2M [18] utilize the Vector Quantized Variational Autoencoders (VQVAE) [19] for motion discrete representation. Additionally, MoMask [14] and T2M-HiFiGPT [20] employ variants like residual VQVAE [21, 22] for their approaches. For the second stage, T2M-GPT [15] introduce Generative Pretrained Transformer (GPT) for text-to-code sequence prediction. MoMask [14] improve it by employ Masked Transformer for code sequence generation conditioned by text. Given the remarkable success of diffusion-based generative models in other domains [25, 26], the former method employ a conditional diffusion model for human motion generation by MDM [27], MotionDiffuse [16], ReMoDiffuse [17], FineMoGen [28] and MLD [29]. This approach aims to learn a robust probabilistic mapping from textual descriptors to motion sequences. However, diffusion-based models often exhibit lower motion-text matching accuracy and slower inference speed due to the iterative nature of diffusion processes. Considering fidelity, training efficiency, the two-stage methods has attract more attention. Although the two-stage approach can achieve high fidelity and semantic consistency, the generation process lacks stochasticity, leading to reduced diversity in the generated results. In contrast, our Diverse-T2M is able to generate diverse motion sequences while maintaining a high level of semantic consistency with the accompanying text.

II-B Motion Discrete Representation.

Motion data is important in the development of motion synthesis tasks. The presentation of motion data in the generative algorithm have the straightforward joint positions and the Master Motor Map (MMM) format. For mainstream datasets, i.e., HumanML3d and KIT, they employ two motion representations: 1) the classical SMPL-based [43, 44, 45] motion parameters and 2) the redundant hand-crafted motion feature [39, 46] with a combination of joints features. In order to generate actions with plausibility and fidelity, some work has discretized real actions into a number of representations and used the discretized representations to de-combine them into complete action sequences. Recent works such as T2M-GPT [15] and Att-T2M [18] utilize the Vector Quantized Variational Autoencoders (VQVAE) [19] for motion discrete representation. Additionally, MoMask [14] and T2M-HiFiGPT [20] employ variants like residual VQVAE [21, 22] for their approaches.

III Proposed Method

III-A Overview of Diverse-T2M

Notation. Text-to-motion generation aims to generate 3D human motions from text descriptions. As shown in Figure 3, given the text $s$ as conditional information, it is expected to generate a motion sequence $M=\{m_{t}\}_{t=1}^{T}$ consistent with the text $s$, where $T$ denotes the motion length. Here, $m_{t}$ represents the pose at time $t$, which contains information about the angular and linear velocities of the body’s root, as well as the positions, velocities, and rotations of the local joints in the root space. We denote the text-to-motion generation datasets by $\mathcal{X}=\{s_{i},M_{i})\}_{i=1}^{N}$, which is the collection of $N$ semantically corresponding text-motion pairs. Our Diverse-T2M follows a simple and classic two-stage paradigm [14, 15, 12], which involves following two stages:

• •

Stage 1: Motion Discrete Representation. We learn discrete representations for motion by learning the mapping between the motion space and a discretized feature space, i.e., a code space, where each code represents a discretized feature. In particular, it consists of three key components: the codebook $C$, which defines the discrete representation space; the encoder $\mathcal{E}$, which maps from the motion space to the discretized feature space; and the decoder $\mathcal{D}$, which performs the reverse mapping. With a well-learned discrete representation, any motion can be transformed into a sequence of code that represents discretized features with feature extraction by $\mathcal{E}$ and quantization by $C$ (Section III-B1).

• •

Stage 2: Text-to-codes Predication. After the discrete representation of motion, we directly focus on the process of generating code sequence in the discretized feature space conditional on text, which are then mapped to a motion sequence via the aforementioned trained decoder $\mathcal{D}$. With a discretized feature space as the medium, the text-to-motion generation process is completed.

Due to individual differences, each person’s behavioral patterns are unique, and even the same individual will not exhibit exactly the same behavior repeatedly. Therefore, a given text description should correspond to various motions, creating a broad solution space for the text. Despite some recent two-stage methods enable the generation of precise and high-quality human motions controlled by text, they still struggle to achieve generation diversity. To address this challenge, we introduces uncertainty into the generation process, enabling the generation of highly diverse motions while preserving the semantic consistency of the text. Specifically, we propose a novel perspective that utilizes noise signals as carriers of diversity information in transformer-based methods, i.e., conducts motion generation under noise and text conditions (see Section III-B2). Moreover, we construct a latent space to map text to a data distribution, replacing the one-to-one mapping, and integrate a latent space sampler to introduce random sampling into the generation process, further enhancing the uncertainty within the generation (see Section III-C).

III-B Diverse-T2M

III-B1 Motion Discrete Representation

In this section, we aim to apply the VQ-VAE [19] to learn discrete representations, which are beneficial for generative models. As shown in Figure 3-top, VQ-VAE aims to reconstruct the motion sequence using an auto-encoder and a learnable $K$-size codebook $C=\{c_{k}\}_{k=1}^{K}$, where each code $k$ corresponds to a discretized feature $c_{k}\in\mathbb{R}^{d_{c}}$ and $d_{c}$ represents the discretized feature dimension. In detail, given the motion sequence $M=\{m_{t}\}_{t=1}^{T}$, the encoder $\mathcal{E}$ of the autoencoder encodes it into a latent feature sequence denoted as $Z=\{z_{i}\}_{i=1}^{T/l}$ where $z_{i}\in\mathbb{R}^{d_{c}}$ and $l$ is the temporal downsampling rate of $\mathcal{E}$. For each latent feature $z_{i}$, the quantization through $C$ is to find the most similar element in $C$ and form $\hat{Z}=\{\hat{z}_{i}\}_{i=1}^{T/l}$ as follows:

[TABLE]

where $\hat{z}_{i}$ represents the quantized feature of $z_{i}$. Finally, the quantized feature sequence $\hat{Z}$ is then decoded back into motion sequence $M$ by the decoder $\mathcal{D}$.

To reduce information loss and improve reconstruction quality, we implement residual quantization in motion reconstruction following [21]. The objective is to approximate each feature $\bm{z}_{i}$ as closely as possible with $\hat{z}_{i}$ by modeling the residuals between them, i.e., $r_{i}=z_{i}-\hat{z}_{i}$. Concretely, we set $V$ layer residual quantization with $V$ codebooks $\{C^{v}\}_{v=0}^{V}$. We recursively conduct residual quantization as follows:

[TABLE]

where $r_{i}^{v}$ and $\hat{r}_{i}^{v}$ represent the $v$-th layer’s residual feature and quantized residual feature respectively. After residual quantization, the final approximation of latent sequence $z_{i}$ is the sum of all quantized sequences $\sum_{v=0}^{V}\hat{r}^{v}_{i}$, which is then fed into decoder $\mathcal{D}$ for motion reconstruction.

III-B2 Motion Generation under Noise and Text Conditions

Distinguishing from motion generation conditioned only on text, we include noise as one of the conditions for motion generation as well, increasing the uncertainty of the generation, and thus improving the diversity.

Particularly, instead of training a text-conditional model $p_{\theta}(z|s)$ alone, we simultaneously train a noise-conditional model $p_{\theta}(z|\epsilon)$ alongside it. A variational text-to-codes predictor (Section III-C) is used to parameterize both models, $p_{\theta}(z|s)$ and $p_{\theta}(z|\epsilon)$. Similar to the classifier-free diffusion guidance approach, while training separate models for $p_{\theta}(z|s)$ and $p_{\theta}(z|\epsilon)$ is possible, joint training is simpler to implement, does not complicate the training process, and avoids increasing the total number of parameters. The noise-conditional and text-conditional models are trained jointly by fusion the text-conditional signal $s$ with the noise-conditional signal with a certain probability $p_{\text{noise}}$, which is treated as a hyperparameter.

Specifically, given the text $s$, we use CLIP to extract the text feature as text signal $g_{s}$. We sample a noise signal $g_{\epsilon}$ from Gaussian noise that is the same size as the text signal. The text signal and noise signal are then fed into the variational text-to-motion predictor separately to obtain codes probability distribution $I_{\text{text}},I_{\text{noise}}\in\mathbb{R}^{T\times K}$, where $T$ and $K$ represent the predicted motion length and size of the codebook respectively. At timestamp $t$, the probability distribution of the motion codes is as follows:

[TABLE]

Similar to classifier-free diffusion guidance, we fuse the probability distributions obtained from the text signal and the noise signal as follows:

[TABLE]

$w$ is the fusion weight, which can balance the diversity and semantic consistency with text description. Then, we select the code sequence $\{c_{i}\}_{i=1}^{T/l}$ that matches the text signal through the probability distribution $I$.

Once the code sequence consistent with the given text description is obtained, it is translated into the motion feature sequence $\{f_{i}\}_{i=1}^{T/l}$ using the codebook. Finally, a well-trained decoder converts the motion feature sequence into the motion sequence $\hat{M}=\{m_{t}\}_{t=1}^{T}$.

Length Stochastic Enhancement. The text-to-motion generation datasets $\mathcal{X}=\{s_{i},M_{i})\}_{i=1}^{N}$ collects $N$ semantically corresponding text-motion pairs. Although text and motion are semantically corresponding in each pair, the motion in each pair must not be the unique solution to text. Each text has many motions corresponding to it, and this is also true in terms of the motion length. To increase data diversity during training, we randomly stretch and compress the motion sequences in equal proportions. Considering that some text includes speed descriptions such as ’fast’ and ’slow,’ we avoid data conflicts from data augmentation by only enhancing the length of motions corresponding to noisy signals during training. This approach maintains semantic consistency while enhancing data diversity.

III-C Variational Text-to-codes Predictor

The input to the variational text-to-codes predictor is the signal feature $g$, which can be either a text signal $g_{s}$ or a noise signal $g_{\epsilon}$. The output is the codes probability distribution, specifically $I_{\text{text}}$ for $g_{s}$ and $I_{\text{noise}}$ for $g_{\epsilon}$, both belong to $\mathbb{R}^{T\times K}$.

For the traditional transformer-based generative models in two-stage methods [14, 15], it construct one-to-one “text-to-codes” prediction for the $(g,I)$ pairs as follows:

[TABLE]

where $f_{P}$ is implemented with the Generative Pretrained Transformer or mask transformer. We use the mask transformer here.

To encourage greater diversity, we integrate latent space sampler within our variational text-to-codes predictor, aimed at enhancing expressive diversity, as shown in Figure 4. In details, we construct a latent space and map signal feature into this latent space. Within this latent space, each signal feature corresponds to a data distribution rather than a fixed vector, as follows:

[TABLE]

Specifically, as shown in 4, for a given signal feature $g$ during training, we concat it with its corresponding masked motion tockens and input into $N$ transformer layers. Then obtain the fusion feature $e$. Then latent space sampler predicts a distribution $p(z\mid e)$ based on $e$, from which a latent embedding $z$ is sampled. Next, the latent embedding $z$ is passed through a multi-layer perceptron (MLP) with residual connections, mapping it to code probability distribution through the codebook. During infernce, the masked motion tockens are replaced with an empty tokens.

Then, we approximate the posterior probability model $p(z\mid e)$ with a parametric inference model $q_{\phi}(z\mid e)$. With $\phi$ indicating the parameters of this inference model, also called the variational parameters. We optimize the variational parameters $\phi$ such that:

[TABLE]

The distribution $q_{\phi}(z\mid e)$ is parameterized using deep neural networks:

[TABLE]

we assume that $z$ follows a multivariate Gaussian distribution where each dimension is independent. We implement the mean layer and standard deviation layer with self attention layer to predicts a mean vector $\mu$ and a log magnitude variance vector $log\sigma^{2}$ of the multivariate Gaussian distribution based on $e$. By leveraging the reparameterization technique, a $z$ is sampled as:

[TABLE]

Note that latent space sampler is differentiable with the reparameterization technique.

III-D Training Diverse-T2M

III-D1 Optimization Goal of Motion Discrete Representation

Overall, the residual VQ-VAE is trained via a motion reconstruction loss combined with a latent embedding loss at each quantization layer:

[TABLE]

where $\operatorname{sg}[\cdot]$ denotes the stop-gradient operation, and $\beta$ a weighting factor for embedding constraint. The codebooks are updated via exponential moving average and codebook reset following [14, 15].

III-D2 Optimization Goal of Motion Generation under Noise and Text Conditions

We jointly train the noise-conditional model and the text-conditional model. Specifically we mix the training samples of noise signal and text signal following a certain ratio to train the model. The loss function for supervising model learning are as follows:

[TABLE]

$\mathcal{L}_{\text{tran }}$ is used to supervise the mask transformer to generate the correct probability distribution, and $\mathcal{L}_{\text{kl}}$ is used to supervise the latent space to learn a distribution closer to a Gaussian distribution.

For loss function $\mathcal{L}_{\text{tran}}$, each signal-code sequence pair is a training sample. Our training goal is to predict the masked tockens. We directly maximize the log-likelihood of the data distribution $p(\hat{M}\mid s)$:

[TABLE]

the likelihood of the full sequence is denoted as follows:

[TABLE]

where $[mask]_{t}$ indicates whether the $i$-th token is masked, which is set to 0 if masked, and 1 otherwise. The residual layer is predicted without using mask following [14].

For supervising the latent space to learn a distribution closer to a Gaussian distribution, $\mathcal{L}_{\text{kl}}$ is difined as:

[TABLE]

$D$ represents the feature dimension and $KL$ represents the Kullback-Leibler divergence.

IV Experiments

In this section, we evaluate the text-to-motion generation performance of our proposed Diverse-T2M and carry out detailed ablation studies to explore the contribution of each component to the performance of Diverse-T2M. Meanwhile, we compare our method with existing state-of-the-art methods on three two benchmark datasets.

IV-A Experimental Setting

Evaluation Datasets. In this study, we evaluate our proposed method on two widely recognized text-to-motion generation datasets. KIT Motion-Language (KIT-ML) [54] and HumanML3D [39]. We follow the evaluation protocol proposed in HumanML3D [39, 12].

Evaluation Metrics. We follow the common metrics of prior works [39] to evaluate the text-to-motion generation performance. Global representations of motion and text descriptions are first extracted with the pre-trained network in [39], and then measured by the following five metrics:

• •

R-Precision. Given one motion sequence and 32 text descriptions (1 ground-truth and 31 randomly selected mismatched descriptions), we rank the Euclidean distances between the motion and text embeddings. Top-1, Top-2, and Top-3 accuracy of motion-to-text retrieval are reported.

• •

Frechet Inception Distance (FID). We calculate the distribution distance between the generated and real motion using FID [55] on the extracted motion features.

• •

Multimodal Distance (MM-Dist). The average Euclidean distances between each text feature and the generated motion feature from this text.

• •

Multimodality (MModality). For one text description, we generate 30 motion sequences forming 10 pairs of motion. We extract motion features and compute the average Euclidean distances of the pairs. We finally report the average over all the text descriptions.

Experimental Settings. We implemented our Diverse-T2M in PyTorch [56] by performing all experiments on 1×3090 GPUs. In our experiments, the downsampling rate of encoder and the residual lay number are set to 4 and 6 respectively. The size of codebook is 512 × 512, i.e., 512 512-dimension dictionary vectors. Following [47, 14, 15], the dataset HumanML3D and KIT-ML are extracted into motion features with dimensions 263 and 251 respectively, which related to local joints position, velocity, and rotations in root space as well as global translation and rotations. The features are calculated from SMPL [45] joints. And the joint number for these two datasets is set to 22 and 21. During inference, the mixing ratio $w$ for the generation of the noise signal and text signal results is set to 3. During training, the KL divergence is multiplied by 1e-5 when supervised.

IV-B Benchmark Evaluation

This section will compare our method with state-of-the-art text-to-motion generation methods on HumanML3D [39] and KIT Motion-Language (KIT-ML) [54]. The results are presented in Table I and II, respectively. They show that our method achieves the state-of-the-art performance, state-of-the-art, which substantially increase the diversity of generation while ensuring that the generated motion sequences have a high semantic consistency with the conditional text, validating the efficacy of our proposed method.

HumanML3D. Table I presents the comparison results of our method with the state-of-the-art methods in R-Precision, Frechet Inception Distance (FID), Multimodal Distance (MM-Dist) and Multimodality (MModality) metrics. Our Diverse-T2m outperforms the state-of-the-art methods in most key metrics, i.e., ranks first for metrics R-Precision (Top-1, Top-2, Top-3), MM-Dist, MModality and ranks second for FID.

MModality reflects the ability of generating diverse motions based on a text specified by a user, and we can find that our diverse-T2M substantially outperforms all previous methods in this metric. The previous best performing method for generating diversity was MDM (MultiModality 2.799), which we outperform by 43 percent. MDM generates a relatively high diversity of motions, but the text has less control over the motions, which is reflected in the metric MultiModal Dist, which is almost twice as worse than ours. This phenomenon is also present in R Precision. Compared to previous methods that generate higher diversity, we have a stronger ability to diversify the generation, and in addition to this, our method has stronger textual control as well as more realism for the generated motions.

For previous diffusion-based methods, such as ReMoDiffuse, MotionLCM , Motion Mamba, we outperform these models with higher motion-text matching accuracy, i.e., we gain significant improvement (+1.5 Top-1, + 1.9 Top-2, +1.4 Top-3) on the best ReMoDiffuse of the three. Not only do we outperform them all in the semantic consistency metrics, i.e., R Precision and MultiModal Dist, but we are better at generating diversity, such as outperform ReMoDiffuse 121% and Motion Mamba by 73% in MultiModality. In addition, these models often exhibit slower inference speed due to the iterative nature of diffusion processes.

In contrast to MoMask, which possess best ability of text control during generation, our diverse-T2M not only do we outperform them all in the semantic consistency metrics they specialize in, but we are better at generating diversity. In details, we gain the improvement (+0.4 Top-1, + 0.4 Top-2, +0.2 Top-3) on MoMask and outperform it by 220% in MultiModality.

KIT-ML. We further evaluate our method on the commonly used large-scale dataset KIT-ML. As the results in Table II shows, our Diverse-T2m outperforms the state-of-the-art methods in key metrics, i.e., ranks first for metrics R-Precision (Top-1), MModality and ranks second for R-Precision (Top-2, Top-3) and MultiModal Dist.

The results on the KIT-ML dataset indicate that our diverse-T2M is at a high level in the performance of semantic consistency, which proves that we can generate semantically well-fitting motions with a textual description. We have the highest generation diversity, much higher than other methods, while maintaining the ability to maintain excellent text control, such as outperform ReMoDiffuse 179%, 206% and Motion Mamba by 106% in MultiModality.

IV-C Ablation Study

We conduct a series of ablative experiments to evaluate the performance contribution of each component of our proposed Diverse-T2M.

Baseline setting: 1) Motion discrete representation: Since for the discrete representation of the motion is used the technique community already has the technique VQ-VAE or RVQ-VAE, we perform ablation experiments on both baselines. 2) classifier-free guidancer strategy: following [14, 20], they set $\emptyset$ to some text, we make this as our baseline for verify the effectiveness of our proposed motion generation under noise and text conditions.

Effectiveness of Variational Text-to-codes Predictor. As shown in Table III, our designed Variational Text-to-codes Predictor achieves performance improvements over both VQ-VAE and RVQ-VAE baseline. In details, it brings a boost in generation diversity (+0.774 MultiModality for VQ-VAE setting and + 0.67 MultiModality for RVQ-VAE setting). In the Variational Text-to-codes Predictor, we construct a hidden space within transformer structure, changing the original one-to-one mapping to map a data point to a distribution in the hidden space. Furthermore, we design a sampler that maps to a probability distribution of the motion code after sampling the distribution corresponding to the text in the hidden space. These designs can effectively overcome the problem of poor diversity of mapping-based generation methods despite their high control accuracy. In previous transformer-based methods, a text can only be mapped to a point in the feature space, whereas we can map a text to a data distribution in the hidden space. These are more in line with the objective laws of text-motion correspondence in the real world: due to individual differences, a sentence describing an motion should have a large solution space containing a large number of solutions that are different but consistent with the textual description. In addition to diversity, there are improvements in other areas, such as in the metrics R-precision and MultiModal Dist reflecting semantic consistency (+0.4 Top1, +0.2 Top2, +0.1 Top3 for VQ-VAE setting and +0.6 Top1, +0.2 Top2, +0.3 Top3 for RVQ-VAE setting), and in the metrics reflecting the quality of the reconstruction FID over the baseline. This reflects the superiority of the modules we designed.

Effectiveness of Noise Signal. Table III shows that the noise signal can significantly improve the generation diversity while maintaining a high level of text control accuracy(+2.143 MultiModality for VQ-VAE setting and + 2.085 MultiModality for RVQ-VAE setting). Diversity must be carried by diverse information, and noise can assume such a random source of information, for example, in diffusion-based methods, the input noise is sampled randomly, which keeps the generated images diverse. As an example, when we design diverse-T2M to generate thirty motions for the same text that match its description, although the text signals are all the same, the noise signals are different for each inference, and the noise carries a diversity of information carriers. As an example, when we design diverse-T2M to generate thirty motions for the same text that match its description, although the text signals are all the same, the noise signals are different for each inference, and the noise carries the diversity of information carriers, which enables the thirty inference to generate motions that are diverse and match the text description.

In addition, we conducted experiments with fusion ratio settings for distributions generated from text and noise signals. In Figure 5, The larger $w$ is, the more the noise signal affects the final generated motion. Digram (b) indicates that the generation diversity is continuously getting larger as $w$ increases. Analyzing digram (a,c,d) reveals that as $w$ increases, it causes both retrieval accuracy and multimodal distance performance to deteriorate, due to the decreasing control of the text. The experiments with the $w$ parameter reflect that the role of both the text signal and the noise signal on the accuracy of motion control and the diversity of motions, and that a balance between the two can be satisfied by generating both motions that conform to the text description and a variety of motions.

IV-D Analysis of Diversity of Motion Generation

Qualitative Comparison. As shown in Figure 6, the text we use for qualitative comparison is a long text, which contains both a complex action flow and rich control details, such as “use the right hand”, etc., which is a great test of the control of motion generation. We tested our method against two other sota methods used for comparison, using a given text as input and letting each method generate thirty motions and visualize six randomly selected motions. Many control lapses can be seen in the T2mM-GPT, such as using the left hand where the right hand should be used, as shown in the black dashed box in the figure. The MoMask method, on the other hand, although the motions are all exact, each of them is almost identical and has a very low diversity. In contrast, our approach can both conform to textual descriptions where the text controls and play arbitrarily within reasonable limits where the text does not.

Analysis. The reason why our method produces diverse results for multiple inference of the same text is because the noise signal is different for each inference, which carries the information needed for diversity, and secondly, the text and noise signals are mapped to the data distribution in latent space for each inference, and the sampling is randomized each time. Together, these affect the generative diversity of our Diverse-T2M.

IV-E Visualization Results

We provide some result in Figure 7. It can be seen that our approach can clearly generate 3D spatial movement of positions, such as going up and down the steps in example (a); as well as movements with interaction with the outside world, such as sitting down in (b); and precision control, with only a half-turn in (c). These examples demonstrate that we maintain a high level of textual control.

IV-F Applications

Here we present the application in a business scenario, as shown in Figure 8. After completing character modeling for the application scenario, animators often need to manually animate the character frame-by-frame, which is resource-intensive. With our Diverse-T2M, animators only need to input the required text, as illustrated at the bottom of Figure 8, and multiple animations consistent with the text can be automatically generated. These animations can then be selected or further refined to meet specific business needs, thus greatly reducing the cost of animation production.

Additionally, for non-human animation requirements, the motions generated by models trained on human text-to-motion datasets can still be effectively utilized. As illustrated in the chick character in Figure 8, by rigging the chick character with a human skeletal structure, anthropomorphic motions can be driven using human skeletal sequences.

When the text-to-motion model exhibits greater diversity, it can enhance the visual richness and dynamism of application scenes, preventing homogenization. Furthermore, it offers a broader array of foundational options for animators to perform secondary processing. The demo videos are provided in the supplemental materials.

V Conclusion

We propose Diverse-T2M for text-to-motion generation, which introduces uncertainty into the generation process to strength the ability of human motion generation diversity. Our results on text-to-motion generation benchmark datasets demonstrate that our method significantly enhances diversity while maintaining state-of-the-art performance in text consistency. By proposing the use of noise signals as carriers of diversity and constructing a latent space that encodes text into a probabilistic representation, we effectively introduce uncertainty into the generation process. Additionally, we demonstrate the usefulness of our approach in real-world business, as well as the potential for adapting non-human character animation.