Text-guided 3D Human Motion Generation with Keyframe-based Parallel Skip Transformer

TL;DR

This paper introduces KeyMotion, a novel approach for text-guided 3D human motion generation that uses keyframes, a VAE, and a Parallel Skip Transformer to produce realistic sequences efficiently, outperforming existing methods.

Contribution

The paper presents a new framework combining keyframe generation, VAE-based latent space projection, and a Parallel Skip Transformer for improved text-guided human motion synthesis.

Findings

Achieves state-of-the-art results on HumanML3D dataset.

Outperforms others on R-precision and MultiModal Distance metrics.

Provides competitive performance on KIT dataset with top metrics.

Abstract

Text-driven human motion generation is an emerging task in animation and humanoid robot design. Existing algorithms directly generate the full sequence which is computationally expensive and prone to errors as it does not pay special attention to key poses, a process that has been the cornerstone of animation for decades. We propose KeyMotion, that generates plausible human motion sequences corresponding to input text by first generating keyframes followed by in-filling. We use a Variational Autoencoder (VAE) with Kullback-Leibler regularization to project the keyframes into a latent space to reduce dimensionality and further accelerate the subsequent diffusion process. For the reverse diffusion, we propose a novel Parallel Skip Transformer that performs cross-modal attention between the keyframe latents and text condition. To complete the motion sequence, we propose a text-guided Transformer designed to perform motion-in-filling, ensuring the preservation of both fidelity and adherence to the physical constraints of human motion. Experiments show that our method achieves state-of-theart results on the HumanML3D dataset outperforming others on all R-precision metrics and MultiModal Distance. KeyMotion also achieves competitive performance on the KIT dataset, achieving the best results on Top3 R-precision, FID, and Diversity metrics.