Enhancing Motion Variation in Text-to-Motion Models via Pose and Video Conditioned Editing

TL;DR

This paper introduces a novel method for text-to-motion generation that uses video and image conditions to enhance motion diversity, enabling the creation of unseen human motions with realistic quality.

Contribution

The proposed approach leverages video and image conditions as priors and posteriors to generate diverse and unseen human motions beyond training data limitations.

Findings

Enables generation of motions not in training data

Produces realistic unseen motions like kicking and squatting

User study shows comparable realism to existing datasets

Abstract

Text-to-motion models that generate sequences of human poses from textual descriptions are garnering significant attention. However, due to data scarcity, the range of motions these models can produce is still limited. For instance, current text-to-motion models cannot generate a motion of kicking a football with the instep of the foot, since the training data only includes martial arts kicks. We propose a novel method that uses short video clips or images as conditions to modify existing basic motions. In this approach, the model's understanding of a kick serves as the prior, while the video or image of a football kick acts as the posterior, enabling the generation of the desired motion. By incorporating these additional modalities as conditions, our method can create motions not present in the training set, overcoming the limitations of text-motion datasets. A user study with 26 participants demonstrated that our approach produces unseen motions with realism comparable to commonly represented motions in text-motion datasets (e.g., HumanML3D), such as walking, running, squatting, and kicking.