Motion-2-To-3: Leveraging 2D Motion Data for 3D Motion Generations

TL;DR

This paper presents a novel framework that leverages 2D human motion data extracted from videos to enhance text-driven 3D human motion generation, reducing reliance on costly 3D motion capture data.

Contribution

It introduces a disentanglement approach for local joint motion and global movement, enabling effective learning from 2D data and fine-tuning with 3D data for realistic motion synthesis.

Findings

Efficiently utilizes 2D data for 3D motion generation.

Supports a wider range of realistic human motions.

Demonstrates effectiveness on benchmark datasets and novel prompts.

Abstract

Text-driven human motion synthesis has showcased its potential for revolutionizing motion design in the movie and game industry. Existing methods often rely on 3D motion capture data, which requires special setups, resulting in high costs for data acquisition, ultimately limiting the diversity and scope of human motion. In contrast, 2D human videos offer a vast and accessible source of motion data, covering a wider range of styles and activities. In this paper, we explore the use of 2D human motion extracted from videos as an alternative data source to improve text-driven 3D motion generation. Our approach introduces a novel framework that disentangles local joint motion from global movements, enabling efficient learning of local motion priors from 2D data. We first train a single-view 2D local motion generator on a large dataset of text-2D motion pairs. Then we fine-tune the generator with 3D data, transforming it into a multi-view generator that predicts view-consistent local joint motion and root dynamics. Evaluations on the well-acknowledged dataset and novel text prompts demonstrate that our method can efficiently utilize 2D data, supporting a wider range of realistic 3D human motion generation. Our code is publicly available at https://zju3dv.github.io/Motion-2-to-3/.