MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controls

TL;DR

MotionCraft introduces a unified diffusion transformer for multimodal whole-body motion generation, effectively handling diverse control modalities and motion formats through a coarse-to-fine training strategy and novel graph modeling.

Contribution

The paper presents MotionCraft, a novel framework with plug-and-play multimodal control, a two-stage training process, and a new benchmark, MC-Bench, for improved multimodal motion generation.

Findings

Achieves state-of-the-art results on multiple motion generation tasks

Effectively models static and dynamic human topology graphs

Addresses motion format inconsistency with the new MC-Bench benchmark

Abstract

Whole-body multimodal motion generation, controlled by text, speech, or music, has numerous applications including video generation and character animation. However, employing a unified model to achieve various generation tasks with different condition modalities presents two main challenges: motion distribution drifts across different tasks (e.g., co-speech gestures and text-driven daily actions) and the complex optimization of mixed conditions with varying granularities (e.g., text and audio). Additionally, inconsistent motion formats across different tasks and datasets hinder effective training toward multimodal motion generation. In this paper, we propose MotionCraft, a unified diffusion transformer that crafts whole-body motion with plug-and-play multimodal control. Our framework employs a coarse-to-fine training strategy, starting with the first stage of text-to-motion semantic pre-training, followed by the second stage of multimodal low-level control adaptation to handle conditions of varying granularities. To effectively learn and transfer motion knowledge across different distributions, we design MC-Attn for parallel modeling of static and dynamic human topology graphs. To overcome the motion format inconsistency of existing benchmarks, we introduce MC-Bench, the first available multimodal whole-body motion generation benchmark based on the unified SMPL-X format. Extensive experiments show that MotionCraft achieves state-of-the-art performance on various standard motion generation tasks.