MotionDuet: Dual-Conditioned 3D Human Motion Generation with Video-Regularized Text Learning

TL;DR

MotionDuet is a multimodal framework that generates realistic 3D human motions by aligning video-derived representations with textual prompts, bridging the gap between different modalities.

Contribution

It introduces dual-conditioning with video and text, along with novel encoding, transformation, and harmonization techniques for improved motion generation.

Findings

Outperforms state-of-the-art baselines in realism and controllability.

Effectively integrates video and text cues for diverse motion synthesis.

Demonstrates robustness across various motion generation tasks.

Abstract

3D Human motion generation is pivotal across film, animation, gaming, and embodied intelligence. Traditional 3D motion synthesis relies on costly motion capture, while recent work shows that 2D videos provide rich, temporally coherent observations of human behavior. Existing approaches, however, either map high-level text descriptions to motion or rely solely on video conditioning, leaving a gap between generated dynamics and real-world motion statistics. We introduce MotionDuet, a multimodal framework that aligns motion generation with the distribution of video-derived representations. In this dual-conditioning paradigm, video cues extracted from a pretrained model (e.g., VideoMAE) ground low-level motion dynamics, while textual prompts provide semantic intent. To bridge the distribution gap across modalities, we propose Dual-stream Unified Encoding and Transformation (DUET) and a Distribution-Aware Structural Harmonization (DASH) loss. DUET fuses video-informed cues into the motion latent space via unified encoding and dynamic attention, while DASH aligns motion trajectories with both distributional and structural statistics of video features. An auto-guidance mechanism further balances textual and visual signals by leveraging a weakened copy of the model, enhancing controllability without sacrificing diversity. Extensive experiments demonstrate that MotionDuet generates realistic and controllable human motions, surpassing strong state-of-the-art baselines.