ReAlign: Bilingual Text-to-Motion Generation via Step-Aware Reward-Guided Alignment

TL;DR

ReAlign introduces a novel reward-guided diffusion approach for bilingual text-to-motion generation, addressing dataset scarcity and alignment issues to produce semantically consistent and high-quality 3D motions from bilingual texts.

Contribution

The paper presents a new bilingual human motion dataset, a unified bilingual diffusion model, and a reward-guided sampling method to improve alignment and motion quality.

Findings

Enhanced text-motion alignment accuracy

Improved motion realism and diversity

Outperforms existing state-of-the-art methods

Abstract

Bilingual text-to-motion generation, which synthesizes 3D human motions from bilingual text inputs, holds immense potential for cross-linguistic applications in gaming, film, and robotics. However, this task faces critical challenges: the absence of bilingual motion-language datasets and the misalignment between text and motion distributions in diffusion models, leading to semantically inconsistent or low-quality motions. To address these challenges, we propose BiHumanML3D, a novel bilingual human motion dataset, which establishes a crucial benchmark for bilingual text-to-motion generation models. Furthermore, we propose a Bilingual Motion Diffusion model (BiMD), which leverages cross-lingual aligned representations to capture semantics, thereby achieving a unified bilingual model. Building upon this, we propose Reward-guided sampling Alignment (ReAlign) method, comprising a step-aware reward model to assess alignment quality during sampling and a reward-guided strategy that directs the diffusion process toward an optimally aligned distribution. This reward model integrates step-aware tokens and combines a text-aligned module for semantic consistency and a motion-aligned module for realism, refining noisy motions at each timestep to balance probability density and alignment. Experiments demonstrate that our approach significantly improves text-motion alignment and motion quality compared to existing state-of-the-art methods. Project page: https://wengwanjiang.github.io/ReAlign-page/.