Towards Robust and Controllable Text-to-Motion via Masked Autoregressive Diffusion

TL;DR

This paper introduces MoMADiff, a novel framework combining masked modeling and diffusion for robust, controllable 3D human motion generation from text, capable of generalizing to unseen motions with fine-grained control.

Contribution

It proposes a new motion generation method that integrates masked autoregressive diffusion with user-specified keyframes for enhanced control and generalization.

Findings

Outperforms state-of-the-art in motion quality and control

Demonstrates strong generalization to novel motions

Supports flexible keyframe-based motion synthesis

Abstract

Generating 3D human motion from text descriptions remains challenging due to the diverse and complex nature of human motion. While existing methods excel within the training distribution, they often struggle with out-of-distribution motions, limiting their applicability in real-world scenarios. Existing VQVAE-based methods often fail to represent novel motions faithfully using discrete tokens, which hampers their ability to generalize beyond seen data. Meanwhile, diffusion-based methods operating on continuous representations often lack fine-grained control over individual frames. To address these challenges, we propose a robust motion generation framework MoMADiff, which combines masked modeling with diffusion processes to generate motion using frame-level continuous representations. Our model supports flexible user-provided keyframe specification, enabling precise control over both spatial and temporal aspects of motion synthesis. MoMADiff demonstrates strong generalization capability on novel text-to-motion datasets with sparse keyframes as motion prompts. Extensive experiments on two held-out datasets and two standard benchmarks show that our method consistently outperforms state-of-the-art models in motion quality, instruction fidelity, and keyframe adherence. The code is available at: https://github.com/zzysteve/MoMADiff