MoSa: Motion Generation with Scalable Autoregressive Modeling

TL;DR

MoSa introduces a hierarchical, scalable autoregressive framework for text-driven 3D human motion generation, achieving state-of-the-art quality and efficiency with fewer inference steps and strong generalization capabilities.

Contribution

The paper proposes MoSa, a novel hierarchical motion generation method that employs a multi-scale token preservation strategy and scalable autoregressive modeling, significantly improving speed and quality over prior approaches.

Findings

MoSa achieves an FID of 0.06 on Motion-X, outperforming previous methods.

Inference time is reduced by 27% compared to prior models.

MoSa effectively generalizes to motion editing tasks without additional fine-tuning.

Abstract

We introduce MoSa, a novel hierarchical motion generation framework for text-driven 3D human motion generation that enhances the Vector Quantization-guided Generative Transformers (VQ-GT) paradigm through a coarse-to-fine scalable generation process. In MoSa, we propose a Multi-scale Token Preservation Strategy (MTPS) integrated into a hierarchical residual vector quantization variational autoencoder (RQ-VAE). MTPS employs interpolation at each hierarchical quantization to effectively retain coarse-to-fine multi-scale tokens. With this, the generative transformer supports Scalable Autoregressive (SAR) modeling, which predicts scale tokens, unlike traditional methods that predict only one token at each step. Consequently, MoSa requires only 10 inference steps, matching the number of RQ-VAE quantization layers. To address potential reconstruction degradation from frequent interpolation, we propose CAQ-VAE, a lightweight yet expressive convolution-attention hybrid VQ-VAE. CAQ-VAE enhances residual block design and incorporates attention mechanisms to better capture global dependencies. Extensive experiments show that MoSa achieves state-of-the-art generation quality and efficiency, outperforming prior methods in both fidelity and speed. On the Motion-X dataset, MoSa achieves an FID of 0.06 (versus MoMask's 0.20) while reducing inference time by 27 percent. Moreover, MoSa generalizes well to downstream tasks such as motion editing, requiring no additional fine-tuning. The code is available at https://mosa-web.github.io/MoSa-web