MotionGPT3: Human Motion as a Second Modality

TL;DR

MotionGPT3 introduces a bimodal motion-language model that encodes motion in a continuous latent space and uses a dual-stream Transformer to improve understanding and generation, achieving faster convergence and state-of-the-art results.

Contribution

It proposes a novel bimodal model with continuous motion encoding and a dual-stream Transformer to reduce interference and accelerate training.

Findings

2x faster convergence in training loss

up to 4x faster convergence in validation

state-of-the-art performance on motion benchmarks

Abstract

With the rapid progress of large language models (LLMs), multimodal frameworks that unify understanding and generation have become promising, yet they face increasing complexity as the number of modalities and tasks grows. We observe that motion quantization introduces approximation errors that cap motion quality, and that unifying discrete text and continuous motion within a single-stream backbone amplifies cross-modal interference. Motivated by recent multi-branch Transformer designs that separate signals from different modalities, we propose MotionGPT3, a bimodal motion-language model for both understanding and generation. MotionGPT3 encodes raw motion into a continuous latent space using a variational autoencoder (VAE), thereby avoiding quantization-induced artifacts, while leveraging the semantic prior of pretrained language models. A dual-stream Transformer with shared attention preserves modality-specific routes while enabling controlled, bidirectional information flow, which reduces interference, stabilizing optimization, and empirically accelerates convergence without degrading fidelity. For multimodal joint training, a generate-then-align three-stage schedule further improves stability and limits cross-task interference. Experiments show that MotionGPT3 achieves 2x faster convergence in training loss and up to 4x faster convergence in validation, while maintaining state-of-the-art performance on standard motion understanding and motion generation benchmarks.