MoLingo: Motion-Language Alignment for Text-to-Motion Generation

TL;DR

MoLingo introduces a novel text-to-motion model that uses semantic alignment and cross-attention to generate realistic human motions closely matching textual descriptions, advancing the state of the art.

Contribution

The paper proposes a semantic-aligned motion encoder and cross-attention conditioning to improve diffusion-based text-to-motion generation.

Findings

Semantic alignment improves diffusion effectiveness

Cross-attention enhances motion realism and text alignment

Achieves state-of-the-art results on standard metrics

Abstract

We introduce MoLingo, a text-to-motion (T2M) model that generates realistic, lifelike human motion by denoising in a continuous latent space. Recent works perform latent space diffusion, either on the whole latent at once or auto-regressively over multiple latents. In this paper, we study how to make diffusion on continuous motion latents work best. We focus on two questions: (1) how to build a semantically aligned latent space so diffusion becomes more effective, and (2) how to best inject text conditioning so the motion follows the description closely. We propose a semantic-aligned motion encoder trained with frame-level text labels so that latents with similar text meaning stay close, which makes the latent space more diffusion-friendly. We also compare single-token conditioning with a multi-token cross-attention scheme and find that cross-attention gives better motion realism and text-motion alignment. With semantically aligned latents, auto-regressive generation, and cross-attention text conditioning, our model sets a new state of the art in human motion generation on standard metrics and in a user study. We will release our code and models for further research and downstream usage.