UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation

TL;DR

UniMotion is a pioneering unified framework that simultaneously understands and generates human motion, language, and images, overcoming previous modality restrictions and quantization errors through continuous representations and novel alignment strategies.

Contribution

It introduces a novel unified architecture with continuous modality treatment, a Cross-Modal Aligned Motion VAE, and self-supervised pre-training methods for comprehensive motion-text-vision tasks.

Findings

Achieves state-of-the-art across seven diverse tasks.

Excels in cross-modal understanding, generation, and editing.

Demonstrates strong performance on compositional tasks.

Abstract

We present UniMotion, to our knowledge the first unified framework for simultaneous understanding and generation of human motion, natural language, and RGB images within a single architecture. Existing unified models handle only restricted modality subsets (e.g., Motion-Text or static Pose-Image) and predominantly rely on discrete tokenization, which introduces quantization errors and disrupts temporal continuity. UniMotion overcomes both limitations through a core principle: treating motion as a first-class continuous modality on equal footing with RGB. A novel Cross-Modal Aligned Motion VAE (CMA-VAE) and symmetric dual-path embedders construct parallel continuous pathways for Motion and RGB within a shared LLM backbone. To inject visual-semantic priors into motion representations without requiring images at inference, we propose Dual-Posterior KL Alignment (DPA), which distills a vision-fused encoder's richer posterior into the motion-only encoder. To address the cold-start problem -- where text supervision alone is too sparse to calibrate the newly introduced motion pathway -- we further propose Latent Reconstruction Alignment (LRA), a self-supervised pre-training strategy that uses dense motion latents as unambiguous conditions to co-calibrate the embedder, backbone, and flow head, establishing a stable motion-aware foundation for all downstream tasks. UniMotion achieves state-of-the-art performance across seven tasks spanning any-to-any understanding, generation, and editing among the three modalities, with especially strong advantages on cross-modal compositional tasks.