UMo: Unified Sparse Motion Modeling for Real-Time Co-Speech Avatars

TL;DR

UMo introduces a unified sparse motion modeling framework that enables real-time, high-fidelity co-speech avatar animations by processing text, audio, and motion within a single architecture.

Contribution

The paper proposes UMo, a novel architecture combining sparse modeling and a Mixture-of-Experts framework for efficient, high-quality, real-time co-speech avatar animation.

Findings

UMo achieves superior animation quality under low latency.

It maintains fine-grained speech-motion alignment in real-time.

UMo demonstrates effective facial and gesture animation generation.

Abstract

Speech-driven gestures and facial animations are fundamental to expressive digital avatars in games, virtual production, and interactive media. However, existing methods are either limited to a single modality for audio motion alignment, failing to fully utilize the potential of massive human motion data, or are constrained by the representation ability and throughput of multimodal models, which makes it difficult to achieve high-quality motion generation or real-time performance. We present UMo, a unified sparse motion modeling architecture for real-time co-speech avatars, which processes text, audio, and motion tokens within a unified formulation. Leveraging a spatially sparse Mixture-of-Experts framework and a temporally sparse, keyframe-centric design, UMo efficiently performs real-time dense reconstruction, enabling temporally coherent and high-fidelity animation generation for both facial expressions and gestures. Furthermore, we implement a multi-stage training strategy with targeted audio augmentation to enhance acoustic diversity and semantic consistency. Consequently, UMo preserves fine-grained speech-motion alignment even under strict latency constraints. Extensive quantitative and qualitative evaluations show that UMo achieves better output quality under low latency and real-time performance constraints, offering a practical solution for high-fidelity real-time co-speech avatars.