The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion
Changan Chen, Juze Zhang, Shrinidhi K. Lakshmikanth, Yusu Fang, Ruizhi, Shao, Gordon Wetzstein, Li Fei-Fei, Ehsan Adeli

TL;DR
This paper introduces a multimodal language model that unifies verbal and non-verbal human motion, enabling flexible, data-efficient generation and understanding of gestures, speech, and emotions for more natural virtual communication.
Contribution
It presents a novel framework that combines verbal and non-verbal cues using multimodal language models, with a new pre-training strategy for improved performance and versatility.
Findings
Achieves state-of-the-art co-speech gesture generation
Requires less training data than previous models
Enables editable gesture generation and emotion prediction
Abstract
Human communication is inherently multimodal, involving a combination of verbal and non-verbal cues such as speech, facial expressions, and body gestures. Modeling these behaviors is essential for understanding human interaction and for creating virtual characters that can communicate naturally in applications like games, films, and virtual reality. However, existing motion generation models are typically limited to specific input modalities -- either speech, text, or motion data -- and cannot fully leverage the diversity of available data. In this paper, we propose a novel framework that unifies verbal and non-verbal language using multimodal language models for human motion understanding and generation. This model is flexible in taking text, speech, and motion or any combination of them as input. Coupled with our novel pre-training strategy, our model not only achieves…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHuman Motion and Animation
