Enabling Synergistic Full-Body Control in Prompt-Based Co-Speech Motion Generation

TL;DR

This paper introduces SynTalker, a novel method for full-body co-speech motion generation that enables precise control of body movements based on speech and prompts, overcoming dataset limitations.

Contribution

The paper presents a multi-stage training and diffusion-based inference approach to achieve synergistic full-body motion control from text prompts and speech.

Findings

Supports precise full-body motion control

Handles diverse human activities beyond training data

Outperforms existing methods in flexibility and accuracy

Abstract

Current co-speech motion generation approaches usually focus on upper body gestures following speech contents only, while lacking supporting the elaborate control of synergistic full-body motion based on text prompts, such as talking while walking. The major challenges lie in 1) the existing speech-to-motion datasets only involve highly limited full-body motions, making a wide range of common human activities out of training distribution; 2) these datasets also lack annotated user prompts. To address these challenges, we propose SynTalker, which utilizes the off-the-shelf text-to-motion dataset as an auxiliary for supplementing the missing full-body motion and prompts. The core technical contributions are two-fold. One is the multi-stage training process which obtains an aligned embedding space of motion, speech, and prompts despite the significant distributional mismatch in motion between speech-to-motion and text-to-motion datasets. Another is the diffusion-based conditional inference process, which utilizes the separate-then-combine strategy to realize fine-grained control of local body parts. Extensive experiments are conducted to verify that our approach supports precise and flexible control of synergistic full-body motion generation based on both speeches and user prompts, which is beyond the ability of existing approaches.