ExpGest: Expressive Speaker Generation Using Diffusion Model and Hybrid Audio-Text Guidance

TL;DR

ExpGest is a novel diffusion-based framework that generates expressive, natural, and controllable full-body gestures by integrating synchronized text and audio information, addressing previous limitations of mechanical gestures.

Contribution

It introduces a hybrid audio-text guidance method and a semantic-gesture alignment in latent space, enabling mixed generation modes and improved expressiveness over prior models.

Findings

Outperforms state-of-the-art models in expressiveness and naturalness

Effectively learns from combined text-driven and audio-driven datasets

Achieves controllable global motion in gesture generation

Abstract

Existing gesture generation methods primarily focus on upper body gestures based on audio features, neglecting speech content, emotion, and locomotion. These limitations result in stiff, mechanical gestures that fail to convey the true meaning of audio content. We introduce ExpGest, a novel framework leveraging synchronized text and audio information to generate expressive full-body gestures. Unlike AdaIN or one-hot encoding methods, we design a noise emotion classifier for optimizing adversarial direction noise, avoiding melody distortion and guiding results towards specified emotions. Moreover, aligning semantic and gestures in the latent space provides better generalization capabilities. ExpGest, a diffusion model-based gesture generation framework, is the first attempt to offer mixed generation modes, including audio-driven gestures and text-shaped motion. Experiments show that our framework effectively learns from combined text-driven motion and audio-induced gesture datasets, and preliminary results demonstrate that ExpGest achieves more expressive, natural, and controllable global motion in speakers compared to state-of-the-art models.