QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation

TL;DR

QPGesture introduces a quantization-based, phase-guided framework for speech-driven gesture generation, effectively addressing jitter and asynchronous speech-gesture relationships to produce more natural gestures.

Contribution

It proposes a novel gesture VQ-VAE codebook and phase-guided matching approach, improving gesture-speech alignment and naturalness over prior methods.

Findings

Outperforms recent approaches in gesture generation quality

Effectively alleviates gesture jittering issues

Enhances speech-gesture synchronization

Abstract

Speech-driven gesture generation is highly challenging due to the random jitters of human motion. In addition, there is an inherent asynchronous relationship between human speech and gestures. To tackle these challenges, we introduce a novel quantization-based and phase-guided motion-matching framework. Specifically, we first present a gesture VQ-VAE module to learn a codebook to summarize meaningful gesture units. With each code representing a unique gesture, random jittering problems are alleviated effectively. We then use Levenshtein distance to align diverse gestures with different speech. Levenshtein distance based on audio quantization as a similarity metric of corresponding speech of gestures helps match more appropriate gestures with speech, and solves the alignment problem of speech and gestures well. Moreover, we introduce phase to guide the optimal gesture matching based on the semantics of context or rhythm of audio. Phase guides when text-based or speech-based gestures should be performed to make the generated gestures more natural. Extensive experiments show that our method outperforms recent approaches on speech-driven gesture generation. Our code, database, pre-trained models, and demos are available at https://github.com/YoungSeng/QPGesture.