VarGes: Improving Variation in Co-Speech 3D Gesture Generation via StyleCLIPS

TL;DR

VarGes introduces a novel framework that enhances co-speech 3D gesture generation by integrating visual style cues and audio features, resulting in more diverse and natural gestures in virtual human interactions.

Contribution

The paper presents VarGes, a new variation-driven approach combining style-reference video data and advanced encoding to improve gesture diversity and naturalness in co-speech gesture synthesis.

Findings

Outperforms existing methods in gesture diversity

Achieves higher naturalness in generated gestures

Validated on benchmark datasets

Abstract

Generating expressive and diverse human gestures from audio is crucial in fields like human-computer interaction, virtual reality, and animation. Though existing methods have achieved remarkable performance, they often exhibit limitations due to constrained dataset diversity and the restricted amount of information derived from audio inputs. To address these challenges, we present VarGes, a novel variation-driven framework designed to enhance co-speech gesture generation by integrating visual stylistic cues while maintaining naturalness. Our approach begins with the Variation-Enhanced Feature Extraction (VEFE) module, which seamlessly incorporates \textcolor{blue}{style-reference} video data into a 3D human pose estimation network to extract StyleCLIPS, thereby enriching the input with stylistic information. Subsequently, we employ the Variation-Compensation Style Encoder (VCSE), a transformer-style encoder equipped with an additive attention mechanism pooling layer, to robustly encode diverse StyleCLIPS representations and effectively manage stylistic variations. Finally, the Variation-Driven Gesture Predictor (VDGP) module fuses MFCC audio features with StyleCLIPS encodings via cross-attention, injecting this fused data into a cross-conditional autoregressive model to modulate 3D human gesture generation based on audio input and stylistic clues. The efficacy of our approach is validated on benchmark datasets, where it outperforms existing methods in terms of gesture diversity and naturalness. The code and video results will be made publicly available upon acceptance:https://github.com/mookerr/VarGES/ .