C2G2: Controllable Co-speech Gesture Generation with Latent Diffusion Model

TL;DR

C2G2 introduces a controllable, high-fidelity co-speech gesture generation framework using latent diffusion models, enabling stable, temporally consistent, and editable gestures with speaker-specific control for digital avatars.

Contribution

The paper presents a novel two-stage temporal dependency enhancement strategy and a repainting control mechanism within a latent diffusion framework for improved gesture generation.

Findings

Outperforms state-of-the-art methods on benchmark datasets.

Achieves stable and temporally consistent gesture synthesis.

Enables flexible editing and speaker-specific gesture control.

Abstract

Co-speech gesture generation is crucial for automatic digital avatar animation. However, existing methods suffer from issues such as unstable training and temporal inconsistency, particularly in generating high-fidelity and comprehensive gestures. Additionally, these methods lack effective control over speaker identity and temporal editing of the generated gestures. Focusing on capturing temporal latent information and applying practical controlling, we propose a Controllable Co-speech Gesture Generation framework, named C2G2. Specifically, we propose a two-stage temporal dependency enhancement strategy motivated by latent diffusion models. We further introduce two key features to C2G2, namely a speaker-specific decoder to generate speaker-related real-length skeletons and a repainting strategy for flexible gesture generation/editing. Extensive experiments on benchmark gesture datasets verify the effectiveness of our proposed C2G2 compared with several state-of-the-art baselines. The link of the project demo page can be found at https://c2g2-gesture.github.io/c2_gesture