CoCoGesture: Toward Coherent Co-speech 3D Gesture Generation in the Wild

TL;DR

CoCoGesture introduces a large-scale dataset and a diffusion-based framework with a novel training paradigm for generating diverse, coherent 3D gestures from unseen speech inputs, significantly improving zero-shot speech-to-gesture synthesis.

Contribution

The paper presents a new large-scale co-speech 3D gesture dataset and a diffusion model with a fine-tuning approach using audio ControlNet and Mixture-of-Gesture-Experts for improved gesture generation.

Findings

Outperforms state-of-the-art zero-shot speech-to-gesture methods

Constructed GES-X dataset with over 40 million gesture instances

Demonstrates vivid and diverse gesture synthesis from unseen speech prompts

Abstract

Deriving co-speech 3D gestures has seen tremendous progress in virtual avatar animation. Yet, the existing methods often produce stiff and unreasonable gestures with unseen human speech inputs due to the limited 3D speech-gesture data. In this paper, we propose CoCoGesture, a novel framework enabling vivid and diverse gesture synthesis from unseen human speech prompts. Our key insight is built upon the custom-designed pretrain-fintune training paradigm. At the pretraining stage, we aim to formulate a large generalizable gesture diffusion model by learning the abundant postures manifold. Therefore, to alleviate the scarcity of 3D data, we first construct a large-scale co-speech 3D gesture dataset containing more than 40M meshed posture instances across 4.3K speakers, dubbed GES-X. Then, we scale up the large unconditional diffusion model to 1B parameters and pre-train it to be our gesture experts. At the finetune stage, we present the audio ControlNet that incorporates the human voice as condition prompts to guide the gesture generation. Here, we construct the audio ControlNet through a trainable copy of our pre-trained diffusion model. Moreover, we design a novel Mixture-of-Gesture-Experts (MoGE) block to adaptively fuse the audio embedding from the human speech and the gesture features from the pre-trained gesture experts with a routing mechanism. Such an effective manner ensures audio embedding is temporal coordinated with motion features while preserving the vivid and diverse gesture generation. Extensive experiments demonstrate that our proposed CoCoGesture outperforms the state-of-the-art methods on the zero-shot speech-to-gesture generation. The dataset will be publicly available at: https://mattie-e.github.io/GES-X/