Democratizing High-Fidelity Co-Speech Gesture Video Generation

TL;DR

This paper introduces a lightweight diffusion-based framework for generating realistic co-speech gesture videos, leveraging 2D skeletons and a new large-scale dataset to improve synchronization and visual quality.

Contribution

It presents a novel skeleton-conditioned diffusion model and the first public dataset for high-fidelity co-speech gesture video synthesis, enhancing accessibility and performance.

Findings

Outperforms state-of-the-art in visual quality and synchronization

Generalizes well across speakers and contexts

Provides a new dataset with extensive annotations

Abstract

Co-speech gesture video generation aims to synthesize realistic, audio-aligned videos of speakers, complete with synchronized facial expressions and body gestures. This task presents challenges due to the significant one-to-many mapping between audio and visual content, further complicated by the scarcity of large-scale public datasets and high computational demands. We propose a lightweight framework that utilizes 2D full-body skeletons as an efficient auxiliary condition to bridge audio signals with visual outputs. Our approach introduces a diffusion model conditioned on fine-grained audio segments and a skeleton extracted from the speaker's reference image, predicting skeletal motions through skeleton-audio feature fusion to ensure strict audio coordination and body shape consistency. The generated skeletons are then fed into an off-the-shelf human video generation model with the speaker's reference image to synthesize high-fidelity videos. To democratize research, we present CSG-405-the first public dataset with 405 hours of high-resolution videos across 71 speech types, annotated with 2D skeletons and diverse speaker demographics. Experiments show that our method exceeds state-of-the-art approaches in visual quality and synchronization while generalizing across speakers and contexts. Code, models, and CSG-405 are publicly released at https://mpi-lab.github.io/Democratizing-CSG/