UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons

TL;DR

UnifiedGesture introduces a diffusion model-based approach for speech-driven gesture synthesis that unifies multiple skeleton standards, improving realism and diversity in generated gestures across datasets.

Contribution

The paper proposes a novel diffusion model with a retargeting network and reinforcement learning to unify gesture representations and enhance speech-gesture alignment across diverse datasets.

Findings

Outperforms recent methods in CCA, FGD, and human-likeness metrics.

Effectively unifies multiple gesture datasets with different skeleton standards.

Generates more realistic and diverse gestures aligned with speech.

Abstract

The automatic co-speech gesture generation draws much attention in computer animation. Previous works designed network structures on individual datasets, which resulted in a lack of data volume and generalizability across different motion capture standards. In addition, it is a challenging task due to the weak correlation between speech and gestures. To address these problems, we present UnifiedGesture, a novel diffusion model-based speech-driven gesture synthesis approach, trained on multiple gesture datasets with different skeletons. Specifically, we first present a retargeting network to learn latent homeomorphic graphs for different motion capture standards, unifying the representations of various gestures while extending the dataset. We then capture the correlation between speech and gestures based on a diffusion model architecture using cross-local attention and self-attention to generate better speech-matched and realistic gestures. To further align speech and gesture and increase diversity, we incorporate reinforcement learning on the discrete gesture units with a learned reward function. Extensive experiments show that UnifiedGesture outperforms recent approaches on speech-driven gesture generation in terms of CCA, FGD, and human-likeness. All code, pre-trained models, databases, and demos are available to the public at https://github.com/YoungSeng/UnifiedGesture.