3DGesPolicy: Phoneme-Aware Holistic Co-Speech Gesture Generation Based on Action Control

TL;DR

3DGesPolicy introduces a novel action-based diffusion framework for generating coherent, expressive, and speech-aligned holistic co-speech gestures by modeling motion as unified trajectories and integrating multi-modal signals.

Contribution

It reformulates gesture generation as a continuous trajectory control problem and proposes a multi-modal fusion module for better semantic and expressive alignment.

Findings

Outperforms state-of-the-art methods in naturalness and expressiveness

Ensures spatial and semantic coherence in gesture generation

Effectively models inter-frame motion patterns

Abstract

Generating holistic co-speech gestures that integrate full-body motion with facial expressions suffers from semantically incoherent coordination on body motion and spatially unstable meaningless movements due to existing part-decomposed or frame-level regression methods, We introduce 3DGesPolicy, a novel action-based framework that reformulates holistic gesture generation as a continuous trajectory control problem through diffusion policy from robotics. By modeling frame-to-frame variations as unified holistic actions, our method effectively learns inter-frame holistic gesture motion patterns and ensures both spatially and semantically coherent movement trajectories that adhere to realistic motion manifolds. To further bridge the gap in expressive alignment, we propose a Gesture-Audio-Phoneme (GAP) fusion module that can deeply integrate and refine multi-modal signals, ensuring structured and fine-grained alignment between speech semantics, body motion, and facial expressions. Extensive quantitative and qualitative experiments on the BEAT2 dataset demonstrate the effectiveness of our 3DGesPolicy across other state-of-the-art methods in generating natural, expressive, and highly speech-aligned holistic gestures.