DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion

TL;DR

DF-3DFace introduces a diffusion-based approach for speech-driven 3D facial animation that captures one-to-many relationships, enabling realistic and varied facial expressions, identities, and poses without needing reference meshes.

Contribution

The paper presents a novel diffusion-driven method for speech-to-3D face synthesis that models complex variability in facial attributes and introduces a large-scale dataset for training.

Findings

Successfully generates diverse facial shapes and motions from speech.

Achieves more realistic facial animations than existing methods.

Effectively models identity, pose, and facial motion jointly.

Abstract

Speech-driven 3D facial animation has gained significant attention for its ability to create realistic and expressive facial animations in 3D space based on speech. Learning-based methods have shown promising progress in achieving accurate facial motion synchronized with speech. However, one-to-many nature of speech-to-3D facial synthesis has not been fully explored: while the lip accurately synchronizes with the speech content, other facial attributes beyond speech-related motions are variable with respect to the speech. To account for the potential variance in the facial attributes within a single speech, we propose DF-3DFace, a diffusion-driven speech-to-3D face mesh synthesis. DF-3DFace captures the complex one-to-many relationships between speech and 3D face based on diffusion. It concurrently achieves aligned lip motion by exploiting audio-mesh synchronization and masked conditioning. Furthermore, the proposed method jointly models identity and pose in addition to facial motions so that it can generate 3D face animation without requiring a reference identity mesh and produce natural head poses. We contribute a new large-scale 3D facial mesh dataset, 3D-HDTF to enable the synthesis of variations in identities, poses, and facial motions of 3D face mesh. Extensive experiments demonstrate that our method successfully generates highly variable facial shapes and motions from speech and simultaneously achieves more realistic facial animation than the state-of-the-art methods.