Learning Disentangled Speech- and Expression-Driven Blendshapes for 3D Talking Face Animation

TL;DR

This paper introduces a method to generate emotionally expressive 3D talking faces by disentangling speech-driven and emotion-driven blendshapes, enabling realistic animation with limited emotional data.

Contribution

It proposes a novel linear additive model with a sparsity constraint to disentangle speech and emotion blendshapes for 3D facial animation.

Findings

Achieves natural expression and lip-sync in 3D talking face animation.

Outperforms existing methods in emotional expressivity.

Maintains accurate lip synchronization while adding expressions.

Abstract

Expressions are fundamental to conveying human emotions. With the rapid advancement of AI-generated content (AIGC), realistic and expressive 3D facial animation has become increasingly crucial. Despite recent progress in speech-driven lip-sync for talking-face animation, generating emotionally expressive talking faces remains underexplored. A major obstacle is the scarcity of real emotional 3D talking-face datasets due to the high cost of data capture. To address this, we model facial animation driven by both speech and emotion as a linear additive problem. Leveraging a 3D talking-face dataset with neutral expressions (VOCAset) and a dataset of 3D expression sequences (Florence4D), we jointly learn a set of blendshapes driven by speech and emotion. We introduce a sparsity constraint loss to encourage disentanglement between the two types of blendshapes while allowing the model to capture inherent secondary cross-domain deformations present in the training data. The learned blendshapes can be further mapped to the expression and jaw pose parameters of the FLAME model, enabling the animation of 3D Gaussian avatars. Qualitative and quantitative experiments demonstrate that our method naturally generates talking faces with specified expressions while maintaining accurate lip synchronization. Perceptual studies further show that our approach achieves superior emotional expressivity compared to existing methods, without compromising lip-sync quality.