Speech-Driven 3D Face Animation with Composite and Regional Facial Movements

TL;DR

This paper proposes a novel speech-driven 3D face animation method that incorporates both composite and regional facial movements, using adaptive modulation and a non-autoregressive model to produce vivid, high-quality animations efficiently.

Contribution

It introduces an adaptive modulation module for global facial movement adjustment and a regional focus mechanism, along with a non-autoregressive backbone for improved speech-to-3D face animation.

Findings

Outperforms state-of-the-art methods in qualitative assessments

Achieves higher accuracy in reproducing facial nuances

Enables efficient inference with high-frequency detail preservation

Abstract

Speech-driven 3D face animation poses significant challenges due to the intricacy and variability inherent in human facial movements. This paper emphasizes the importance of considering both the composite and regional natures of facial movements in speech-driven 3D face animation. The composite nature pertains to how speech-independent factors globally modulate speech-driven facial movements along the temporal dimension. Meanwhile, the regional nature alludes to the notion that facial movements are not globally correlated but are actuated by local musculature along the spatial dimension. It is thus indispensable to incorporate both natures for engendering vivid animation. To address the composite nature, we introduce an adaptive modulation module that employs arbitrary facial movements to dynamically adjust speech-driven facial movements across frames on a global scale. To accommodate the regional nature, our approach ensures that each constituent of the facial features for every frame focuses on the local spatial movements of 3D faces. Moreover, we present a non-autoregressive backbone for translating audio to 3D facial movements, which maintains high-frequency nuances of facial movements and facilitates efficient inference. Comprehensive experiments and user studies demonstrate that our method surpasses contemporary state-of-the-art approaches both qualitatively and quantitatively.