Learning Phonetic Context-Dependent Viseme for Enhancing Speech-Driven 3D Facial Animation

TL;DR

This paper introduces a phonetic context-aware loss function for speech-driven 3D facial animation, improving the naturalness and smoothness of facial movements by modeling viseme coarticulation effects.

Contribution

It proposes a novel loss that explicitly incorporates phonetic context and coarticulation weights to enhance animation quality beyond traditional frame-wise methods.

Findings

Improved quantitative metrics over baseline methods.

Enhanced visual quality and naturalness of facial animations.

Demonstrated effectiveness of phonetic context modeling in synthesis.

Abstract

Speech-driven 3D facial animation aims to generate realistic facial movements synchronized with audio. Traditional methods primarily minimize reconstruction loss by aligning each frame with ground-truth. However, this frame-wise approach often fails to capture the continuity of facial motion, leading to jittery and unnatural outputs due to coarticulation. To address this, we propose a novel phonetic context-aware loss, which explicitly models the influence of phonetic context on viseme transitions. By incorporating a viseme coarticulation weight, we assign adaptive importance to facial movements based on their dynamic changes over time, ensuring smoother and perceptually consistent animations. Extensive experiments demonstrate that replacing the conventional reconstruction loss with ours improves both quantitative metrics and visual quality. It highlights the importance of explicitly modeling phonetic context-dependent visemes in synthesizing natural speech-driven 3D facial animation. Project page: https://cau-irislab.github.io/interspeech25/