Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animation

TL;DR

This paper introduces a joint audio-text model that leverages high-level contextual text embeddings to improve expressive speech-driven 3D facial animation, capturing diverse facial motions and expressions more realistically.

Contribution

The novel integration of pre-trained language model embeddings with audio features enhances the synthesis of expressive facial animations beyond phoneme-level approaches.

Findings

Outperforms existing state-of-the-art methods in realism and synchronization.

Effectively captures diverse upper face expressions.

Demonstrates superior results in quantitative, qualitative, and perceptual evaluations.

Abstract

Speech-driven 3D facial animation with accurate lip synchronization has been widely studied. However, synthesizing realistic motions for the entire face during speech has rarely been explored. In this work, we present a joint audio-text model to capture the contextual information for expressive speech-driven 3D facial animation. The existing datasets are collected to cover as many different phonemes as possible instead of sentences, thus limiting the capability of the audio-based model to learn more diverse contexts. To address this, we propose to leverage the contextual text embeddings extracted from the powerful pre-trained language model that has learned rich contextual representations from large-scale text data. Our hypothesis is that the text features can disambiguate the variations in upper face expressions, which are not strongly correlated with the audio. In contrast to prior approaches which learn phoneme-level features from the text, we investigate the high-level contextual text features for speech-driven 3D facial animation. We show that the combined acoustic and textual modalities can synthesize realistic facial expressions while maintaining audio-lip synchronization. We conduct the quantitative and qualitative evaluations as well as the perceptual user study. The results demonstrate the superior performance of our model against existing state-of-the-art approaches.