LSF-Animation: Label-Free Speech-Driven Facial Animation via Implicit Feature Representation

TL;DR

LSF-Animation is a new speech-driven facial animation framework that implicitly captures emotion and identity features from speech, improving generalization to unseen speakers and emotions without manual labels, and enhances animation realism.

Contribution

It introduces an implicit feature extraction method and a Hierarchical Interaction Fusion Block for better emotion and identity integration in facial animation.

Findings

Outperforms state-of-the-art in expressiveness and realism.

Improves generalization to unseen speakers and emotions.

Eliminates need for manual emotion and identity labels.

Abstract

Speech-driven 3D facial animation has attracted increasing interest since its potential to generate expressive and temporally synchronized digital humans. While recent works have begun to explore emotion-aware animation, they still depend on explicit one-hot encodings to represent identity and emotion with given emotion and identity labels, which limits their ability to generalize to unseen speakers. Moreover, the emotional cues inherently present in speech are often neglected, limiting the naturalness and adaptability of generated animations. In this work, we propose LSF-Animation, a novel framework that eliminates the reliance on explicit emotion and identity feature representations. Specifically, LSF-Animation implicitly extracts emotion information from speech and captures the identity features from a neutral facial mesh, enabling improved generalization to unseen speakers and emotional states without requiring manual labels. Furthermore, we introduce a Hierarchical Interaction Fusion Block (HIFB), which employs a fusion token to integrate dual transformer features and more effectively integrate emotional, motion-related and identity-related cues. Extensive experiments conducted on the 3DMEAD dataset demonstrate that our method surpasses recent state-of-the-art approaches in terms of emotional expressiveness, identity generalization, and animation realism. The source code will be released at: https://github.com/Dogter521/LSF-Animation.