Meta-Learning Empowered Meta-Face: Personalized Speaking Style Adaptation for Audio-Driven 3D Talking Face Animation

TL;DR

MetaFace is a meta-learning based approach that enables personalized, adaptable 3D talking face animation across diverse speaking styles, outperforming existing methods in efficiency and accuracy.

Contribution

The paper introduces MetaFace, a novel meta-learning framework with innovative components for effective and efficient speaking style adaptation in 3D face animation.

Findings

MetaFace outperforms existing baselines in speaking style adaptation.

MetaFace establishes a new state-of-the-art in personalized 3D face animation.

The approach improves efficiency in model optimization and style learning.

Abstract

Audio-driven 3D face animation is increasingly vital in live streaming and augmented reality applications. While remarkable progress has been observed, most existing approaches are designed for specific individuals with predefined speaking styles, thus neglecting the adaptability to varied speaking styles. To address this limitation, this paper introduces MetaFace, a novel methodology meticulously crafted for speaking style adaptation. Grounded in the novel concept of meta-learning, MetaFace is composed of several key components: the Robust Meta Initialization Stage (RMIS) for fundamental speaking style adaptation, the Dynamic Relation Mining Neural Process (DRMN) for forging connections between observed and unobserved speaking styles, and the Low-rank Matrix Memory Reduction Approach to enhance the efficiency of model optimization as well as learning style details. Leveraging these novel designs, MetaFace not only significantly outperforms robust existing baselines but also establishes a new state-of-the-art, as substantiated by our experimental results.