AdaMesh: Personalized Facial Expressions and Head Poses for Adaptive Speech-Driven 3D Facial Animation

TL;DR

AdaMesh is a novel adaptive speech-driven 3D facial animation method that personalizes facial expressions and head poses using minimal reference video data, outperforming existing approaches in vividness and style preservation.

Contribution

The paper introduces AdaMesh, which learns personalized facial and head pose styles efficiently from short reference videos without extensive fine-tuning, using MoLoRA and pose retrieval techniques.

Findings

Outperforms state-of-the-art methods in style preservation and vividness.

Effectively captures personalized facial expressions with minimal reference data.

Generates realistic and vivid facial animations aligned with speech.

Abstract

Speech-driven 3D facial animation aims at generating facial movements that are synchronized with the driving speech, which has been widely explored recently. Existing works mostly neglect the person-specific talking style in generation, including facial expression and head pose styles. Several works intend to capture the personalities by fine-tuning modules. However, limited training data leads to the lack of vividness. In this work, we propose AdaMesh, a novel adaptive speech-driven facial animation approach, which learns the personalized talking style from a reference video of about 10 seconds and generates vivid facial expressions and head poses. Specifically, we propose mixture-of-low-rank adaptation (MoLoRA) to fine-tune the expression adapter, which efficiently captures the facial expression style. For the personalized pose style, we propose a pose adapter by building a discrete pose prior and retrieving the appropriate style embedding with a semantic-aware pose style matrix without fine-tuning. Extensive experimental results show that our approach outperforms state-of-the-art methods, preserves the talking style in the reference video, and generates vivid facial animation. The supplementary video and code will be available at https://adamesh.github.io.