JEAN: Joint Expression and Audio-guided NeRF-based Talking Face Generation

TL;DR

JEAN is a novel NeRF-based method for generating realistic talking face videos that accurately synchronize lip movements with audio while preserving facial expressions and identity, trained on monocular videos without ground truth.

Contribution

It introduces a joint framework that disentangles audio and expression features using self-supervised and contrastive learning, enabling high-fidelity, synchronized talking face synthesis.

Findings

Achieves state-of-the-art lip synchronization accuracy

Demonstrates high-quality facial expression transfer

Operates effectively without ground truth data

Abstract

We introduce a novel method for joint expression and audio-guided talking face generation. Recent approaches either struggle to preserve the speaker identity or fail to produce faithful facial expressions. To address these challenges, we propose a NeRF-based network. Since we train our network on monocular videos without any ground truth, it is essential to learn disentangled representations for audio and expression. We first learn audio features in a self-supervised manner, given utterances from multiple subjects. By incorporating a contrastive learning technique, we ensure that the learned audio features are aligned to the lip motion and disentangled from the muscle motion of the rest of the face. We then devise a transformer-based architecture that learns expression features, capturing long-range facial expressions and disentangling them from the speech-specific mouth movements. Through quantitative and qualitative evaluation, we demonstrate that our method can synthesize high-fidelity talking face videos, achieving state-of-the-art facial expression transfer along with lip synchronization to unseen audio.