Audio-driven Talking Face Generation with Stabilized Synchronization Loss

TL;DR

This paper proposes a novel approach for talking face generation that improves lip synchronization and visual quality by addressing training instability and lip leaking issues with new loss functions and a silent-lip generator.

Contribution

Introduces stabilized synchronization loss and AVSyncNet, along with a silent-lip generator, to enhance lip sync accuracy and visual quality in talking face videos.

Findings

Outperforms state-of-the-art methods in visual quality

Achieves better lip synchronization accuracy

Validates contributions through ablation studies

Abstract

Talking face generation aims to create realistic videos with accurate lip synchronization and high visual quality, using given audio and reference video while preserving identity and visual characteristics. In this paper, we start by identifying several issues with existing synchronization learning methods. These involve unstable training, lip synchronization, and visual quality issues caused by lip-sync loss, SyncNet, and lip leaking from the identity reference. To address these issues, we first tackle the lip leaking problem by introducing a silent-lip generator, which changes the lips of the identity reference to alleviate leakage. We then introduce stabilized synchronization loss and AVSyncNet to overcome problems caused by lip-sync loss and SyncNet. Experiments show that our model outperforms state-of-the-art methods in both visual quality and lip synchronization. Comprehensive ablation studies further validate our individual contributions and their cohesive effects.