Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation

TL;DR

This paper introduces an audio-visual speech representation expert to improve lip synchronization in talking face video generation and proposes new metrics for evaluating synchronization quality.

Contribution

It presents a novel use of AV-HuBERT for both training and evaluating talking face generation, enhancing synchronization accuracy and assessment.

Findings

Improved lip synchronization accuracy in generated videos

Proposed three new evaluation metrics for lip sync quality

Demonstrated effectiveness through experiments and ablation studies

Abstract

In the task of talking face generation, the objective is to generate a face video with lips synchronized to the corresponding audio while preserving visual details and identity information. Current methods face the challenge of learning accurate lip synchronization while avoiding detrimental effects on visual quality, as well as robustly evaluating such synchronization. To tackle these problems, we propose utilizing an audio-visual speech representation expert (AV-HuBERT) for calculating lip synchronization loss during training. Moreover, leveraging AV-HuBERT's features, we introduce three novel lip synchronization evaluation metrics, aiming to provide a comprehensive assessment of lip synchronization performance. Experimental results, along with a detailed ablation study, demonstrate the effectiveness of our approach and the utility of the proposed evaluation metrics.