Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert

TL;DR

This paper introduces a novel approach for talking face generation that incorporates a lip-reading expert to improve the intelligibility of lip movements, achieving state-of-the-art results in lip-reading accuracy and synchronization.

Contribution

It proposes using a lip-reading expert with contrastive learning and a transformer to enhance lip movement intelligibility and synchronization in speech-driven face generation.

Findings

Over 38% WER on LRS2 dataset

27.8% accuracy on LRW dataset

State-of-the-art lip-speech synchronization

Abstract

Talking face generation, also known as speech-to-lip generation, reconstructs facial motions concerning lips given coherent speech input. The previous studies revealed the importance of lip-speech synchronization and visual quality. Despite much progress, they hardly focus on the content of lip movements i.e., the visual intelligibility of the spoken words, which is an important aspect of generation quality. To address the problem, we propose using a lip-reading expert to improve the intelligibility of the generated lip regions by penalizing the incorrect generation results. Moreover, to compensate for data scarcity, we train the lip-reading expert in an audio-visual self-supervised manner. With a lip-reading expert, we propose a novel contrastive learning to enhance lip-speech synchronization, and a transformer to encode audio synchronically with video, while considering global temporal dependency of audio. For evaluation, we propose a new strategy with two different lip-reading experts to measure intelligibility of the generated videos. Rigorous experiments show that our proposal is superior to other State-of-the-art (SOTA) methods, such as Wav2Lip, in reading intelligibility i.e., over 38% Word Error Rate (WER) on LRS2 dataset and 27.8% accuracy on LRW dataset. We also achieve the SOTA performance in lip-speech synchronization and comparable performances in visual quality.