LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading

TL;DR

LipVoicer is a novel lip-to-speech method that leverages lip reading and text guidance via a diffusion model to generate highly intelligible, natural, and synchronized speech from silent videos, outperforming existing approaches.

Contribution

The paper introduces LipVoicer, integrating lip reading and text modality with a diffusion model for improved speech synthesis from silent videos, especially on challenging datasets.

Findings

Outperforms baselines on LRS2 and LRS3 datasets.

Significantly improves speech intelligibility and naturalness.

Reduces Word Error Rate (WER) in generated speech.

Abstract

Lip-to-speech involves generating a natural-sounding speech synchronized with a soundless video of a person talking. Despite recent advances, current methods still cannot produce high-quality speech with high levels of intelligibility for challenging and realistic datasets such as LRS3. In this work, we present LipVoicer, a novel method that generates high-quality speech, even for in-the-wild and rich datasets, by incorporating the text modality. Given a silent video, we first predict the spoken text using a pre-trained lip-reading network. We then condition a diffusion model on the video and use the extracted text through a classifier-guidance mechanism where a pre-trained ASR serves as the classifier. LipVoicer outperforms multiple lip-to-speech baselines on LRS2 and LRS3, which are in-the-wild datasets with hundreds of unique speakers in their test set and an unrestricted vocabulary. Moreover, our experiments show that the inclusion of the text modality plays a major role in the intelligibility of the produced speech, readily perceptible while listening, and is empirically reflected in the substantial reduction of the WER metric. We demonstrate the effectiveness of LipVoicer through human evaluation, which shows that it produces more natural and synchronized speech signals compared to competing methods. Finally, we created a demo showcasing LipVoicer's superiority in producing natural, synchronized, and intelligible speech, providing additional evidence of its effectiveness. Project page and code: https://github.com/yochaiye/LipVoicer