LipDiffuser: Lip-to-Speech Generation with Conditional Diffusion Models

TL;DR

LipDiffuser introduces a novel diffusion-based model for converting silent videos of lips into natural, speaker-identifiable speech, outperforming existing methods in quality and similarity.

Contribution

The paper proposes LipDiffuser, a new conditional diffusion model with innovative visual conditioning and a neural vocoder for improved lip-to-speech synthesis.

Findings

Outperforms existing lip-to-speech models in perceptual quality

Achieves higher speaker similarity in synthesized speech

Maintains competitive automatic speech recognition performance

Abstract

We present LipDiffuser, a conditional diffusion model for lip-to-speech generation synthesizing natural and intelligible speech directly from silent video recordings. Our approach leverages the magnitude-preserving ablated diffusion model (MP-ADM) architecture as a denoiser model. To effectively condition the model, we incorporate visual features using magnitude-preserving feature-wise linear modulation (MP-FiLM) alongside speaker embeddings. A neural vocoder then reconstructs the speech waveform from the generated mel-spectrograms. Evaluations on LRS3 demonstrate that LipDiffuser outperforms existing lip-to-speech baselines in perceptual speech quality and speaker similarity, while remaining competitive in downstream automatic speech recognition. These findings are also supported by a formal listening experiment.