LipSody: Lip-to-Speech Synthesis with Enhanced Prosody Consistency

TL;DR

LipSody is a novel lip-to-speech synthesis framework that significantly improves prosody consistency by integrating speaker identity, linguistic content, and emotional cues from facial videos, enhancing speech naturalness and speaker similarity.

Contribution

It introduces a prosody-guiding strategy that combines multiple facial cues to enhance prosody in lip-to-speech synthesis, addressing limitations of previous diffusion-based models.

Findings

Improved prosody metrics such as pitch deviation and energy consistency.

Enhanced speaker similarity in synthesized speech.

Substantial performance gains over prior models in prosody-related evaluations.

Abstract

Lip-to-speech synthesis aims to generate speech audio directly from silent facial video by reconstructing linguistic content from lip movements, providing valuable applications in situations where audio signals are unavailable or degraded. While recent diffusion-based models such as LipVoicer have demonstrated impressive performance in reconstructing linguistic content, they often lack prosodic consistency. In this work, we propose LipSody, a lip-to-speech framework enhanced for prosody consistency. LipSody introduces a prosody-guiding strategy that leverages three complementary cues: speaker identity extracted from facial images, linguistic content derived from lip movements, and emotional context inferred from face video. Experimental results demonstrate that LipSody substantially improves prosody-related metrics, including global and local pitch deviations, energy consistency, and speaker similarity, compared to prior approaches.