RESOUND: Speech Reconstruction from Silent Videos via Acoustic-Semantic Decomposed Modeling

TL;DR

RESOUND is a novel lip-to-speech system that reconstructs natural, expressive speech from silent videos by decomposing linguistic and prosodic features and integrating speech units for improved synthesis.

Contribution

It introduces a source-filter inspired model with separate semantic and acoustic paths, and incorporates speech units to enhance speech reconstruction from silent videos.

Findings

Effective across multiple benchmarks

Improves speech naturalness and intelligibility

Preserves speaker identity

Abstract

Lip-to-speech (L2S) synthesis, which reconstructs speech from visual cues, faces challenges in accuracy and naturalness due to limited supervision in capturing linguistic content, accents, and prosody. In this paper, we propose RESOUND, a novel L2S system that generates intelligible and expressive speech from silent talking face videos. Leveraging source-filter theory, our method involves two components: an acoustic path to predict prosody and a semantic path to extract linguistic features. This separation simplifies learning, allowing independent optimization of each representation. Additionally, we enhance performance by integrating speech units, a proven unsupervised speech representation technique, into waveform generation alongside mel-spectrograms. This allows RESOUND to synthesize prosodic speech while preserving content and speaker identity. Experiments conducted on two standard L2S benchmarks confirm the effectiveness of the proposed method across various metrics.