Audio-visual video-to-speech synthesis with synthesized input audio

TL;DR

This paper explores using both video and synthesized audio inputs during training and inference for video-to-speech synthesis, improving speech reconstruction from silent videos.

Contribution

It introduces a method that employs pre-trained models to generate input audio, enhancing video-to-speech synthesis with both visual and synthesized audio data.

Findings

Effective speech reconstruction with raw waveforms

Successful use of mel spectrograms as targets

Improved synthesis performance with combined inputs

Abstract

Video-to-speech synthesis involves reconstructing the speech signal of a speaker from a silent video. The implicit assumption of this task is that the sound signal is either missing or contains a high amount of noise/corruption such that it is not useful for processing. Previous works in the literature either use video inputs only or employ both video and audio inputs during training, and discard the input audio pathway during inference. In this work we investigate the effect of using video and audio inputs for video-to-speech synthesis during both training and inference. In particular, we use pre-trained video-to-speech models to synthesize the missing speech signals and then train an audio-visual-to-speech synthesis model, using both the silent video and the synthesized speech as inputs, to predict the final reconstructed speech. Our experiments demonstrate that this approach is successful with both raw waveforms and mel spectrograms as target outputs.