End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks

TL;DR

This paper introduces an end-to-end GAN-based model for video-to-speech synthesis that directly generates realistic speech waveforms from video input without intermediate steps, outperforming previous methods on multiple datasets.

Contribution

The work presents the first end-to-end GAN model for video-to-speech synthesis capable of producing intelligible speech directly from raw video, including in-the-wild scenarios.

Findings

Outperforms previous methods on GRID and LRW datasets

Produces highly realistic and intelligible speech from video

Effective for both constrained and wild datasets

Abstract

Video-to-speech is the process of reconstructing the audio speech from a video of a spoken utterance. Previous approaches to this task have relied on a two-step process where an intermediate representation is inferred from the video, and is then decoded into waveform audio using a vocoder or a waveform reconstruction algorithm. In this work, we propose a new end-to-end video-to-speech model based on Generative Adversarial Networks (GANs) which translates spoken video to waveform end-to-end without using any intermediate representation or separate waveform synthesis algorithm. Our model consists of an encoder-decoder architecture that receives raw video as input and generates speech, which is then fed to a waveform critic and a power critic. The use of an adversarial loss based on these two critics enables the direct synthesis of raw audio waveform and ensures its realism. In addition, the use of our three comparative losses helps establish direct correspondence between the generated audio and the input video. We show that this model is able to reconstruct speech with remarkable realism for constrained datasets such as GRID, and that it is the first end-to-end model to produce intelligible speech for LRW (Lip Reading in the Wild), featuring hundreds of speakers recorded entirely `in the wild'. We evaluate the generated samples in two different scenarios -- seen and unseen speakers -- using four objective metrics which measure the quality and intelligibility of artificial speech. We demonstrate that the proposed approach outperforms all previous works in most metrics on GRID and LRW.