DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embedding

TL;DR

This paper introduces DiffV2S, a diffusion-based video-to-speech synthesis model that uses vision-guided speaker embeddings extracted without audio during inference, achieving state-of-the-art results.

Contribution

The paper proposes a novel vision-guided speaker embedding extractor using self-supervised learning and prompt tuning, enabling speech synthesis solely from visual input.

Findings

Achieves state-of-the-art performance in video-to-speech synthesis

Maintains phoneme details and speaker identity in generated speech

Does not require audio information during inference

Abstract

Recent research has demonstrated impressive results in video-to-speech synthesis which involves reconstructing speech solely from visual input. However, previous works have struggled to accurately synthesize speech due to a lack of sufficient guidance for the model to infer the correct content with the appropriate sound. To resolve the issue, they have adopted an extra speaker embedding as a speaking style guidance from a reference auditory information. Nevertheless, it is not always possible to obtain the audio information from the corresponding video input, especially during the inference time. In this paper, we present a novel vision-guided speaker embedding extractor using a self-supervised pre-trained model and prompt tuning technique. In doing so, the rich speaker embedding information can be produced solely from input visual information, and the extra audio information is not necessary during the inference time. Using the extracted vision-guided speaker embedding representations, we further develop a diffusion-based video-to-speech synthesis model, so called DiffV2S, conditioned on those speaker embeddings and the visual representation extracted from the input video. The proposed DiffV2S not only maintains phoneme details contained in the input video frames, but also creates a highly intelligible mel-spectrogram in which the speaker identities of the multiple speakers are all preserved. Our experimental results show that DiffV2S achieves the state-of-the-art performance compared to the previous video-to-speech synthesis technique.