VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selection

TL;DR

This paper presents VisageSynTalk, a novel approach for silent talking face video-to-speech synthesis that effectively disentangles speech content and speaker identity, enabling high-quality speech generation even for unseen speakers.

Contribution

It introduces speech-visage feature selection and disentanglement techniques to improve unseen speaker video-to-speech synthesis performance.

Findings

Achieves high speech intelligibility for unseen speakers.

Effectively separates speech content from speaker identity.

Validated on multiple datasets showing improved synthesis quality.

Abstract

The goal of this work is to reconstruct speech from a silent talking face video. Recent studies have shown impressive performance on synthesizing speech from silent talking face videos. However, they have not explicitly considered on varying identity characteristics of different speakers, which place a challenge in the video-to-speech synthesis, and this becomes more critical in unseen-speaker settings. Our approach is to separate the speech content and the visage-style from a given silent talking face video. By guiding the model to independently focus on modeling the two representations, we can obtain the speech of high intelligibility from the model even when the input video of an unseen subject is given. To this end, we introduce speech-visage selection that separates the speech content and the speaker identity from the visual features of the input video. The disentangled representations are jointly incorporated to synthesize speech through visage-style based synthesizer which generates speech by coating the visage-styles while maintaining the speech content. Thus, the proposed framework brings the advantage of synthesizing the speech containing the right content even with the silent talking face video of an unseen subject. We validate the effectiveness of the proposed framework on the GRID, TCD-TIMIT volunteer, and LRW datasets.