Vclip: Face-based Speaker Generation by Face-voice Association Learning

TL;DR

Vclip introduces a face-voice association learning method using CLIP to improve face-based speech synthesis, achieving high cross-modal verification accuracy and producing voices that perceptually match reference faces.

Contribution

The paper presents a novel approach leveraging CLIP for face-voice association and a retrieval-based speaker generation method for improved face-based speech synthesis.

Findings

Achieved 89.63% cross-modal verification AUC on Voxceleb.

Bridged face and voice features for better synthesis quality.

Enhanced TTS with face-voice consistency feedback.

Abstract

This paper discusses the task of face-based speech synthesis, a kind of personalized speech synthesis where the synthesized voices are constrained to perceptually match with a reference face image. Due to the lack of TTS-quality audio-visual corpora, previous approaches suffer from either low synthesis quality or domain mismatch induced by a knowledge transfer scheme. This paper proposes a new approach called Vclip that utilizes the facial-semantic knowledge of the CLIP encoder on noisy audio-visual data to learn the association between face and voice efficiently, achieving 89.63% cross-modal verification AUC score on Voxceleb testset. The proposed method then uses a retrieval-based strategy, combined with GMM-based speaker generation module for a downstream TTS system, to produce probable target speakers given reference images. Experimental results demonstrate that the proposed Vclip system in conjunction with the retrieval step can bridge the gap between face and voice features for face-based speech synthesis. And using the feedback information distilled from downstream TTS helps to synthesize voices that match closely with reference faces. Demos available at sos1sos2sixteen.github.io/vclip.