Residual-guided Personalized Speech Synthesis based on Face Image

TL;DR

This paper introduces a novel face-based residual-guided model for personalized speech synthesis that leverages face images to generate speech features, reducing the need for extensive audio data training.

Contribution

The work proposes a face-encoder integrated residual-guided approach with a new tri-item loss function for personalized speech synthesis, innovatively using face images instead of large audio datasets.

Findings

Speech quality comparable to audio-based models

Effective face feature guidance improves synthesis

Reduces dependency on large audio datasets

Abstract

Previous works derive personalized speech features by training the model on a large dataset composed of his/her audio sounds. It was reported that face information has a strong link with the speech sound. Thus in this work, we innovatively extract personalized speech features from human faces to synthesize personalized speech using neural vocoder. A Face-based Residual Personalized Speech Synthesis Model (FR-PSS) containing a speech encoder, a speech synthesizer and a face encoder is designed for PSS. In this model, by designing two speech priors, a residual-guided strategy is introduced to guide the face feature to approach the true speech feature in the training. Moreover, considering the error of feature's absolute values and their directional bias, we formulate a novel tri-item loss function for face encoder. Experimental results show that the speech synthesized by our model is comparable to the personalized speech synthesized by training a large amount of audio data in previous works.