Content-Dependent Fine-Grained Speaker Embedding for Zero-Shot Speaker Adaptation in Text-to-Speech Synthesis

TL;DR

This paper introduces a content-dependent fine-grained speaker embedding method for zero-shot text-to-speech adaptation, significantly improving speaker similarity by capturing personal pronunciation nuances.

Contribution

It proposes a novel local content embedding approach with a reference attention module to better model individual pronunciation characteristics in zero-shot TTS.

Findings

Enhanced speaker similarity in synthesized speech for unseen speakers.

Effective modeling of personal pronunciation traits improves TTS quality.

Outperforms previous global speaker embedding methods.

Abstract

Zero-shot speaker adaptation aims to clone an unseen speaker's voice without any adaptation time and parameters. Previous researches usually use a speaker encoder to extract a global fixed speaker embedding from reference speech, and several attempts have tried variable-length speaker embedding. However, they neglect to transfer the personal pronunciation characteristics related to phoneme content, leading to poor speaker similarity in terms of detailed speaking styles and pronunciation habits. To improve the ability of the speaker encoder to model personal pronunciation characteristics, we propose content-dependent fine-grained speaker embedding for zero-shot speaker adaptation. The corresponding local content embeddings and speaker embeddings are extracted from a reference speech, respectively. Instead of modeling the temporal relations, a reference attention module is introduced to model the content relevance between the reference speech and the input text, and to generate the fine-grained speaker embedding for each phoneme encoder output. The experimental results show that our proposed method can improve speaker similarity of synthesized speeches, especially for unseen speakers.