An Exploration of ECAPA-TDNN and x-vector Speaker Representations in Zero-shot Multi-speaker TTS

TL;DR

This study compares different speaker embedding methods in zero-shot multi-speaker TTS, revealing that the original H/ASP encoder outperforms ECAPA-TDNN and x-vectors in speaker similarity, emphasizing the need for empirical validation.

Contribution

It provides a systematic comparison of speaker encoders in zero-shot TTS, demonstrating that popular recognition embeddings may not always be optimal for TTS applications.

Findings

H/ASP encoder outperforms ECAPA-TDNN and x-vectors in speaker similarity

ECAPA-TDNN performs better than x-vectors but worse than H/ASP

Empirical evaluation highlights the importance of task-specific testing for speaker embeddings

Abstract

Zero-shot multi-speaker text-to-speech (TTS) systems rely on speaker embeddings to synthesize speech in the voice of an unseen speaker, using only a short reference utterance. While many speaker embeddings have been developed for speaker recognition, their relative effectiveness in zero-shot TTS remains underexplored. In this work, we employ a YourTTS-based TTS system to compare three different speaker encoders - YourTTS's original H/ASP encoder, x-vector embeddings, and ECAPA-TDNN embeddings - within an otherwise fixed zero-shot TTS framework. All models were trained on the same dataset of Czech read speech and evaluated on 24 out-of-domain target speakers using both subjective and objective methods. The subjective evaluation was conducted via a listening test focused on speaker similarity, while the objective evaluation measured cosine distances between speaker embeddings extracted from synthesized and real utterances. Across both evaluations, the original H/ASP encoder consistently outperformed the alternatives, with ECAPA-TDNN showing better results than x-vectors. These findings suggest that, despite the popularity of ECAPA-TDNN in speaker recognition, it does not necessarily offer improvements for speaker similarity in zero-shot TTS in this configuration. Our study highlights the importance of empirical evaluation when reusing speaker recognition embeddings in TTS and provides a framework for additional future comparisons.