SATTS: Speaker Attractor Text to Speech, Learning to Speak by Learning to Separate

TL;DR

SATTS introduces a novel TTS method leveraging speaker attractors for zero-shot speaker adaptation, enabling natural speech synthesis for unseen speakers even with suboptimal reference recordings.

Contribution

This work pioneers the use of speaker attractors for zero-shot multi-speaker TTS, enhancing speaker adaptation without extensive training data.

Findings

SATTS can synthesize natural speech for unseen speakers.

Effective even with reverberant or mixed reference signals.

Demonstrates improved speaker adaptation in TTS systems.

Abstract

The mapping of text to speech (TTS) is non-deterministic, letters may be pronounced differently based on context, or phonemes can vary depending on various physiological and stylistic factors like gender, age, accent, emotions, etc. Neural speaker embeddings, trained to identify or verify speakers are typically used to represent and transfer such characteristics from reference speech to synthesized speech. Speech separation on the other hand is the challenging task of separating individual speakers from an overlapping mixed signal of various speakers. Speaker attractors are high-dimensional embedding vectors that pull the time-frequency bins of each speaker's speech towards themselves while repelling those belonging to other speakers. In this work, we explore the possibility of using these powerful speaker attractors for zero-shot speaker adaptation in multi-speaker TTS synthesis and propose speaker attractor text to speech (SATTS). Through various experiments, we show that SATTS can synthesize natural speech from text from an unseen target speaker's reference signal which might have less than ideal recording conditions, i.e. reverberations or mixed with other speakers.