SelfTTS: cross-speaker style transfer through explicit embedding disentanglement and self-refinement using self-augmentation

TL;DR

SelfTTS is a novel TTS model that enables cross-speaker style transfer by disentangling speaker and emotion features without external encoders, using contrastive learning and self-refinement for improved naturalness.

Contribution

It introduces explicit embedding disentanglement with GRL and cosine similarity, along with MPCL and self-augmentation, to enhance cross-speaker style transfer in TTS.

Findings

Achieves superior emotional naturalness (eMOS) over baselines

Demonstrates robust stability in target timbre and emotion

Uses self-augmentation to improve speech naturalness

Abstract

This paper presents SelfTTS, a text-to-speech (TTS) model designed for cross-speaker style transfer that eliminates the need for external pre-trained speaker or emotion encoders. The architecture achieves emotional expressivity in neutral speakers through an explicit disentanglement strategy utilizing Gradient Reversal Layers (GRL) combined with cosine similarity loss to decouple speaker and emotion information. We introduce Multi Positive Contrastive Learning (MPCL) to induce clustered representations of speaker and emotion embeddings based on their respective labels. Furthermore, SelfTTS employs a self-refinement strategy via Self-Augmentation, exploiting the model's voice conversion capabilities to enhance the naturalness of synthesized speech. Experimental results demonstrate that SelfTTS achieves superior emotional naturalness (eMOS) and robust stability in target timbre and emotion compared to state-of-the-art baselines.