DiEmo-TTS: Disentangled Emotion Representations via Self-Supervised Distillation for Cross-Speaker Emotion Transfer in Text-to-Speech

TL;DR

DiEmo-TTS introduces a self-supervised distillation approach with clustering and dual transformer conditioning to improve cross-speaker emotion transfer by better disentangling emotion from speaker traits in speech synthesis.

Contribution

It presents a novel self-supervised distillation framework with clustering and a dual transformer to enhance emotion-speech disentanglement in TTS, especially for unlabeled data.

Findings

Effective emotion embedding disentanglement demonstrated

Improved cross-speaker emotion transfer quality

Robustness to unlabeled data scenarios

Abstract

Cross-speaker emotion transfer in speech synthesis relies on extracting speaker-independent emotion embeddings for accurate emotion modeling without retaining speaker traits. However, existing timbre compression methods fail to fully separate speaker and emotion characteristics, causing speaker leakage and degraded synthesis quality. To address this, we propose DiEmo-TTS, a self-supervised distillation method to minimize emotional information loss and preserve speaker identity. We introduce cluster-driven sampling and information perturbation to preserve emotion while removing irrelevant factors. To facilitate this process, we propose an emotion clustering and matching approach using emotional attribute prediction and speaker embeddings, enabling generalization to unlabeled data. Additionally, we designed a dual conditioning transformer to integrate style features better. Experimental results confirm the effectiveness of our method in learning speaker-irrelevant emotion embeddings.