Incremental Disentanglement for Environment-Aware Zero-Shot Text-to-Speech Synthesis

TL;DR

This paper introduces IDEA-TTS, a novel zero-shot TTS method that effectively disentangles environment, speaker, and text factors to synthesize high-quality, environment-aware speech for unseen speakers, with state-of-the-art results.

Contribution

The paper proposes an incremental disentanglement process for environment-aware zero-shot TTS, integrating environment and speaker embeddings for improved speech synthesis.

Findings

Superior speech quality and similarity in environment-aware TTS

Effective disentanglement of environment, speaker, and text factors

State-of-the-art performance in environment conversion tasks

Abstract

This paper proposes an Incremental Disentanglement-based Environment-Aware zero-shot text-to-speech (TTS) method, dubbed IDEA-TTS, that can synthesize speech for unseen speakers while preserving the acoustic characteristics of a given environment reference speech. IDEA-TTS adopts VITS as the TTS backbone. To effectively disentangle the environment, speaker, and text factors, we propose an incremental disentanglement process, where an environment estimator is designed to first decompose the environmental spectrogram into an environment mask and an enhanced spectrogram. The environment mask is then processed by an environment encoder to extract environment embeddings, while the enhanced spectrogram facilitates the subsequent disentanglement of the speaker and text factors with the condition of the speaker embeddings, which are extracted from the environmental speech using a pretrained environment-robust speaker encoder. Finally, both the speaker and environment embeddings are conditioned into the decoder for environment-aware speech generation. Experimental results demonstrate that IDEA-TTS achieves superior performance in the environment-aware TTS task, excelling in speech quality, speaker similarity, and environmental similarity. Additionally, IDEA-TTS is also capable of the acoustic environment conversion task and achieves state-of-the-art performance.