DRSpeech: Degradation-Robust Text-to-Speech Synthesis with Frame-Level and Utterance-Level Acoustic Representation Learning

TL;DR

DRSpeech introduces a novel noise-robust TTS approach that effectively handles both additive noise and environmental distortions by joint frame-level and utterance-level acoustic representation learning, leading to higher-quality speech synthesis.

Contribution

The paper presents a degradation-robust TTS framework with a new regularization technique for disentangling environmental embeddings from linguistic and speaker information.

Findings

Significantly improved speech quality in noisy and reverberant conditions.

Effective joint modeling of time-variant and time-invariant noises.

Outperforms previous noise-robust TTS methods.

Abstract

Most text-to-speech (TTS) methods use high-quality speech corpora recorded in a well-designed environment, incurring a high cost for data collection. To solve this problem, existing noise-robust TTS methods are intended to use noisy speech corpora as training data. However, they only address either time-invariant or time-variant noises. We propose a degradation-robust TTS method, which can be trained on speech corpora that contain both additive noises and environmental distortions. It jointly represents the time-variant additive noises with a frame-level encoder and the time-invariant environmental distortions with an utterance-level encoder. We also propose a regularization method to attain clean environmental embedding that is disentangled from the utterance-dependent information such as linguistic contents and speaker characteristics. Evaluation results show that our method achieved significantly higher-quality synthetic speech than previous methods in the condition including both additive noise and reverberation.