High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models

TL;DR

This paper introduces a diffusion model-based, minimally-supervised speech synthesis system that improves controllability, prosody diversity, and audio fidelity by addressing limitations of existing methods.

Contribution

It presents a novel diffusion model framework for high-fidelity speech synthesis with minimal supervision, utilizing contrastive token-acoustic pretraining and continuous regression tasks.

Findings

Outperforms baseline methods in quality

Enhances prosodic diversity and controllability

Reduces waveform distortion

Abstract

Text-to-speech (TTS) methods have shown promising results in voice cloning, but they require a large number of labeled text-speech pairs. Minimally-supervised speech synthesis decouples TTS by combining two types of discrete speech representations(semantic \& acoustic) and using two sequence-to-sequence tasks to enable training with minimal supervision. However, existing methods suffer from information redundancy and dimension explosion in semantic representation, and high-frequency waveform distortion in discrete acoustic representation. Autoregressive frameworks exhibit typical instability and uncontrollability issues. And non-autoregressive frameworks suffer from prosodic averaging caused by duration prediction models. To address these issues, we propose a minimally-supervised high-fidelity speech synthesis method, where all modules are constructed based on the diffusion models. The non-autoregressive framework enhances controllability, and the duration diffusion model enables diversified prosodic expression. Contrastive Token-Acoustic Pretraining (CTAP) is used as an intermediate semantic representation to solve the problems of information redundancy and dimension explosion in existing semantic coding methods. Mel-spectrogram is used as the acoustic representation. Both semantic and acoustic representations are predicted by continuous variable regression tasks to solve the problem of high-frequency fine-grained waveform distortion. Experimental results show that our proposed method outperforms the baseline method. We provide audio samples on our website.