Multi-GradSpeech: Towards Diffusion-based Multi-Speaker Text-to-speech Using Consistent Diffusion Models

TL;DR

Multi-GradSpeech introduces a consistent diffusion model for multi-speaker text-to-speech, effectively reducing sampling drift and outperforming existing methods like Grad-TTS and fine-tuning in multi-speaker scenarios.

Contribution

The paper proposes the Consistent Diffusion Model (CDM) for multi-speaker TTS, addressing sampling drift issues and improving performance over prior diffusion-based models.

Findings

Significant performance improvements over Grad-TTS in multi-speaker TTS.

Outperforms fine-tuning approaches in multi-speaker scenarios.

Demonstrates the effectiveness of enforcing consistency during training.

Abstract

Despite imperfect score-matching causing drift in training and sampling distributions of diffusion models, recent advances in diffusion-based acoustic models have revolutionized data-sufficient single-speaker Text-to-Speech (TTS) approaches, with Grad-TTS being a prime example. However, the sampling drift problem leads to these approaches struggling in multi-speaker scenarios in practice due to more complex target data distribution compared to single-speaker scenarios. In this paper, we present Multi-GradSpeech, a multi-speaker diffusion-based acoustic models which introduces the Consistent Diffusion Model (CDM) as a generative modeling approach. We enforce the consistency property of CDM during the training process to alleviate the sampling drift problem in the inference stage, resulting in significant improvements in multi-speaker TTS performance. Our experimental results corroborate that our proposed approach can improve the performance of different speakers involved in multi-speaker TTS compared to Grad-TTS, even outperforming the fine-tuning approach. Audio samples are available at https://welkinyang.github.io/multi-gradspeech/