Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment

TL;DR

This paper introduces A-DMA, a novel training acceleration method for diffusion-based text-to-speech models that uses dual modality alignment to reduce training time and improve performance.

Contribution

The paper presents a new alignment pipeline leveraging both text and speech modalities to accelerate training and enhance model performance.

Findings

A-DMA doubles the convergence speed of diffusion TTS models.

A-DMA achieves superior performance compared to baseline methods.

The proposed method reduces reliance on complex diffusion learning.

Abstract

The goal of this paper is to optimize the training process of diffusion-based text-to-speech models. While recent studies have achieved remarkable advancements, their training demands substantial time and computational costs, largely due to the implicit guidance of diffusion models in learning complex intermediate representations. To address this, we propose A-DMA, an effective strategy for Accelerating training with Dual Modality Alignment. Our method introduces a novel alignment pipeline leveraging both text and speech modalities: text-guided alignment, which incorporates contextual representations, and speech-guided alignment, which refines semantic representations. By aligning hidden states with discriminative features, our training scheme reduces the reliance on diffusion models for learning complex representations. Extensive experiments demonstrate that A-DMA doubles the convergence speed while achieving superior performance over baselines. Code and demo samples are available at: https://github.com/ZhikangNiu/A-DMA