BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis

TL;DR

This paper introduces BDDM, a bilateral denoising diffusion model for speech synthesis that achieves high-quality audio generation with significantly fewer sampling steps, enabling faster and more efficient speech synthesis.

Contribution

The paper presents a novel bilateral diffusion model with a new training objective, enabling faster sampling and leveraging pre-trained models for high-quality speech synthesis.

Findings

Generates high-fidelity speech with as few as three sampling steps.

Achieves comparable or better quality than state-of-the-art vocoders.

Speeds up sampling by over 140 times compared to previous methods.

Abstract

Diffusion probabilistic models (DPMs) and their extensions have emerged as competitive generative models yet confront challenges of efficient sampling. We propose a new bilateral denoising diffusion model (BDDM) that parameterizes both the forward and reverse processes with a schedule network and a score network, which can train with a novel bilateral modeling objective. We show that the new surrogate objective can achieve a lower bound of the log marginal likelihood tighter than a conventional surrogate. We also find that BDDM allows inheriting pre-trained score network parameters from any DPMs and consequently enables speedy and stable learning of the schedule network and optimization of a noise schedule for sampling. Our experiments demonstrate that BDDMs can generate high-fidelity audio samples with as few as three sampling steps. Moreover, compared to other state-of-the-art diffusion-based neural vocoders, BDDMs produce comparable or higher quality samples indistinguishable from human speech, notably with only seven sampling steps (143x faster than WaveGrad and 28.6x faster than DiffWave). We release our code at https://github.com/tencent-ailab/bddm.