Boosting Diffusion Model for Spectrogram Up-sampling in Text-to-speech: An Empirical Study

TL;DR

This paper empirically investigates diffusion models for spectrogram up-sampling in text-to-speech systems, aiming to improve quality and efficiency for real-time applications.

Contribution

It introduces a systematic study of diffusion model architectures and objectives specifically for spectrogram up-sampling in TTS, addressing a key bottleneck.

Findings

Improved speech quality with diffusion-based up-sampling.

Enhanced efficiency suitable for streaming synthesis.

Better objective and subjective metrics compared to baseline methods.

Abstract

Scaling text-to-speech (TTS) with autoregressive language model (LM) to large-scale datasets by quantizing waveform into discrete speech tokens is making great progress to capture the diversity and expressiveness in human speech, but the speech reconstruction quality from discrete speech token is far from satisfaction depending on the compressed speech token compression ratio. Generative diffusion models trained with score-matching loss and continuous normalized flow trained with flow-matching loss have become prominent in generation of images as well as speech. LM based TTS systems usually quantize speech into discrete tokens and generate these tokens autoregressively, and finally use a diffusion model to up sample coarse-grained speech tokens into fine-grained codec features or mel-spectrograms before reconstructing into waveforms with vocoder, which has a high latency and is not realistic for real time speech applications. In this paper, we systematically investigate varied diffusion models for up sampling stage, which is the main bottleneck for streaming synthesis of LM and diffusion-based architecture, we present the model architecture, objective and subjective metrics to show quality and efficiency improvement.