VoiceLDM: Text-to-Speech with Environmental Context

TL;DR

VoiceLDM is a novel text-to-audio model that generates contextually accurate audio from environmental and content prompts using latent diffusion, pretrained models, and dual guidance, surpassing ground truth speech intelligibility.

Contribution

The paper introduces VoiceLDM, a new model that incorporates environmental and content prompts for improved controllable text-to-audio synthesis using latent diffusion and pretrained models.

Findings

Generates plausible audio aligned with input prompts.

Surpasses ground truth speech intelligibility on AudioCaps.

Achieves competitive zero-shot TTS and T-Audio results.

Abstract

This paper presents VoiceLDM, a model designed to produce audio that accurately follows two distinct natural language text prompts: the description prompt and the content prompt. The former provides information about the overall environmental context of the audio, while the latter conveys the linguistic content. To achieve this, we adopt a text-to-audio (TTA) model based on latent diffusion models and extend its functionality to incorporate an additional content prompt as a conditional input. By utilizing pretrained contrastive language-audio pretraining (CLAP) and Whisper, VoiceLDM is trained on large amounts of real-world audio without manual annotations or transcriptions. Additionally, we employ dual classifier-free guidance to further enhance the controllability of VoiceLDM. Experimental results demonstrate that VoiceLDM is capable of generating plausible audio that aligns well with both input conditions, even surpassing the speech intelligibility of the ground truth audio on the AudioCaps test set. Furthermore, we explore the text-to-speech (TTS) and zero-shot text-to-audio capabilities of VoiceLDM and show that it achieves competitive results. Demos and code are available at https://voiceldm.github.io.