Cross-Utterance Conditioned VAE for Speech Generation

TL;DR

This paper introduces CUC-VAE S2, a novel speech synthesis framework that uses cross-utterance information and VAEs to generate more natural, expressive, and editable speech by leveraging contextual prosody from surrounding sentences.

Contribution

The paper proposes a new cross-utterance conditioned VAE framework with algorithms for TTS and speech editing, improving naturalness and flexibility in speech synthesis.

Findings

Enhanced prosody and naturalness in synthesized speech.

Effective speech editing with realistic audio generation.

Significant improvements demonstrated on LibriTTS dataset.

Abstract

Speech synthesis systems powered by neural networks hold promise for multimedia production, but frequently face issues with producing expressive speech and seamless editing. In response, we present the Cross-Utterance Conditioned Variational Autoencoder speech synthesis (CUC-VAE S2) framework to enhance prosody and ensure natural speech generation. This framework leverages the powerful representational capabilities of pre-trained language models and the re-expression abilities of variational autoencoders (VAEs). The core component of the CUC-VAE S2 framework is the cross-utterance CVAE, which extracts acoustic, speaker, and textual features from surrounding sentences to generate context-sensitive prosodic features, more accurately emulating human prosody generation. We further propose two practical algorithms tailored for distinct speech synthesis applications: CUC-VAE TTS for text-to-speech and CUC-VAE SE for speech editing. The CUC-VAE TTS is a direct application of the framework, designed to generate audio with contextual prosody derived from surrounding texts. On the other hand, the CUC-VAE SE algorithm leverages real mel spectrogram sampling conditioned on contextual information, producing audio that closely mirrors real sound and thereby facilitating flexible speech editing based on text such as deletion, insertion, and replacement. Experimental results on the LibriTTS datasets demonstrate that our proposed models significantly enhance speech synthesis and editing, producing more natural and expressive speech.