MPE-TTS: Customized Emotion Zero-Shot Text-To-Speech Using Multi-Modal Prompt

TL;DR

This paper introduces MPE-TTS, a multi-modal prompt-based zero-shot TTS system that disentangles speech components and uses diverse prompts to generate expressive speech with improved naturalness and similarity.

Contribution

The paper presents a novel multi-modal prompt emotion encoder and a diffusion-based acoustic model for zero-shot TTS with customizable emotions from various prompt types.

Findings

Outperforms existing ZS-TTS systems in naturalness and similarity

Effectively utilizes text, image, and speech prompts for emotion control

Demonstrates superior performance through objective and subjective evaluations

Abstract

Most existing Zero-Shot Text-To-Speech(ZS-TTS) systems generate the unseen speech based on single prompt, such as reference speech or text descriptions, which limits their flexibility. We propose a customized emotion ZS-TTS system based on multi-modal prompt. The system disentangles speech into the content, timbre, emotion and prosody, allowing emotion prompts to be provided as text, image or speech. To extract emotion information from different prompts, we propose a multi-modal prompt emotion encoder. Additionally, we introduce an prosody predictor to fit the distribution of prosody and propose an emotion consistency loss to preserve emotion information in the predicted prosody. A diffusion-based acoustic model is employed to generate the target mel-spectrogram. Both objective and subjective experiments demonstrate that our system outperforms existing systems in terms of naturalness and similarity. The samples are available at https://mpetts-demo.github.io/mpetts_demo/.