UMETTS: A Unified Framework for Emotional Text-to-Speech Synthesis with Multimodal Prompts

TL;DR

UMETTS introduces a multimodal framework for emotional TTS that aligns emotional cues from text, audio, and visual inputs to produce more expressive and emotionally accurate speech, surpassing existing methods.

Contribution

The paper presents UMETTS, a novel multimodal emotional TTS framework with contrastive learning for emotion alignment and improved speech synthesis quality.

Findings

Enhanced emotion accuracy in synthesized speech

Improved speech naturalness over traditional E-TTS

Effective multimodal emotional cue integration

Abstract

Emotional Text-to-Speech (E-TTS) synthesis has garnered significant attention in recent years due to its potential to revolutionize human-computer interaction. However, current E-TTS approaches often struggle to capture the intricacies of human emotions, primarily relying on oversimplified emotional labels or single-modality input. In this paper, we introduce the Unified Multimodal Prompt-Induced Emotional Text-to-Speech System (UMETTS), a novel framework that leverages emotional cues from multiple modalities to generate highly expressive and emotionally resonant speech. The core of UMETTS consists of two key components: the Emotion Prompt Alignment Module (EP-Align) and the Emotion Embedding-Induced TTS Module (EMI-TTS). (1) EP-Align employs contrastive learning to align emotional features across text, audio, and visual modalities, ensuring a coherent fusion of multimodal information. (2) Subsequently, EMI-TTS integrates the aligned emotional embeddings with state-of-the-art TTS models to synthesize speech that accurately reflects the intended emotions. Extensive evaluations show that UMETTS achieves significant improvements in emotion accuracy and speech naturalness, outperforming traditional E-TTS methods on both objective and subjective metrics.