PromptEVC: Controllable Emotional Voice Conversion with Natural Language Prompts

TL;DR

PromptEVC introduces a novel method for controllable emotional voice conversion using natural language prompts, enabling precise emotion manipulation and improved naturalness in synthesized speech.

Contribution

It proposes a new framework that uses natural language prompts and emotion descriptors to achieve flexible and fine-grained emotion control in voice conversion.

Findings

Outperforms state-of-the-art methods in emotion conversion accuracy

Enables detailed control over emotion intensity and mixed emotions

Enhances naturalness through prosody modeling and speaker identity preservation

Abstract

Controllable emotional voice conversion (EVC) aims to manipulate emotional expressions to increase the diversity of synthesized speech. Existing methods typically rely on predefined labels, reference audios, or prespecified factor values, often overlooking individual differences in emotion perception and expression. In this paper, we introduce PromptEVC that utilizes natural language prompts for precise and flexible emotion control. To bridge text descriptions with emotional speech, we propose emotion descriptor and prompt mapper to generate fine-grained emotion embeddings, trained jointly with reference embeddings. To enhance naturalness, we present a prosody modeling and control pipeline that adjusts the rhythm based on linguistic content and emotional cues. Additionally, a speaker encoder is incorporated to preserve identity. Experimental results demonstrate that PromptEVC outperforms state-of-the-art controllable EVC methods in emotion conversion, intensity control, mixed emotion synthesis, and prosody manipulation. Speech samples are available at https://jeremychee4.github.io/PromptEVC/.