ClapFM-EVC: High-Fidelity and Flexible Emotional Voice Conversion with Dual Control from Natural Language and Speech

TL;DR

ClapFM-EVC is a novel emotional voice conversion framework that achieves high-fidelity, flexible, and interpretable emotional speech synthesis driven by natural language prompts or reference speech, with adjustable emotion intensity.

Contribution

It introduces EVC-CLAP for cross-modal emotional feature extraction and a flow matching model for high-quality speech reconstruction, advancing emotional voice conversion technology.

Findings

High-quality emotional speech conversion validated by evaluations

Flexible control via natural language prompts and reference speech

Enhanced emotion expressiveness and naturalness

Abstract

Despite great advances, achieving high-fidelity emotional voice conversion (EVC) with flexible and interpretable control remains challenging. This paper introduces ClapFM-EVC, a novel EVC framework capable of generating high-quality converted speech driven by natural language prompts or reference speech with adjustable emotion intensity. We first propose EVC-CLAP, an emotional contrastive language-audio pre-training model, guided by natural language prompts and categorical labels, to extract and align fine-grained emotional elements across speech and text modalities. Then, a FuEncoder with an adaptive intensity gate is presented to seamless fuse emotional features with Phonetic PosteriorGrams from a pre-trained ASR model. To further improve emotion expressiveness and speech naturalness, we propose a flow matching model conditioned on these captured features to reconstruct Mel-spectrogram of source speech. Subjective and objective evaluations validate the effectiveness of ClapFM-EVC.