EmoCat: Language-agnostic Emotional Voice Conversion

TL;DR

EmoCat is a novel language-agnostic emotional voice conversion model that uses adversarial training and a gradient reversal technique to effectively transfer emotions across languages with limited data.

Contribution

It introduces a new adversarial training method with a gradient reversal enhancement for improved emotion transfer in voice conversion.

Findings

Achieves high-quality emotion conversion in German with less than 45 minutes of data.

Can convert to different emotions but has limitations in emotion intensity.

Maintains audio quality comparable to original recordings for most emotion intensities.

Abstract

Emotional voice conversion models adapt the emotion in speech without changing the speaker identity or linguistic content. They are less data hungry than text-to-speech models and allow to generate large amounts of emotional data for downstream tasks. In this work we propose EmoCat, a language-agnostic emotional voice conversion model. It achieves high-quality emotion conversion in German with less than 45 minutes of German emotional recordings by exploiting large amounts of emotional data in US English. EmoCat is an encoder-decoder model based on CopyCat, a voice conversion system which transfers prosody. We use adversarial training to remove emotion leakage from the encoder to the decoder. The adversarial training is improved by a novel contribution to gradient reversal to truly reverse gradients. This allows to remove only the leaking information and to converge to better optima with higher conversion performance. Evaluations show that Emocat can convert to different emotions but misses on emotion intensity compared to the recordings, especially for very expressive emotions. EmoCat is able to achieve audio quality on par with the recordings for five out of six tested emotion intensities.