A Diffeomorphic Flow-based Variational Framework for Multi-speaker Emotion Conversion

TL;DR

This paper presents a novel variational CycleGAN framework for non-parallel multi-speaker emotion conversion in speech, utilizing a stochastic loss with KL divergence and a learnable prosodic deformation for improved speaker generalization.

Contribution

It introduces a variational CycleGAN with a KL divergence term and a learnable prosodic deformation model, enhancing emotion conversion robustness and out-of-distribution speaker handling.

Findings

High performance against state-of-the-art baselines

Robustness to unseen speakers

Effective prosodic feature modeling

Abstract

This paper introduces a new framework for non-parallel emotion conversion in speech. Our framework is based on two key contributions. First, we propose a stochastic version of the popular CycleGAN model. Our modified loss function introduces a Kullback Leibler (KL) divergence term that aligns the source and target data distributions learned by the generators, thus overcoming the limitations of sample wise generation. By using a variational approximation to this stochastic loss function, we show that our KL divergence term can be implemented via a paired density discriminator. We term this new architecture a variational CycleGAN (VCGAN). Second, we model the prosodic features of target emotion as a smooth and learnable deformation of the source prosodic features. This approach provides implicit regularization that offers key advantages in terms of better range alignment to unseen and out of distribution speakers. We conduct rigorous experiments and comparative studies to demonstrate that our proposed framework is fairly robust with high performance against several state-of-the-art baselines.