In-the-wild Speech Emotion Conversion Using Disentangled Self-Supervised Representations and Neural Vocoder-based Resynthesis

TL;DR

This paper presents a novel in-the-wild speech emotion conversion method that disentangles lexical, speaker, and emotional features using self-supervised learning and resynthesizes speech with a neural vocoder, focusing on arousal control.

Contribution

It introduces a self-supervised disentanglement approach combined with neural vocoder resynthesis for emotion conversion without parallel data, emphasizing arousal dimension control.

Findings

Effective emotion conversion conditioned on emotional content.

Synthesizes natural speech with better quality for mid-scale arousal.

Less accurate synthesis for extreme arousal levels.

Abstract

Speech emotion conversion aims to convert the expressed emotion of a spoken utterance to a target emotion while preserving the lexical information and the speaker's identity. In this work, we specifically focus on in-the-wild emotion conversion where parallel data does not exist, and the problem of disentangling lexical, speaker, and emotion information arises. In this paper, we introduce a methodology that uses self-supervised networks to disentangle the lexical, speaker, and emotional content of the utterance, and subsequently uses a HiFiGAN vocoder to resynthesise the disentangled representations to a speech signal of the targeted emotion. For better representation and to achieve emotion intensity control, we specifically focus on the aro\-usal dimension of continuous representations, as opposed to performing emotion conversion on categorical representations. We test our methodology on the large in-the-wild MSP-Podcast dataset. Results reveal that the proposed approach is aptly conditioned on the emotional content of input speech and is capable of synthesising natural-sounding speech for a target emotion. Results further reveal that the methodology better synthesises speech for mid-scale arousal (2 to 6) than for extreme arousal (1 and 7).