Zero-Shot Emotion Transfer For Cross-Lingual Speech Synthesis

TL;DR

This paper introduces a neural framework for zero-shot cross-lingual speech synthesis that effectively transfers emotions between languages by modeling language-specific prosody and shared emotional expressions, enhancing naturalness and emotional consistency.

Contribution

It proposes a novel neural architecture with modules for language-specific prosody and shared emotional features, enabling zero-shot emotion transfer without emotional training data.

Findings

Effective cross-lingual emotion transfer demonstrated

Improved naturalness of synthetic speech

Successful bi-lingual emotional speech synthesis

Abstract

Zero-shot emotion transfer in cross-lingual speech synthesis aims to transfer emotion from an arbitrary speech reference in the source language to the synthetic speech in the target language. Building such a system faces challenges of unnatural foreign accents and difficulty in modeling the shared emotional expressions of different languages. Building on the DelightfulTTS neural architecture, this paper addresses these challenges by introducing specifically-designed modules to model the language-specific prosody features and language-shared emotional expressions separately. Specifically, the language-specific speech prosody is learned by a non-autoregressive predictive coding (NPC) module to improve the naturalness of the synthetic cross-lingual speech. The shared emotional expression between different languages is extracted from a pre-trained self-supervised model HuBERT with strong generalization capabilities. We further use hierarchical emotion modeling to capture more comprehensive emotions across different languages. Experimental results demonstrate the proposed framework's effectiveness in synthesizing bi-lingual emotional speech for the monolingual target speaker without emotional training data.