Controlling Emotion in Text-to-Speech with Natural Language Prompts

TL;DR

This paper introduces a transformer-based text-to-speech system that uses natural language prompts to control emotional expression in speech synthesis, achieving accurate emotion transfer while preserving speaker identity and speech quality.

Contribution

It presents a novel approach combining prompt-derived embeddings with speaker information for emotion control in TTS, trained on merged emotional datasets for better generalization.

Findings

Effective emotion transfer from prompts to speech

High speech quality and intelligibility maintained

Accurate speaker identity preservation

Abstract

In recent years, prompting has quickly become one of the standard ways of steering the outputs of generative machine learning models, due to its intuitive use of natural language. In this work, we propose a system conditioned on embeddings derived from an emotionally rich text that serves as prompt. Thereby, a joint representation of speaker and prompt embeddings is integrated at several points within a transformer-based architecture. Our approach is trained on merged emotional speech and text datasets and varies prompts in each training iteration to increase the generalization capabilities of the model. Objective and subjective evaluation results demonstrate the ability of the conditioned synthesis system to accurately transfer the emotions present in a prompt to speech. At the same time, precise tractability of speaker identities as well as overall high speech quality and intelligibility are maintained.