Controllable Speaking Styles Using a Large Language Model

TL;DR

This paper introduces a novel approach using large language models to directly suggest prosodic modifications for controllable TTS, enabling style and dialogue-appropriate prosody without reference utterances.

Contribution

It proposes leveraging LLMs to guide prosodic control in TTS directly from prompts, bypassing the need for reference utterances or prompt-labelled speech training.

Findings

Rated most appropriate in 50% of cases versus 31% for baseline.

Demonstrated control of speaking style and dialogue-appropriate prosody.

Effective prompt-based control without reference utterances.

Abstract

Reference-based Text-to-Speech (TTS) models can generate multiple, prosodically-different renditions of the same target text. Such models jointly learn a latent acoustic space during training, which can be sampled from during inference. Controlling these models during inference typically requires finding an appropriate reference utterance, which is non-trivial. Large generative language models (LLMs) have shown excellent performance in various language-related tasks. Given only a natural language query text (the prompt), such models can be used to solve specific, context-dependent tasks. Recent work in TTS has attempted similar prompt-based control of novel speaking style generation. Those methods do not require a reference utterance and can, under ideal conditions, be controlled with only a prompt. But existing methods typically require a prompt-labelled speech corpus for jointly training a prompt-conditioned encoder. In contrast, we instead employ an LLM to directly suggest prosodic modifications for a controllable TTS model, using contextual information provided in the prompt. The prompt can be designed for a multitude of tasks. Here, we give two demonstrations: control of speaking style; prosody appropriate for a given dialogue context. The proposed method is rated most appropriate in 50% of cases vs. 31% for a baseline model.