OV-InstructTTS: Towards Open-Vocabulary Instruct Text-to-Speech

TL;DR

This paper introduces OV-InstructTTS, a novel open-vocabulary Instruct Text-to-Speech system that uses a reasoning-driven framework and a new dataset to better follow high-level instructions and enhance speech expressiveness.

Contribution

The paper presents OV-InstructTTS, a new paradigm with a curated dataset and reasoning framework that improves instruction-following and expressiveness in TTS systems.

Findings

Significant improvement in instruction-following fidelity.

Enhanced speech expressiveness through reasoning-based synthesis.

Effective handling of high-level, open-vocabulary instructions.

Abstract

Instruct Text-to-Speech (InstructTTS) leverages natural language descriptions as style prompts to guide speech synthesis. However, existing InstructTTS methods mainly rely on a direct combination of audio-related labels or their diverse rephrasings, making it difficult to handle flexible, high-level instructions. Such rigid control is insufficient for users such as content creators who wish to steer generation with descriptive instructions. To address these constraints, we introduce OV-InstructTTS, a new paradigm for open-vocabulary InstructTTS. We propose a comprehensive solution comprising a newly curated dataset, OV-Speech, and a novel reasoning-driven framework. The OV-Speech dataset pairs speech with open-vocabulary instructions, each augmented with a reasoning process that connects high-level instructions to acoustic features. The reasoning-driven framework infers emotional, acoustic, and paralinguistic information from open-vocabulary instructions before synthesizing speech. Evaluations show that this reasoning-driven approach significantly improves instruction-following fidelity and speech expressiveness. We believe this work can inspire the next user-friendly InstructTTS systems with stronger generalization and real-world applicability. The dataset and demos are publicly available on our project page.