InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems

TL;DR

InstructTTSEval introduces a comprehensive benchmark with tasks and metrics to evaluate complex natural-language instruction following in text-to-speech systems, aiming to improve flexibility and accuracy.

Contribution

The paper presents InstructTTSEval, a new benchmark with diverse tasks and an automatic evaluation method for assessing instruction-following in TTS models.

Findings

Current TTS systems show significant room for improvement in instruction following.

The benchmark includes English and Chinese datasets with 6,000 test cases.

Automatic evaluation with Gemini provides consistent assessment of instruction adherence.

Abstract

In modern speech synthesis, paralinguistic information--such as a speaker's vocal timbre, emotional state, and dynamic prosody--plays a critical role in conveying nuance beyond mere semantics. Traditional Text-to-Speech (TTS) systems rely on fixed style labels or inserting a speech prompt to control these cues, which severely limits flexibility. Recent attempts seek to employ natural-language instructions to modulate paralinguistic features, substantially improving the generalization of instruction-driven TTS models. Although many TTS systems now support customized synthesis via textual description, their actual ability to interpret and execute complex instructions remains largely unexplored. In addition, there is still a shortage of high-quality benchmarks and automated evaluation metrics specifically designed for instruction-based TTS, which hinders accurate assessment and iterative optimization of these models. To address these limitations, we introduce InstructTTSEval, a benchmark for measuring the capability of complex natural-language style control. We introduce three tasks, namely Acoustic-Parameter Specification, Descriptive-Style Directive, and Role-Play, including English and Chinese subsets, each with 1k test cases (6k in total) paired with reference audio. We leverage Gemini as an automatic judge to assess their instruction-following abilities. Our evaluation of accessible instruction-following TTS systems highlights substantial room for further improvement. We anticipate that InstructTTSEval will drive progress toward more powerful, flexible, and accurate instruction-following TTS.