NVBench: A Benchmark for Speech Synthesis with Non-Verbal Vocalizations

TL;DR

NVBench is a comprehensive bilingual benchmark for evaluating speech synthesis systems' ability to generate and control non-verbal vocalizations like laughs and sighs, addressing a key gap in speech naturalness assessment.

Contribution

The paper introduces NVBench, a standardized bilingual benchmark with a multi-axis protocol for assessing NVV generation, control, and salience in speech synthesis systems.

Findings

NVV controllability often decouples from speech quality.

Low-SNR oral cues and long-duration NVVs are persistent challenges.

NVBench enables fair comparison of diverse speech synthesis systems.

Abstract

Non-verbal vocalizations (NVVs) like laugh, sigh, and sob are essential for human-like speech, yet standardized evaluation remains limited in jointly assessing whether systems can generate the intended NVVs, place them correctly, and keep them salient without harming speech. We present Non-verbal Vocalization Benchmark (NVBench), a bilingual (English/Chinese) benchmark that evaluates speech synthesis with NVVs. NVBench pairs a unified 45-type taxonomy with a curated bilingual dataset and introduces a multi-axis protocol that separates general speech naturalness and quality from NVV-specific controllability, placement, and salience. We benchmark 15 TTS systems using objective metrics, listening tests, and an LLM-based multi-rater evaluation. Results reveal that NVVs controllability often decouples from quality, while low-SNR oral cues and long-duration affective NVVs remain persistent bottlenecks. NVBench enables fair cross-system comparison across diverse control interfaces under a unified, standardized framework.