NonverbalTTS: A Public English Corpus of Text-Aligned Nonverbal Vocalizations with Emotion Annotations for Text-to-Speech

TL;DR

NonverbalTTS introduces a comprehensive, annotated dataset of nonverbal vocalizations with emotion labels to enhance expressive speech synthesis models, enabling better modeling of diverse NVs in TTS systems.

Contribution

The paper presents a new open-source dataset with diverse NVs and emotions, along with a pipeline for annotation and demonstrates improved TTS performance using this dataset.

Findings

Fine-tuning TTS models on NVTTS achieves parity with closed-source systems.

The dataset improves modeling of nonverbal vocalizations in TTS.

Automated detection and human validation ensure high-quality annotations.

Abstract

Current expressive speech synthesis models are constrained by the limited availability of open-source datasets containing diverse nonverbal vocalizations (NVs). In this work, we introduce NonverbalTTS (NVTTS), a 17-hour open-access dataset annotated with 10 types of NVs (e.g., laughter, coughs) and 8 emotional categories. The dataset is derived from popular sources, VoxCeleb and Expresso, using automated detection followed by human validation. We propose a comprehensive pipeline that integrates automatic speech recognition (ASR), NV tagging, emotion classification, and a fusion algorithm to merge transcriptions from multiple annotators. Fine-tuning open-source text-to-speech (TTS) models on the NVTTS dataset achieves parity with closed-source systems such as CosyVoice2, as measured by both human evaluation and automatic metrics, including speaker similarity and NV fidelity. By releasing NVTTS and its accompanying annotation guidelines, we address a key bottleneck in expressive TTS research. The dataset is available at https://huggingface.co/datasets/deepvk/NonverbalTTS.