A Dataset for Automatic Assessment of TTS Quality in Spanish

TL;DR

This paper introduces a novel Spanish TTS quality assessment dataset with 4,326 samples from 52 systems, validated through subjective testing and used to train models achieving high prediction accuracy.

Contribution

It provides the first comprehensive Spanish TTS quality dataset and demonstrates its utility for training effective naturalness prediction models.

Findings

Models achieved a mean absolute error of 0.8 on the MOS scale.

The dataset covers diverse TTS systems and human voices.

Validation shows the dataset's potential to improve Spanish TTS research.

Abstract

This work addresses the development of a database for the automatic assessment of text-to-speech (TTS) systems in Spanish, aiming to improve the accuracy of naturalness prediction models. The dataset consists of 4,326 audio samples from 52 different TTS systems and human voices and is, up to our knowledge, the first of its kind in Spanish. To label the audios, a subjective test was designed based on the ITU-T Rec. P.807 standard and completed by 92 participants. Furthermore, the utility of the collected dataset was validated by training automatic naturalness prediction systems. We explored two approaches: fine-tuning an existing model originally trained for English, and training small downstream networks on top of frozen self-supervised speech models. Our models achieve a mean absolute error of 0.8 on a five-point MOS scale. Further analysis demonstrates the quality and diversity of the developed dataset, and its potential to advance TTS research in Spanish.