SingMOS: An extensive Open-Source Singing Voice Dataset for MOS Prediction

TL;DR

SingMOS is a comprehensive open-source dataset of singing voice samples with human ratings, designed to improve MOS prediction models in the singing domain, addressing data scarcity and copyright issues.

Contribution

This paper introduces SingMOS, a high-quality, diverse singing voice dataset with human annotations, filling a critical gap in singing MOS prediction research.

Findings

Dataset covers Chinese and Japanese singing samples.

Data analysis confirms diversity and reliability.

Provides insights for future singing MOS prediction models.

Abstract

In speech generation tasks, human subjective ratings, usually referred to as the opinion score, are considered the "gold standard" for speech quality evaluation, with the mean opinion score (MOS) serving as the primary evaluation metric. Due to the high cost of human annotation, several MOS prediction systems have emerged in the speech domain, demonstrating good performance. These MOS prediction models are trained using annotations from previous speech-related challenges. However, compared to the speech domain, the singing domain faces data scarcity and stricter copyright protections, leading to a lack of high-quality MOS-annotated datasets for singing. To address this, we propose SingMOS, a high-quality and diverse MOS dataset for singing, covering a range of Chinese and Japanese datasets. These synthesized vocals are generated using state-of-the-art models in singing synthesis, conversion, or resynthesis tasks and are rated by professional annotators alongside real vocals. Data analysis demonstrates the diversity and reliability of our dataset. Additionally, we conduct further exploration on SingMOS, providing insights for singing MOS prediction and guidance for the continued expansion of SingMOS.