SOMOS: The Samsung Open MOS Dataset for the Evaluation of Neural Text-to-Speech Synthesis

TL;DR

The SOMOS dataset is a large-scale collection of neural TTS samples with MOS evaluations, designed to improve automatic quality prediction and evaluation of speech synthesis systems.

Contribution

It introduces the first large-scale MOS dataset for neural TTS, enabling better training and assessment of evaluation models for modern speech synthesizers.

Findings

Baseline MOS prediction models perform limitedly on SOMOS.

The dataset covers diverse acoustic models and prosodic variations.

Crowdsourcing practices yield reliable MOS annotations.

Abstract

In this work, we present the SOMOS dataset, the first large-scale mean opinion scores (MOS) dataset consisting of solely neural text-to-speech (TTS) samples. It can be employed to train automatic MOS prediction systems focused on the assessment of modern synthesizers, and can stimulate advancements in acoustic model evaluation. It consists of 20K synthetic utterances of the LJ Speech voice, a public domain speech dataset which is a common benchmark for building neural acoustic models and vocoders. Utterances are generated from 200 TTS systems including vanilla neural acoustic models as well as models which allow prosodic variations. An LPCNet vocoder is used for all systems, so that the samples' variation depends only on the acoustic models. The synthesized utterances provide balanced and adequate domain and length coverage. We collect MOS naturalness evaluations on 3 English Amazon Mechanical Turk locales and share practices leading to reliable crowdsourced annotations for this task. We provide baseline results of state-of-the-art MOS prediction models on the SOMOS dataset and show the limitations that such models face when assigned to evaluate TTS utterances.