The VoiceMOS Challenge 2022

TL;DR

The VoiceMOS Challenge 2022 aimed to advance automatic prediction of human-rated speech quality, involving diverse approaches and revealing the effectiveness of fine-tuning self-supervised models while highlighting challenges with unseen data.

Contribution

This paper introduces the first VoiceMOS Challenge, providing a benchmark for MOS prediction and analyzing the effectiveness of different approaches on diverse speech synthesis data.

Findings

Fine-tuning self-supervised speech models improves MOS prediction accuracy.

Predicting MOS for unseen speakers and systems remains challenging.

The challenge attracted 22 teams from academia and industry.

Abstract

We present the first edition of the VoiceMOS Challenge, a scientific event that aims to promote the study of automatic prediction of the mean opinion score (MOS) of synthetic speech. This challenge drew 22 participating teams from academia and industry who tried a variety of approaches to tackle the problem of predicting human ratings of synthesized speech. The listening test data for the main track of the challenge consisted of samples from 187 different text-to-speech and voice conversion systems spanning over a decade of research, and the out-of-domain track consisted of data from more recent systems rated in a separate listening test. Results of the challenge show the effectiveness of fine-tuning self-supervised speech models for the MOS prediction task, as well as the difficulty of predicting MOS ratings for unseen speakers and listeners, and for unseen systems in the out-of-domain setting.