Rethinking Mean Opinion Scores in Speech Quality Assessment: Aggregation through Quantized Distribution Fitting

TL;DR

This paper introduces a novel score aggregation method for speech quality assessment that models annotator ratings as quantized continuous distributions, leading to improved MOS prediction accuracy.

Contribution

It proposes a new aggregation technique based on quantized distribution fitting that better captures the annotators' underlying continuous score considerations.

Findings

Improved MOS prediction performance with the new aggregation method.

Modeling ratings as quantized distributions enhances speech quality assessment accuracy.

The method outperforms traditional MOSNet predictions.

Abstract

Speech quality assessment (SQA) aims to evaluate the quality of speech samples without relying on time-consuming listener questionnaires. Recent efforts have focused on training neural-based SQA models to predict the mean opinion score (MOS) of speech samples produced by text-to-speech or voice conversion systems. This paper targets the enhancement of MOS prediction models' performance. We propose a novel score aggregation method to address the limitations of conventional annotations for MOS, which typically involve ratings on a scale from 1 to 5. Our method is based on the hypothesis that annotators internally consider continuous scores and then choose the nearest discrete rating. By modeling this process, we approximate the generative distribution of ratings by quantizing the latent continuous distribution. We then use the peak of this latent distribution, estimated through the loss between the quantized distribution and annotated ratings, as a new representative value instead of MOS. Experimental results demonstrate that substituting MOSNet's predicted target with this proposed value improves prediction performance.