Partial Rank Similarity Minimization Method for Quality MOS Prediction of Unseen Speech Synthesis Systems in Zero-Shot and Semi-supervised setting

TL;DR

This paper proposes a partial rank similarity (PRS) loss function for predicting speech quality scores, which outperforms traditional methods in zero-shot and semi-supervised scenarios by focusing on relative ranking rather than absolute MOS values.

Contribution

The introduction of the PRS loss function that emphasizes rank order in MOS prediction models for unseen speech synthesis systems, improving zero-shot and semi-supervised performance.

Findings

PRS outperforms L1 loss in correlation with ground truth

Rank order consideration improves MOS prediction robustness

MSE and linear correlation may be unreliable metrics

Abstract

This paper introduces a novel objective function for quality mean opinion score (MOS) prediction of unseen speech synthesis systems. The proposed function measures the similarity of relative positions of predicted MOS values, in a mini-batch, rather than the actual MOS values. That is the partial rank similarity is measured (PRS) rather than the individual MOS values as with the L1 loss. Our experiments on out-of-domain speech synthesis systems demonstrate that the PRS outperforms L1 loss in zero-shot and semi-supervised settings, exhibiting stronger correlation with ground truth. These findings highlight the importance of considering rank order, as done by PRS, when training MOS prediction models. We also argue that mean squared error and linear correlation coefficient metrics may be unreliable for evaluating MOS prediction models. In conclusion, PRS-trained models provide a robust framework for evaluating speech quality and offer insights for developing high-quality speech synthesis systems. Code and models are available at github.com/nii-yamagishilab/partial_rank_similarity/