A Study on Incorporating Whisper for Robust Speech Assessment

TL;DR

This paper presents MOSA-Net+, an improved speech assessment model that leverages Whisper's acoustic features to enhance the accuracy and robustness of subjective speech quality and intelligibility predictions, outperforming existing models in various tests.

Contribution

The study introduces MOSA-Net+ which effectively incorporates Whisper's features, demonstrating significant performance gains over prior models in noisy and challenging conditions.

Findings

Whisper's embeddings improve prediction accuracy.

Combining Whisper with SSL models yields marginal gains.

MOSA-Net+ outperforms existing models in TMHINT-QI and VoiceMOS Challenge 2023.

Abstract

This research introduces an enhanced version of the multi-objective speech assessment model--MOSA-Net+, by leveraging the acoustic features from Whisper, a large-scaled weakly supervised model. We first investigate the effectiveness of Whisper in deploying a more robust speech assessment model. After that, we explore combining representations from Whisper and SSL models. The experimental results reveal that Whisper's embedding features can contribute to more accurate prediction performance. Moreover, combining the embedding features from Whisper and SSL models only leads to marginal improvement. As compared to intrusive methods, MOSA-Net, and other SSL-based speech assessment models, MOSA-Net+ yields notable improvements in estimating subjective quality and intelligibility scores across all evaluation metrics in Taiwan Mandarin Hearing In Noise test - Quality & Intelligibility (TMHINT-QI) dataset. To further validate its robustness, MOSA-Net+ was tested in the noisy-and-enhanced track of the VoiceMOS Challenge 2023, where it obtained the top-ranked performance among nine systems.