SpeechLMScore: Evaluating speech generation using speech language model

TL;DR

SpeechLMScore is an unsupervised, scalable metric that evaluates speech quality by measuring the likelihood of generated speech sequences using a speech-language model, correlating well with human judgments.

Contribution

It introduces SpeechLMScore, a novel unsupervised metric for speech quality assessment that avoids costly human annotations and domain-shift issues.

Findings

Shows strong correlation with human evaluation scores

Effective across multiple speech generation tasks

Does not require supervised training or annotations

Abstract

While human evaluation is the most reliable metric for evaluating speech generation systems, it is generally costly and time-consuming. Previous studies on automatic speech quality assessment address the problem by predicting human evaluation scores with machine learning models. However, they rely on supervised learning and thus suffer from high annotation costs and domain-shift problems. We propose SpeechLMScore, an unsupervised metric to evaluate generated speech using a speech-language model. SpeechLMScore computes the average log-probability of a speech signal by mapping it into discrete tokens and measures the average probability of generating the sequence of tokens. Therefore, it does not require human annotation and is a highly scalable framework. Evaluation results demonstrate that the proposed metric shows a promising correlation with human evaluation scores on different speech generation tasks including voice conversion, text-to-speech, and speech enhancement.