ReFESS-QI: Reference-Free Evaluation For Speech Separation With Joint Quality And Intelligibility Scoring

TL;DR

This paper presents ReFESS-QI, a reference-free, self-supervised evaluation framework for speech separation that jointly predicts audio quality and intelligibility without needing reference audios.

Contribution

It introduces a novel SSL-based method for evaluating speech separation quality and intelligibility directly from mixtures and separated tracks, applicable in real-world scenarios.

Findings

Achieves 17% WER estimation MAE and 0.77 PCC on WHAMR! dataset.

Attains 1.38 SI-SNR MAE and 0.95 PCC, demonstrating high accuracy.

Proves robustness across various SSL representations.

Abstract

Source separation is a crucial pre-processing step for various speech processing tasks, such as automatic speech recognition (ASR). Traditionally, the evaluation metrics for speech separation rely on the matched reference audios and corresponding transcriptions to assess audio quality and intelligibility. However, they cannot be used to evaluate real-world mixtures for which no reference exists. This paper introduces a text-free reference-free evaluation framework based on self-supervised learning (SSL) representations. The proposed framework utilize the mixture and separated tracks to predict jointly audio quality, through the Scale Invariant Signal to Noise Ratio (SI-SNR) metric, and speech intelligibility through the Word Error Rate (WER) metric. We conducted experiments on the WHAMR! dataset, which shows a WER estimation with a mean absolute error (MAE) of 17% and a Pearson correlation coefficient (PCC) of 0.77; and SI-SNR estimation with an MAE of 1.38 and PCC of 0.95. We further demonstrate the robustness of our estimator by using various SSL representations.