Towards Reliable Objective Evaluation Metrics for Generative Singing Voice Separation Models

TL;DR

This paper investigates the limitations of traditional evaluation metrics for generative singing voice separation models and proposes alternative metrics that better correlate with human perceptual quality.

Contribution

The study analyzes correlations between objective audio quality metrics and human opinion scores, identifying more reliable metrics for evaluating generative singing voice separation models.

Findings

Intrusive embedding-based metrics correlate better with human scores than traditional metrics.

MSE on Music2Latent embeddings yields highest correlation for discriminative models.

Multi-resolution STFT loss and MSE on MERT-L12 embeddings are most effective for generative models.

Abstract

Traditional Blind Source Separation Evaluation (BSS-Eval) metrics were originally designed to evaluate linear audio source separation models based on methods such as time-frequency masking. However, recent generative models may introduce nonlinear relationships between the separated and reference signals, limiting the reliability of these metrics for objective evaluation. To address this issue, we conduct a Degradation Category Rating listening test and analyze correlations between the obtained degradation mean opinion scores (DMOS) and a set of objective audio quality metrics for the task of singing voice separation. We evaluate three state-of-the-art discriminative models and two new competitive generative models. For both discriminative and generative models, intrusive embedding-based metrics show higher correlations with DMOS than conventional intrusive metrics such as BSS-Eval. For discriminative models, the highest correlation is achieved by the MSE computed on Music2Latent embeddings. When it comes to the evaluation of generative models, the strongest correlations are evident for the multi-resolution STFT loss and the MSE calculated on MERT-L12 embeddings, with the latter also providing the most balanced correlation across both model types. Our results highlight the limitations of BSS-Eval metrics for evaluating generative singing voice separation models and emphasize the need for careful selection and validation of alternative evaluation metrics for the task of singing voice separation.