Embedding-Based Intrusive Evaluation Metrics for Musical Source Separation Using MERT Representations

TL;DR

This paper proposes embedding-based intrusive evaluation metrics using MERT representations for musical source separation, demonstrating stronger correlation with perceptual quality than traditional metrics.

Contribution

It introduces MERT embedding-based metrics for MSS evaluation and shows they outperform BSS-Eval metrics in correlating with perceptual quality.

Findings

MERT-based metrics have higher correlation with perceptual ratings than BSS-Eval.

Experiments on two datasets confirm the effectiveness of embedding-based metrics.

Embedding-based metrics outperform traditional metrics across various models and stems.

Abstract

Evaluation of musical source separation (MSS) has traditionally relied on Blind Source Separation Evaluation (BSS-Eval) metrics. However, recent work suggests that BSS-Eval metrics exhibit low correlation between metrics and perceptual audio quality ratings from a listening test, which is considered the gold standard evaluation method. As an alternative approach in singing voice separation, embedding-based intrusive metrics that leverage latent representations from large self-supervised audio models such as Music undERstanding with large-scale self-supervised Training (MERT) embeddings have been introduced. In this work, we analyze the correlation of perceptual audio quality ratings with two intrusive embedding-based metrics: a mean squared error (MSE) and an intrusive variant of the Fr\'echet Audio Distance (FAD) calculated on MERT embeddings. Experiments on two independent datasets show that these metrics correlate more strongly with perceptual audio quality ratings than traditional BSS-Eval metrics across all analyzed stem and model types.