SAM Audio Judge: A Unified Multimodal Framework for Perceptual Evaluation of Audio Separation

TL;DR

SAM Audio Judge introduces a multimodal, reference-free evaluation metric for audio separation that aligns closely with human perception across multiple domains and evaluation dimensions, enabling scalable, automated assessment.

Contribution

The paper presents SAM Audio Judge, a novel multimodal, reference-free metric for audio separation evaluation that supports multiple domains and evaluation aspects, improving scalability and alignment with human judgment.

Findings

High correlation with human perception in audio evaluation

Supports multiple audio domains and prompt types

Enables scalable, automated evaluation processes

Abstract

The performance evaluation remains a complex challenge in audio separation, and existing evaluation metrics are often misaligned with human perception, course-grained, relying on ground truth signals. On the other hand, subjective listening tests remain the gold standard for real-world evaluation, but they are expensive, time-consuming, and difficult to scale. This paper addresses the growing need for automated systems capable of evaluating audio separation without human intervention. The proposed evaluation metric, SAM Audio Judge (SAJ), is a multimodal fine-grained reference-free objective metric, which shows highly alignment with human perceptions. SAJ supports three audio domains (speech, music and general sound events) and three prompt inputs (text, visual and span), covering four different dimensions of evaluation (recall, percision, faithfulness, and overall). SAM Audio Judge also shows potential applications in data filtering, pseudo-labeling large datasets and reranking in audio separation models. We release our code and pre-trained models at: https://github.com/facebookresearch/sam-audio.