Metric Analysis for Spatial Semantic Segmentation of Sound Scenes

TL;DR

This paper introduces CASA-SDR, a new metric for spatial sound scene analysis that better separates source separation quality from classification errors, improving evaluation accuracy in complex audio scenarios.

Contribution

The paper proposes CASA-SDR, a permutation-invariant metric that enhances the evaluation of spatial semantic sound scene analysis by focusing on separation quality independent of classification errors.

Findings

CASA-SDR reduces penalization of label swaps compared to CA-SDR.

CASA-SDR provides more interpretable separation assessments.

Analysis shows CA-SDR can conflate separation and classification errors.

Abstract

Spatial semantic segmentation of sound scenes (S5) consists of jointly performing audio source separation and sound event classification from a multichannel audio mixture. Evaluating S5 systems with separation and classification metrics individually makes system comparison difficult, whereas existing joint metrics, such as the class-aware signal-to-distortion ratio (CA-SDR), can conflate separation and labeling errors. In particular, CA-SDR relies on predicted class labels for source matching, which may obscure label swaps or misclassifications when the underlying source estimates remain perceptually correct. In this work, we introduce the class and source-aware signal-to-distortion ratio (CASA-SDR), a new metric that performs permutation-invariant source matching before computing classification errors, thereby shifting from a classification-focused approach to a separation-focused approach. We first analyze CA-SDR in controlled scenarios with oracle separation and synthetic classification errors, as well as under controlled cross-contamination between sources, and compare its behavior to that of the classical SDR and CASA-SDR. We also study the impact of classification errors on the metrics by introducing error-based and source-based aggregation strategies. Finally, we compare CA-SDR and CASA-SDR on systems submitted to Task 4 of the DCASE 2025 challenge, highlighting the cases where CA-SDR over-penalizes label swaps or poorly separated sources, while CASA-SDR provides a more interpretable separation-centric assessment of S5 performance.