A Framework for the Robust Evaluation of Sound Event Detection

TL;DR

This paper introduces a new, robust framework for evaluating polyphonic sound event detection systems, addressing limitations of traditional metrics and providing more comprehensive performance insights.

Contribution

It proposes a novel evaluation framework using polyphonic ROC curves and a unified score (PSDS), improving robustness and comparability over existing methods.

Findings

Demonstrated improved evaluation consistency on DCASE 2019 data

Provided better insights into data biases and class stability

Enabled application-specific tuning of evaluation metrics

Abstract

This work defines a new framework for performance evaluation of polyphonic sound event detection (SED) systems, which overcomes the limitations of the conventional collar-based event decisions, event F-scores and event error rates. The proposed framework introduces a definition of event detection that is more robust against labelling subjectivity. It also resorts to polyphonic receiver operating characteristic (ROC) curves to deliver more global insight into system performance than F1-scores, and proposes a reduction of these curves into a single polyphonic sound detection score (PSDS), which allows system comparison independently from operating points (OPs). The presented method also delivers better insight into data biases and classification stability across sound classes. Furthermore, it can be tuned to varying applications in order to match a variety of user experience requirements. The benefits of the proposed approach are demonstrated by re-evaluating the baseline and two of the top-performing systems from DCASE 2019 Task 4.