A benchmark of state-of-the-art sound event detection systems evaluated on synthetic soundscapes

TL;DR

This paper benchmarks state-of-the-art sound event detection systems on synthetic soundscapes, analyzing their robustness to noise, temporal localization, and non-target events, revealing strengths and vulnerabilities of different approaches.

Contribution

It provides a comprehensive evaluation of top sound event detection systems on synthetic soundscapes, highlighting their robustness and weaknesses under various conditions.

Findings

Coarse segmentation systems are more robust to noise and temporal localization.

Data augmentation improves robustness to event timing.

Fine segmentation systems tend to falsely detect short events with non-target sounds.

Abstract

This paper proposes a benchmark of submissions to Detection and Classification Acoustic Scene and Events 2021 Challenge (DCASE) Task 4 representing a sampling of the state-of-the-art in Sound Event Detection task. The submissions are evaluated according to the two polyphonic sound detection score scenarios proposed for the DCASE 2021 Challenge Task 4, which allow to make an analysis on whether submissions are designed to perform fine-grained temporal segmentation, coarse-grained temporal segmentation, or have been designed to be polyvalent on the scenarios proposed. We study the solutions proposed by participants to analyze their robustness to varying level target to non-target signal-to-noise ratio and to temporal localization of target sound events. A last experiment is proposed in order to study the impact of non-target events on systems outputs. Results show that systems adapted to provide coarse segmentation outputs are more robust to different target to non-target signal-to-noise ratio and, with the help of specific data augmentation methods, they are more robust to time localization of the original event. Results of the last experiment display that systems tend to spuriously predict short events when non-target events are present. This is particularly true for systems that are tailored to have a fine segmentation.