Revisiting SSL for sound event detection: complementary fusion and adaptive post-processing

TL;DR

This paper evaluates and combines various self-supervised learning models for sound event detection, proposing fusion strategies and adaptive post-processing to improve detection accuracy and robustness.

Contribution

It introduces a systematic framework for fusing heterogeneous SSL representations and an adaptive post-processing method, enhancing SED performance.

Findings

Dual-modal fusion yields complementary performance gains.

CRNN+BEATs achieves top results among individual SSL models.

Adaptive post-processing improves PSDS1 by up to 4%.

Abstract

Self-supervised learning (SSL) models offer powerful representations for sound event detection (SED), yet their synergistic potential remains underexplored. This study systematically evaluates state-of-the-art SSL models to guide optimal model selection and integration for SED. We propose a framework that combines heterogeneous SSL representations (e.g., BEATs, HuBERT, WavLM) through three fusion strategies: individual SSL embedding integration, dual-modal fusion, and full aggregation. Experiments on the DCASE 2023 Task 4 Challenge reveal that dual-modal fusion (e.g., CRNN+BEATs+WavLM) achieves complementary performance gains, while CRNN+BEATs alone delivers the best results among individual SSL models. We further introduce normalized sound event bounding boxes (nSEBBs), an adaptive post-processing method that dynamically adjusts event boundary predictions, improving PSDS1 by up to 4% for standalone SSL models. These findings highlight the compatibility and complementarity of SSL architectures, providing guidance for task-specific fusion and robust SED system design.