Fine-tune the pretrained ATST model for sound event detection

TL;DR

This paper explores fine-tuning a large pretrained self-supervised audio model, ATST-Frame, for sound event detection, achieving state-of-the-art results by effectively adapting the model with both labeled and unlabeled data.

Contribution

It introduces a novel fine-tuning method for the ATST-Frame model in SED, overcoming overfitting and setting new performance benchmarks.

Findings

Achieved new SOTA PSDS1/PSDS2 scores of 0.587/0.812 on DCASE dataset.

Proposed a fine-tuning approach that utilizes both labeled and unlabeled data.

Demonstrated effective adaptation of a large pretrained model for SED tasks.

Abstract

Sound event detection (SED) often suffers from the data deficiency problem. The recent baseline system in the DCASE2023 challenge task 4 leverages the large pretrained self-supervised learning (SelfSL) models to mitigate such restriction, where the pretrained models help to produce more discriminative features for SED. However, the pretrained models are regarded as a frozen feature extractor in the challenge baseline system and most of the challenge submissions, and fine-tuning of the pretrained models has been rarely studied. In this work, we study the fine-tuning method of the pretrained models for SED. We first introduce ATST-Frame, our newly proposed SelfSL model, to the SED system. ATST-Frame was especially designed for learning frame-level representations of audio signals and obtained state-of-the-art (SOTA) performances on a series of downstream tasks. We then propose a fine-tuning method for ATST-Frame using both (in-domain) unlabelled and labelled SED data. Our experiments show that, the proposed method overcomes the overfitting problem when fine-tuning the large pretrained network, and our SED system obtains new SOTA results of 0.587/0.812 PSDS1/PSDS2 scores on the DCASE challenge task 4 dataset.