Effective Pre-Training of Audio Transformers for Sound Event Detection

TL;DR

This paper introduces a comprehensive pre-training pipeline for audio transformers that significantly enhances sound event detection performance, utilizing advanced data augmentation, balanced sampling, and ensemble distillation techniques.

Contribution

It presents a novel pre-training routine specifically designed for audio spectrogram transformers, improving downstream sound event detection accuracy.

Findings

Substantial performance gains on AudioSet frame-level predictions.

Effective pre-training pipeline validated on multiple transformer models.

Public release of high-performance checkpoints for sound event detection.

Abstract

We propose a pre-training pipeline for audio spectrogram transformers for frame-level sound event detection tasks. On top of common pre-training steps, we add a meticulously designed training routine on AudioSet frame-level annotations. This includes a balanced sampler, aggressive data augmentation, and ensemble knowledge distillation. For five transformers, we obtain a substantial performance improvement over previously available checkpoints both on AudioSet frame-level predictions and on frame-level sound event detection downstream tasks, confirming our pipeline's effectiveness. We publish the resulting checkpoints that researchers can directly fine-tune to build high-performance models for sound event detection tasks.