EAT: Self-Supervised Pre-Training with Efficient Audio Transformer

TL;DR

This paper introduces EAT, an efficient self-supervised audio transformer that improves representation learning in audio tasks while significantly reducing pre-training time, inspired by recent advances in image and audio SSL methods.

Contribution

EAT employs a novel bootstrap self-supervised paradigm with a new Utterance-Frame Objective and optimized masking strategies to enhance audio SSL effectiveness and efficiency.

Findings

Achieves state-of-the-art results on multiple audio benchmarks.

Reduces pre-training time by up to 15 times.

Demonstrates superior representation quality with large inverse block masks.

Abstract

Audio self-supervised learning (SSL) pre-training, which aims to learn good representations from unlabeled audio, has made remarkable progress. However, the extensive computational demands during pre-training pose a significant barrier to the potential application and optimization of audio SSL models. In this paper, inspired by the success of data2vec 2.0 in image modality and Audio-MAE in audio modality, we introduce Efficient Audio Transformer (EAT) to further improve the effectiveness and efficiency in audio SSL. The proposed EAT adopts the bootstrap self-supervised training paradigm to the audio domain. A novel Utterance-Frame Objective (UFO) is designed to enhance the modeling capability of acoustic events. Furthermore, we reveal that the masking strategy is critical in audio SSL pre-training, and superior audio representations can be obtained with large inverse block masks. Experiment results demonstrate that EAT achieves state-of-the-art (SOTA) performance on a range of audio-related tasks, including AudioSet (AS-2M, AS-20K), ESC-50, and SPC-2, along with a significant pre-training speedup up to ~15x compared to existing audio SSL models.