Masked Autoencoders that Listen

TL;DR

This paper introduces Audio-MAE, a self-supervised learning method for audio spectrograms based on masked autoencoders, achieving state-of-the-art results on multiple audio classification tasks.

Contribution

It extends image-based Masked Autoencoders to audio spectrograms, incorporating local window attention and demonstrating superior performance without external supervision.

Findings

Sets new state-of-the-art on six audio and speech tasks

Outperforms models with external supervised pre-training

Effective use of local window attention in decoder

Abstract

This paper studies a simple extension of image-based Masked Autoencoders (MAE) to self-supervised representation learning from audio spectrograms. Following the Transformer encoder-decoder design in MAE, our Audio-MAE first encodes audio spectrogram patches with a high masking ratio, feeding only the non-masked tokens through encoder layers. The decoder then re-orders and decodes the encoded context padded with mask tokens, in order to reconstruct the input spectrogram. We find it beneficial to incorporate local window attention in the decoder, as audio spectrograms are highly correlated in local time and frequency bands. We then fine-tune the encoder with a lower masking ratio on target datasets. Empirically, Audio-MAE sets new state-of-the-art performance on six audio and speech classification tasks, outperforming other recent models that use external supervised pre-training. The code and models will be at https://github.com/facebookresearch/AudioMAE.