AudioMAE++: learning better masked audio representations with SwiGLU FFNs

TL;DR

AudioMAE++ introduces architectural improvements to masked autoencoders for audio, utilizing macaron-style transformer blocks with gated linear units, leading to superior performance and scalability on diverse audio tasks.

Contribution

The paper presents AudioMAE++, a novel masked autoencoder architecture with macaron-style transformer blocks and gated linear units, enhancing audio representation learning.

Findings

Outperforms existing MAE approaches on 10 downstream tasks

Demonstrates excellent scaling with up to 4x more parameters

Achieves state-of-the-art results on audio classification and speech benchmarks

Abstract

Masked Autoencoders (MAEs) trained on audio spectrogram patches have emerged as a prominent approach for learning self-supervised audio representations. While several recent papers have evaluated key aspects of training MAEs on audio data, the majority of these approaches still leverage vanilla transformer building blocks, whereas the transformer community has seen steady integration of newer architectural advancements. In this work, we propose AudioMAE++, a revamped audio masked autoencoder with two such enhancements, namely macaron-style transformer blocks with gated linear units. When pretrained on the AudioSet dataset, the proposed AudioMAE++ models outperform existing MAE based approaches on 10 diverse downstream tasks, demonstrating excellent performance on audio classification and speech-based benchmarks. The proposed AudioMAE++ models also demonstrate excellent scaling characteristics, outperforming directly comparable standard MAE baselines with up to 4x more parameters.