Audio Mamba: Pretrained Audio State Space Model For Audio Tagging

TL;DR

Audio Mamba introduces a self-attention-free, state space model for audio tagging that efficiently captures long-range dependencies, achieving comparable results to transformer models with fewer parameters.

Contribution

It proposes a novel self-attention-free approach using state space models for audio tagging, addressing scalability issues of transformer-based models.

Findings

Achieves state-of-the-art performance with fewer parameters

Demonstrates parameter efficiency on multiple datasets

Captures long-range dependencies effectively

Abstract

Audio tagging is an important task of mapping audio samples to their corresponding categories. Recently endeavours that exploit transformer models in this field have achieved great success. However, the quadratic self-attention cost limits the scaling of audio transformer models and further constrains the development of more universal audio models. In this paper, we attempt to solve this problem by proposing Audio Mamba, a self-attention-free approach that captures long audio spectrogram dependency with state space models. Our experimental results on two audio-tagging datasets demonstrate the parameter efficiency of Audio Mamba, it achieves comparable results to SOTA audio spectrogram transformers with one third parameters.