DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable Learners

TL;DR

This paper introduces DASS, a distilled audio state space model that outperforms Transformers on AudioSet and demonstrates superior scalability to long audio durations, addressing previous limitations of SSMs in short and long audio tasks.

Contribution

The paper presents a novel knowledge distillation approach for audio SSMs, achieving state-of-the-art results and introducing a new long-duration audio retrieval test.

Findings

DASS outperforms Transformers on AudioSet with an mAP of 48.9.

DASS effectively retrieves sound events in recordings up to 2.5 hours long.

Audio SSMs show superior duration scalability compared to Transformers.

Abstract

State-space models (SSMs) have emerged as an alternative to Transformers for audio modeling due to their high computational efficiency with long inputs. While recent efforts on Audio SSMs have reported encouraging results, two main limitations remain: First, in 10-second short audio tagging tasks, Audio SSMs still underperform compared to Transformer-based models such as Audio Spectrogram Transformer (AST). Second, although Audio SSMs theoretically support long audio inputs, their actual performance with long audio has not been thoroughly evaluated. To address these limitations, in this paper, 1) We applied knowledge distillation in audio space model training, resulting in a model called Knowledge Distilled Audio SSM (DASS). To the best of our knowledge, it is the first SSM that outperforms the Transformers on AudioSet and achieves an mAP of 48.9; and 2) We designed a new test called Audio Needle In A Haystack (Audio NIAH). We find that DASS, trained with only 10-second audio clips, can retrieve sound events in audio recordings up to 2.5 hours long, while the AST model fails when the input is just 50 seconds, demonstrating SSMs are indeed more duration scalable. Code: https://github.com/Saurabhbhati/DASS, https://huggingface.co/saurabhati/DASS_small_AudioSet_48.9