Multiscale Audio Spectrogram Transformer for Efficient Audio Classification

TL;DR

This paper introduces MAST, a hierarchical multiscale Transformer for audio classification that improves accuracy and efficiency by leveraging multiscale pooling and hierarchical representations, outperforming previous models on multiple datasets.

Contribution

Develops MAST, a multiscale hierarchical Transformer architecture that enhances audio classification accuracy and efficiency without external data, outperforming AST on several benchmarks.

Findings

MAST outperforms AST by 22.2%, 4.4%, and 4.7% on Kinetics-Sounds, Epic-Kitchens-100, and VGGSound.

MAST achieves slightly better accuracy than AST on AudioSet with missing data.

MAST is 5x more efficient in MACs and has 42% fewer parameters than AST.

Abstract

Audio event has a hierarchical architecture in both time and frequency and can be grouped together to construct more abstract semantic audio classes. In this work, we develop a multiscale audio spectrogram Transformer (MAST) that employs hierarchical representation learning for efficient audio classification. Specifically, MAST employs one-dimensional (and two-dimensional) pooling operators along the time (and frequency domains) in different stages, and progressively reduces the number of tokens and increases the feature dimensions. MAST significantly outperforms AST~\cite{gong2021ast} by 22.2\%, 4.4\% and 4.7\% on Kinetics-Sounds, Epic-Kitchens-100 and VGGSound in terms of the top-1 accuracy without external training data. On the downloaded AudioSet dataset, which has over 20\% missing audios, MAST also achieves slightly better accuracy than AST. In addition, MAST is 5x more efficient in terms of multiply-accumulates (MACs) with 42\% reduction in the number of parameters compared to AST. Through clustering metrics and visualizations, we demonstrate that the proposed MAST can learn semantically more separable feature representations from audio signals.