MAST: Multiscale Audio Spectrogram Transformers

TL;DR

MAST introduces a multiscale transformer architecture for audio classification, effectively capturing both low-level and high-level acoustic features, and extends to a self-supervised learning method that improves performance on various audio tasks.

Contribution

The paper proposes a novel multiscale architecture for audio spectrogram transformers and a new self-supervised learning approach, SS-MAST, enhancing audio classification accuracy.

Findings

MAST outperforms AST by 3.4% accuracy on the LAPE Benchmark.

SS-MAST improves over SSAST by 2.6% accuracy.

State-of-the-art results achieved on keyword spotting in Speech Commands.

Abstract

We present Multiscale Audio Spectrogram Transformer (MAST) for audio classification, which brings the concept of multiscale feature hierarchies to the Audio Spectrogram Transformer (AST). Given an input audio spectrogram, we first patchify and project it into an initial temporal resolution and embedding dimension, post which the multiple stages in MAST progressively expand the embedding dimension while reducing the temporal resolution of the input. We use a pyramid structure that allows early layers of MAST operating at a high temporal resolution but low embedding space to model simple low-level acoustic information and deeper temporally coarse layers to model high-level acoustic information with high-dimensional embeddings. We also extend our approach to present a new Self-Supervised Learning (SSL) method called SS-MAST, which calculates a symmetric contrastive loss between latent representations from a student and a teacher encoder, leveraging patch-drop, a novel audio augmentation approach that we introduce. In practice, MAST significantly outperforms AST by an average accuracy of 3.4% across 8 speech and non-speech tasks from the LAPE Benchmark, achieving state-of-the-art results on keyword spotting in Speech Commands. Additionally, our proposed SS-MAST achieves an absolute average improvement of 2.6% over the previously proposed SSAST.