FastAST: Accelerating Audio Spectrogram Transformer via Token Merging and Cross-Model Knowledge Distillation

TL;DR

FastAST combines token merging and cross-model knowledge distillation to significantly accelerate audio spectrogram transformer models, enabling real-time audio analysis with minimal accuracy loss.

Contribution

The paper introduces FastAST, a novel framework that accelerates AST models using token merging and knowledge distillation, achieving faster inference with maintained or improved accuracy.

Findings

FastAST increases inference speed significantly.

FastAST maintains high accuracy with minimal loss.

Integration of ToMe and CMKD improves efficiency and accuracy.

Abstract

Audio classification models, particularly the Audio Spectrogram Transformer (AST), play a crucial role in efficient audio analysis. However, optimizing their efficiency without compromising accuracy remains a challenge. In this paper, we introduce FastAST, a framework that integrates Token Merging (ToMe) into the AST framework. FastAST enhances inference speed without requiring extensive retraining by merging similar tokens in audio spectrograms. Furthermore, during training, FastAST brings about significant speed improvements. The experiments indicate that FastAST can increase audio classification throughput with minimal impact on accuracy. To mitigate the accuracy impact, we integrate Cross-Model Knowledge Distillation (CMKD) into the FastAST framework. Integrating ToMe and CMKD into AST results in improved accuracy compared to AST while maintaining faster inference speeds. FastAST represents a step towards real-time, resource-efficient audio analysis.