Representation-Regularized Convolutional Audio Transformer for Audio Understanding

TL;DR

This paper introduces the Convolutional Audio Transformer (CAT), a hierarchical, efficient SSL framework that captures multi-resolution audio features and aligns representations with pre-trained encoders, significantly improving audio understanding performance.

Contribution

The paper proposes CAT, a novel hierarchical audio transformer with a multi-resolution block and a representation regularization objective, enhancing efficiency and modeling of complex audio signals.

Findings

Outperforms baselines on audio benchmarks

Achieves 5x faster convergence on AudioSet 20k

Effectively models diverse temporal and spectral structures

Abstract

Bootstrap-based Self-Supervised Learning (SSL) has achieved remarkable progress in audio understanding. However, existing methods typically operate at a single level of granularity, limiting their ability to model the diverse temporal and spectral structures inherent in complex audio signals. Furthermore, bootstrapping representations from scratch is computationally expensive, often requiring extensive training to converge. In this work, we propose the Convolutional Audio Transformer (CAT), a unified framework designed to address these challenges. First, to capture hierarchical audio features, CAT incorporates a Multi-resolution Block that aggregates information across varying granularities. Second, to enhance training efficiency, we introduce a Representation Regularization objective. Drawing inspiration from generative modeling, this auxiliary task guides the student model by aligning its predictions with high-quality semantic representations from frozen, pre-trained external encoders. Experimental results demonstrate that CAT significantly outperforms baselines on audio understanding benchmarks. Notably, it achieves competitive performance on the AudioSet 20k dataset with 5 times faster convergence than existing methods. Codes and checkpoints will be released soon at https://github.com/realzhouchushu/CAT.