Scaling up masked audio encoder learning for general audio classification

TL;DR

This paper introduces Dasheng, a large-scale self-supervised audio encoder trained on diverse data, significantly improving general audio classification across speech, music, and environmental sounds.

Contribution

The paper presents Dasheng, a scalable SSL audio encoder trained on 1.2 billion parameters and 272,356 hours of data, achieving state-of-the-art results on multiple audio benchmarks.

Findings

Dasheng outperforms previous models on HEAR benchmark tasks.

It demonstrates strong generalization across speech, music, and environmental sounds.

Nearest-neighbor analysis shows rich, multi-domain audio representations.

Abstract

Despite progress in audio classification, a generalization gap remains between speech and other sound domains, such as environmental sounds and music. Models trained for speech tasks often fail to perform well on environmental or musical audio tasks, and vice versa. While self-supervised (SSL) audio representations offer an alternative, there has been limited exploration of scaling both model and dataset sizes for SSL-based general audio classification. We introduce Dasheng, a simple SSL audio encoder, based on the efficient masked autoencoder framework. Trained with 1.2 billion parameters on 272,356 hours of diverse audio, Dasheng obtains significant performance gains on the HEAR benchmark. It outperforms previous works on CREMA-D, LibriCount, Speech Commands, VoxLingua, and competes well in music and environment classification. Dasheng features inherently contain rich speech, music, and environmental information, as shown in nearest-neighbor classification experiments. Code is available https://github.com/richermans/dasheng/.