Unlocking Strong Supervision: A Data-Centric Study of General-Purpose Audio Pre-Training Methods

TL;DR

This paper advocates for a data-centric approach to audio pre-training, emphasizing high-quality labels and a unified tagging system to improve representation learning across diverse audio tasks.

Contribution

It introduces a new data pipeline with high-fidelity captions and a unified tag system, and systematically studies the impact of data quality and objectives on audio pre-training.

Findings

Data quality and coverage are key performance drivers.

Choice of pre-training objective influences task specialization.

High-quality source data leads to better audio representations.

Abstract

Current audio pre-training seeks to learn unified representations for broad audio understanding tasks, but it remains fragmented and is fundamentally bottlenecked by its reliance on weak, noisy, and scale-limited labels. Drawing lessons from vision's foundational pre-training blueprint, we argue that the audio field must first establish its own large-scale, strong supervision framework. We introduce a new data-centric pipeline that leverages a high-fidelity captioner to create SOTA-quality captions and the first Unified Tag System (UTS) that bridges speech, music, and environmental sounds. We then conduct a systematic comparative study of different pre-training objectives on these strong source data. Our experiments suggest that data quality and coverage are the primary drivers of performance, while the choice of objective dictates downstream task specialization.