AudioMosaic: Contrastive Masked Audio Representation Learning

TL;DR

AudioMosaic introduces a contrastive self-supervised learning method for audio that uses structured masking to efficiently learn transferable, discriminative representations, achieving state-of-the-art results across multiple audio benchmarks.

Contribution

It presents a novel contrastive learning approach with structured masking for audio, enabling efficient large-batch training and improved generalization over generative methods.

Findings

Achieves state-of-the-art results on standard audio benchmarks.

Learns more discriminative utterance-level representations.

Improves performance on audio-language tasks when integrated into models.

Abstract

Audio self-supervised learning (SSL) aims to learn general-purpose representations from large-scale unlabeled audio data. While recent advances have been driven mainly by generative reconstruction objectives, contrastive approaches remain less explored, partly due to the difficulty of designing effective audio augmentations and the large batch sizes required for contrastive pre-training. We introduce \textbf{AudioMosaic}, a contrastive learning-based audio encoder for general audio understanding. During pre-training, AudioMosaic constructs positive pairs by applying structured time-frequency masking to spectrogram patches, which reduces memory usage and enables efficient large-batch training. Compared with generative approaches, the AudioMosaic encoder learns more discriminative utterance-level representations that demonstrate strong transferability across datasets, domains, and acoustic conditions. Extensive experiments show that AudioMosaic achieves state-of-the-art performance on several standard audio benchmarks under both linear probing and fine-tuning. We further show that integrating the pretrained AudioMosaic encoder into audio-language models improves performance on audio-language tasks. The code is publicly available in our \href{https://github.com/HanxunH/AudioMosaic}{GitHub repository}.