MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning

TL;DR

MATPAC++ advances self-supervised audio representation learning by integrating Multiple Choice Learning into masked latent prediction, effectively handling ambiguity in audio content and achieving state-of-the-art results across multiple benchmarks.

Contribution

It introduces a novel enhancement to MATPAC by incorporating MCL to explicitly model prediction ambiguity, improving representation quality and downstream task performance.

Findings

Achieves state-of-the-art results on AudioSet.

Excels in downstream tasks with linear probing and fine-tuning.

Improves efficiency in domain-specific (music) training.

Abstract

Masked latent prediction has emerged as a leading paradigm in self-supervised learning (SSL), especially for general audio and music representation learning. While recent methods have demonstrated strong performance, the role of the predictor module used at the output of such SSL systems remains mainly overlooked, despite being crucial for solving the pretext task at hand. In particular, this module should be able to deal with the ambiguity inherent in audio content, especially when it is composed of multiple sound sources. This work proposes a novel enhancement: integrating Multiple Choice Learning (MCL) to explicitly model prediction ambiguity and improve representation quality. We build on top of the recently proposed MATPAC system, improving its prediction and unsupervised classification pretext tasks with MCL. We extensively evaluate our method, MATPAC++, through both linear probing across multiple downstream tasks and fine-tuning on AudioSet, employing a unified protocol that enables rigorous and fair comparisons with state-of-the-art SSL approaches. Results show that our proposal achieves state-of-the-art when fine-tuned on AudioSet and overall state-of-the-art scores on downstream tasks. Additionally, we examine domain specialisation by training exclusively on music data, where our model achieves state-of-the-art performance with significantly improved efficiency.