Self-Supervised Learning of Audio Representations from Permutations with Differentiable Ranking

TL;DR

This paper introduces a novel self-supervised learning method for audio representations by training models to reorder shuffled spectrogram parts using differentiable ranking, leading to improved downstream audio classification tasks.

Contribution

It presents a new permutation-based pretext task for audio self-supervised learning, overcoming integration challenges with differentiable ranking, and demonstrates its effectiveness in audio classification.

Findings

Improved instrument classification accuracy.

Enhanced pitch estimation performance.

Learning from all permutations yields better representations.

Abstract

Self-supervised pre-training using so-called "pretext" tasks has recently shown impressive performance across a wide range of modalities. In this work, we advance self-supervised learning from permutations, by pre-training a model to reorder shuffled parts of the spectrogram of an audio signal, to improve downstream classification performance. We make two main contributions. First, we overcome the main challenges of integrating permutation inversions into an end-to-end training scheme, using recent advances in differentiable ranking. This was heretofore sidestepped by casting the reordering task as classification, fundamentally reducing the space of permutations that can be exploited. Our experiments validate that learning from all possible permutations improves the quality of the pre-trained representations over using a limited, fixed set. Second, we show that inverting permutations is a meaningful pretext task for learning audio representations in an unsupervised fashion. In particular, we improve instrument classification and pitch estimation of musical notes by reordering spectrogram patches in the time-frequency space.