SLICER: Learning universal audio representations using low-resource self-supervised pre-training

TL;DR

SLICER introduces a novel self-supervised learning method combining clustering and contrasting paradigms to pre-train audio encoders, enabling effective low-resource audio and speech classification.

Contribution

It proposes SLICER, a new SSL approach that integrates instance and cluster-level contrastive learning with a novel augmentation, achieving state-of-the-art results on audio benchmarks.

Findings

Outperforms prior methods on the LAPE Benchmark

Requires significantly less unlabeled data for pre-training

Introduces a new augmentation technique, k-mix

Abstract

We present a new Self-Supervised Learning (SSL) approach to pre-train encoders on unlabeled audio data that reduces the need for large amounts of labeled data for audio and speech classification. Our primary aim is to learn audio representations that can generalize across a large variety of speech and non-speech tasks in a low-resource un-labeled audio pre-training setting. Inspired by the recent success of clustering and contrasting learning paradigms for SSL-based speech representation learning, we propose SLICER (Symmetrical Learning of Instance and Cluster-level Efficient Representations), which brings together the best of both clustering and contrasting learning paradigms. We use a symmetric loss between latent representations from student and teacher encoders and simultaneously solve instance and cluster-level contrastive learning tasks. We obtain cluster representations online by just projecting the input spectrogram into an output subspace with dimensions equal to the number of clusters. In addition, we propose a novel mel-spectrogram augmentation procedure, k-mix, based on mixup, which does not require labels and aids unsupervised representation learning for audio. Overall, SLICER achieves state-of-the-art results on the LAPE Benchmark \cite{9868132}, significantly outperforming DeLoRes-M and other prior approaches, which are pre-trained on $10\times$ larger of unsupervised data. We will make all our codes available on GitHub.