DECAR: Deep Clustering for learning general-purpose Audio Representations

TL;DR

DECAR is a self-supervised audio pre-training method that uses clustering to learn versatile audio representations, which are effective across diverse audio classification tasks.

Contribution

It introduces a clustering-based self-supervised pre-training scheme for general-purpose audio representations, leveraging offline clustering and transfer learning.

Findings

Effective across multiple audio classification tasks

Improves upon previous self-supervised methods

Code and models are publicly available

Abstract

We introduce DECAR, a self-supervised pre-training approach for learning general-purpose audio representations. Our system is based on clustering: it utilizes an offline clustering step to provide target labels that act as pseudo-labels for solving a prediction task. We develop on top of recent advances in self-supervised learning for computer vision and design a lightweight, easy-to-use self-supervised pre-training scheme. We pre-train DECAR embeddings on a balanced subset of the large-scale Audioset dataset and transfer those representations to 9 downstream classification tasks, including speech, music, animal sounds, and acoustic scenes. Furthermore, we conduct ablation studies identifying key design choices and also make all our code and pre-trained models publicly available.