Contrastive Learning of General-Purpose Audio Representations

TL;DR

COLA is a contrastive self-supervised learning method that creates versatile audio representations by distinguishing segments from the same recording versus different recordings, outperforming previous approaches across diverse audio tasks.

Contribution

This paper introduces COLA, a novel contrastive learning framework for general-purpose audio representation learning, leveraging large-scale pre-training and transfer to multiple audio classification tasks.

Findings

COLA significantly outperforms previous self-supervised audio models.

Pre-trained embeddings transfer effectively to diverse audio classification tasks.

Ablation studies identify key design choices for optimal performance.

Abstract

We introduce COLA, a self-supervised pre-training approach for learning a general-purpose representation of audio. Our approach is based on contrastive learning: it learns a representation which assigns high similarity to audio segments extracted from the same recording while assigning lower similarity to segments from different recordings. We build on top of recent advances in contrastive learning for computer vision and reinforcement learning to design a lightweight, easy-to-implement self-supervised model of audio. We pre-train embeddings on the large-scale Audioset database and transfer these representations to 9 diverse classification tasks, including speech, music, animal sounds, and acoustic scenes. We show that despite its simplicity, our method significantly outperforms previous self-supervised systems. We furthermore conduct ablation studies to identify key design choices and release a library to pre-train and fine-tune COLA models.