Self-Supervised Learning from Automatically Separated Sound Scenes

TL;DR

This paper introduces a self-supervised learning approach that leverages automatically separated sound scenes to improve audio representations, achieving competitive results on AudioSet without labeled data.

Contribution

It demonstrates that unsupervised sound separation can enhance self-supervised contrastive learning, and that perfect separation is not necessary for effective representation learning.

Findings

Automatically separated views improve contrastive learning.

Optimal separation quality is not critical for success.

The proposed method rivals state-of-the-art on AudioSet.

Abstract

Real-world sound scenes consist of time-varying collections of sound sources, each generating characteristic sound events that are mixed together in audio recordings. The association of these constituent sound events with their mixture and each other is semantically constrained: the sound scene contains the union of source classes and not all classes naturally co-occur. With this motivation, this paper explores the use of unsupervised automatic sound separation to decompose unlabeled sound scenes into multiple semantically-linked views for use in self-supervised contrastive learning. We find that learning to associate input mixtures with their automatically separated outputs yields stronger representations than past approaches that use the mixtures alone. Further, we discover that optimal source separation is not required for successful contrastive learning by demonstrating that a range of separation system convergence states all lead to useful and often complementary example transformations. Our best system incorporates these unsupervised separation models into a single augmentation front-end and jointly optimizes similarity maximization and coincidence prediction objectives across the views. The result is an unsupervised audio representation that rivals state-of-the-art alternatives on the established shallow AudioSet classification benchmark.