Compositional Audio Representation Learning

TL;DR

This paper introduces novel supervised and unsupervised methods for learning source-centric, disentangled audio representations to improve interpretability and flexibility in machine listening tasks.

Contribution

It proposes two new approaches for source-centric audio representation learning, demonstrating the benefits of supervision and feature reconstruction over baselines.

Findings

Supervised models outperform unsupervised ones in learning source-centric representations.

Reconstructing audio features is more effective than spectrogram reconstruction in unsupervised learning.

Source-centric representations enhance interpretability and decoding flexibility in machine listening.

Abstract

Human auditory perception is compositional in nature -- we identify auditory streams from auditory scenes with multiple sound events. However, such auditory scenes are typically represented using clip-level representations that do not disentangle the constituent sound sources. In this work, we learn source-centric audio representations where each sound source is represented using a distinct, disentangled source embedding in the audio representation. We propose two novel approaches to learning source-centric audio representations: a supervised model guided by classification and an unsupervised model guided by feature reconstruction, both of which outperform the baselines. We thoroughly evaluate the design choices of both approaches using an audio classification task. We find that supervision is beneficial to learn source-centric representations, and that reconstructing audio features is more useful than reconstructing spectrograms to learn unsupervised source-centric representations. Leveraging source-centric models can help unlock the potential of greater interpretability and more flexible decoding in machine listening.