Multi-Source Contrastive Learning from Musical Audio

TL;DR

This paper introduces a novel contrastive learning approach for musical audio that associates song excerpts with specific sources like vocals or instruments, improving downstream task performance and convergence speed.

Contribution

It proposes a new source association strategy in contrastive learning for music, including a modified loss function and source extraction, enhancing representation quality.

Findings

Competitive results in music auto-tagging, instrument, and genre classification

Faster network convergence compared to existing methods

Pre-training can be directed towards specific musical features

Abstract

Contrastive learning constitutes an emerging branch of self-supervised learning that leverages large amounts of unlabeled data, by learning a latent space, where pairs of different views of the same sample are associated. In this paper, we propose musical source association as a pair generation strategy in the context of contrastive music representation learning. To this end, we modify COLA, a widely used contrastive learning audio framework, to learn to associate a song excerpt with a stochastically selected and automatically extracted vocal or instrumental source. We further introduce a novel modification to the contrastive loss to incorporate information about the existence or absence of specific sources. Our experimental evaluation in three different downstream tasks (music auto-tagging, instrument classification and music genre classification) using the publicly available Magna-Tag-A-Tune (MTAT) as a source dataset yields competitive results to existing literature methods, as well as faster network convergence. The results also show that this pre-training method can be steered towards specific features, according to the selected musical source, while also being dependent on the quality of the separated sources.