Leave-One-EquiVariant: Alleviating invariance-related information loss in contrastive music representations

TL;DR

This paper introduces LOEV, a flexible framework for contrastive music representation learning that preserves task-relevant information by selectively controlling invariances, enhancing performance on specific MIR tasks.

Contribution

The paper proposes the LOEV framework and its variant LOEV++, which enable task-adaptive invariance control and disentangled representations in contrastive music learning.

Findings

LOEV improves performance on augmentation-related MIR tasks.

LOEV++ achieves disentangled, attribute-specific representations.

The approach alleviates information loss due to invariance constraints.

Abstract

Contrastive learning has proven effective in self-supervised musical representation learning, particularly for Music Information Retrieval (MIR) tasks. However, reliance on augmentation chains for contrastive view generation and the resulting learnt invariances pose challenges when different downstream tasks require sensitivity to certain musical attributes. To address this, we propose the Leave One EquiVariant (LOEV) framework, which introduces a flexible, task-adaptive approach compared to previous work by selectively preserving information about specific augmentations, allowing the model to maintain task-relevant equivariances. We demonstrate that LOEV alleviates information loss related to learned invariances, improving performance on augmentation related tasks and retrieval without sacrificing general representation quality. Furthermore, we introduce a variant of LOEV, LOEV++, which builds a disentangled latent space by design in a self-supervised manner, and enables targeted retrieval based on augmentation related attributes.