SLAP: Siamese Language-Audio Pretraining Without Negative Samples for Music Understanding

TL;DR

SLAP introduces a scalable, negative-sample-free multimodal pretraining framework for music understanding, improving retrieval, classification, and robustness while reducing memory requirements and enabling large-scale training on limited hardware.

Contribution

It adapts BYOL for multimodal audio-text pretraining, addressing modality gap and scalability issues in music understanding models.

Findings

Outperforms CLAP on text-music retrieval and zero-shot classification.

Achieves competitive MIR task performance with larger or supervised models.

Reduces modality gap and improves robustness to batch size variations.

Abstract

Joint embedding spaces have significantly advanced music understanding and generation by linking text and audio through multimodal contrastive learning. However, these approaches face large memory requirement limitations due to relying on large batch sizes to effectively utilize negative samples. Further, multimodal joint embedding spaces suffer from a modality gap wherein embeddings from different modalities lie in different manifolds of the embedding space. To address these challenges, we propose Siamese Language-Audio Pretraining (SLAP), a novel multimodal pretraining framework that allows learning powerful representations without negative samples. SLAP adapts the Bootstrap Your Own Latent (BYOL) paradigm for multimodal audio-text training, promoting scalability in training multimodal embedding spaces. We illustrate the ability of our model to learn meaningful relationships between music and text -- specifically, we show that SLAP outperforms CLAP on tasks such as text-music retrieval and zero-shot classification. We also observe competitive downstream performance on several MIR tasks, including with larger or supervised models (genre and instrument classification, auto-tagging). Additionally, our approach has attractive properties, such as a quantifiably reduced modality gap and improved robustness to batch size variations on retrieval performance. Finally, its novel formulation unlocks large-scale training on a single GPU through gradient accumulation.