CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentation

TL;DR

CoLLAP introduces a novel contrastive pretraining method that extends the perception window for long-form audio and text, leveraging musical temporal structures to improve multimodal alignment and retrieval tasks.

Contribution

The paper presents a new contrastive learning architecture for long-form music-audio and text, utilizing musical temporal structures and large-scale data to enhance multimodal representation learning.

Findings

Improved retrieval accuracy on long-form music-text datasets.

Effective transfer of pretrained models to diverse music information retrieval tasks.

Demonstrated benefits of temporal structure augmentation in multimodal contrastive learning.

Abstract

Modeling temporal characteristics plays a significant role in the representation learning of audio waveform. We propose Contrastive Long-form Language-Audio Pretraining (\textbf{CoLLAP}) to significantly extend the perception window for both the input audio (up to 5 minutes) and the language descriptions (exceeding 250 words), while enabling contrastive learning across modalities and temporal dynamics. Leveraging recent Music-LLMs to generate long-form music captions for full-length songs, augmented with musical temporal structures, we collect 51.3K audio-text pairs derived from the large-scale AudioSet training dataset, where the average audio length reaches 288 seconds. We propose a novel contrastive learning architecture that fuses language representations with structured audio representations by segmenting each song into clips and extracting their embeddings. With an attention mechanism, we capture multimodal temporal correlations, allowing the model to automatically weigh and enhance the final fusion score for improved contrastive alignment. Finally, we develop two variants of the CoLLAP model with different types of backbone language models. Through comprehensive experiments on multiple long-form music-text retrieval datasets, we demonstrate consistent performance improvement in retrieval accuracy compared with baselines. We also show the pretrained CoLLAP models can be transferred to various music information retrieval tasks, with heterogeneous long-form multimodal contexts.