Semi-Supervised Contrastive Learning for Controllable Video-to-Music Retrieval

TL;DR

This paper introduces a semi-supervised contrastive learning framework that improves video-to-music retrieval by combining self-supervised and label-supervised objectives, allowing controllable and effective cross-modal retrieval.

Contribution

It presents a novel joint embedding approach that integrates self-supervised and supervised contrastive learning for cross-modal video and music retrieval.

Findings

Effective retrieval performance demonstrated on multiple tasks.

Framework allows adjustable focus on self-supervised vs. label information.

Generalizable to various music annotations like emotion and instrument.

Abstract

Content creators often use music to enhance their videos, from soundtracks in movies to background music in video blogs and social media content. However, identifying the best music for a video can be a difficult and time-consuming task. To address this challenge, we propose a novel framework for automatically retrieving a matching music clip for a given video, and vice versa. Our approach leverages annotated music labels, as well as the inherent artistic correspondence between visual and music elements. Distinct from previous cross-modal music retrieval works, our method combines both self-supervised and supervised training objectives. We use self-supervised and label-supervised contrastive learning to train a joint embedding space between music and video. We show the effectiveness of our approach by using music genre labels for the supervised training component, and our framework can be generalized to other music annotations (e.g., emotion, instrument, etc.). Furthermore, our method enables fine-grained control over how much the retrieval process focuses on self-supervised vs. label information at inference time. We evaluate the learned embeddings through a variety of video-to-music and music-to-video retrieval tasks. Our experiments show that the proposed approach successfully combines self-supervised and supervised objectives and is effective for controllable music-video retrieval.