It's Time for Artistic Correspondence in Music and Video

TL;DR

This paper introduces a self-supervised method using Transformer networks to align music and video at an artistic level, significantly improving retrieval accuracy and enabling new applications like attribute-based music retrieval.

Contribution

It presents a novel self-supervised approach leveraging long-term temporal modeling with Transformers to improve music-video correspondence detection.

Findings

Outperforms non-temporal models by a large margin

Achieves up to 10x better retrieval accuracy

Enables attribute-based music retrieval

Abstract

We present an approach for recommending a music track for a given video, and vice versa, based on both their temporal alignment and their correspondence at an artistic level. We propose a self-supervised approach that learns this correspondence directly from data, without any need of human annotations. In order to capture the high-level concepts that are required to solve the task, we propose modeling the long-term temporal context of both the video and the music signals, using Transformer networks for each modality. Experiments show that this approach strongly outperforms alternatives that do not exploit the temporal context. The combination of our contributions improve retrieval accuracy up to 10x over prior state of the art. This strong improvement allows us to introduce a wide range of analyses and applications. For instance, we can condition music retrieval based on visually defined attributes.