Unified Pretraining Target Based Video-music Retrieval With Music Rhythm And Video Optical Flow Information

TL;DR

This paper introduces a unified pretraining approach for video-music retrieval that incorporates music rhythm and optical flow to improve matching accuracy by preserving temporal information.

Contribution

It proposes a novel unified target set for pretraining and utilizes clip-level embeddings with rhythm and optical flow, enhancing temporal correlation modeling.

Findings

Achieves superior retrieval performance over state-of-the-art methods.

Effectively preserves temporal information in video-music matching.

Utilizes music rhythm and optical flow for improved cross-modal alignment.

Abstract

Background music (BGM) can enhance the video's emotion. However, selecting an appropriate BGM often requires domain knowledge. This has led to the development of video-music retrieval techniques. Most existing approaches utilize pretrained video/music feature extractors trained with different target sets to obtain average video/music-level embeddings. The drawbacks are two-fold. One is that different target sets for video/music pretraining may cause the generated embeddings difficult to match. The second is that the underlying temporal correlation between video and music is ignored. In this paper, our proposed approach leverages a unified target set to perform video/music pretraining and produces clip-level embeddings to preserve temporal information. The downstream cross-modal matching is based on the clip-level features with embedded music rhythm and optical flow information. Experiments demonstrate that our proposed method can achieve superior performance over the state-of-the-art methods by a significant margin.