Deep Music Retrieval for Fine-Grained Videos by Exploiting Cross-Modal-Encoded Voice-Overs

TL;DR

This paper introduces a novel cross-modal retrieval framework leveraging voice-overs and attention mechanisms to improve background music retrieval for fine-grained short videos, outperforming existing methods.

Contribution

The paper proposes a new framework using self-attention, cross-modal attention, and dynamic fusion for video-music retrieval, along with creating two new datasets.

Findings

Our method achieves superior retrieval performance over state-of-the-art approaches.

The framework effectively models intra- and inter-modal relationships for fine-grained content.

Experimental results demonstrate significant improvements on the new datasets.

Abstract

Recently, the witness of the rapidly growing popularity of short videos on different Internet platforms has intensified the need for a background music (BGM) retrieval system. However, existing video-music retrieval methods only based on the visual modality cannot show promising performance regarding videos with fine-grained virtual contents. In this paper, we also investigate the widely added voice-overs in short videos and propose a novel framework to retrieve BGM for fine-grained short videos. In our framework, we use the self-attention (SA) and the cross-modal attention (CMA) modules to explore the intra- and the inter-relationships of different modalities respectively. For balancing the modalities, we dynamically assign different weights to the modal features via a fusion gate. For paring the query and the BGM embeddings, we introduce a triplet pseudo-label loss to constrain the semantics of the modal embeddings. As there are no existing virtual-content video-BGM retrieval datasets, we build and release two virtual-content video datasets HoK400 and CFM400. Experimental results show that our method achieves superior performance and outperforms other state-of-the-art methods with large margins.