Music Grounding by Short Video

TL;DR

This paper introduces the new task of Music Grounding by Short Video (MGSV), aiming to localize suitable music segments for short videos, and presents a large benchmark dataset and a unified deep learning baseline method.

Contribution

The paper proposes MGSV as a novel task, creates the MGSV-EC benchmark dataset, and develops MaDe, an end-to-end model for music matching and localization.

Findings

MGSV is a challenging task as shown by extensive experiments.

MaDe achieves strong baseline performance on MGSV-EC.

The dataset contains 53k videos and 35k music moments.

Abstract

Adding proper background music helps complete a short video to be shared. Previous work tackles the task by video-to-music retrieval (V2MR), aiming to find the most suitable music track from a collection to match the content of a given query video. In practice, however, music tracks are typically much longer than the query video, necessitating (manual) trimming of the retrieved music to a shorter segment that matches the video duration. In order to bridge the gap between the practical need for music moment localization and V2MR, we propose a new task termed Music Grounding by Short Video (MGSV). To tackle the new task, we introduce a new benchmark, MGSV-EC, which comprises a diverse set of 53k short videos associated with 35k different music moments from 4k unique music tracks. Furthermore, we develop a new baseline method, MaDe, which performs both video-to-music matching and music moment detection within a unified end-to-end deep network. Extensive experiments on MGSV-EC not only highlight the challenging nature of MGSV but also set MaDe as a strong baseline.