Pop Music Highlighter: Marking the Emotion Keypoints

TL;DR

This paper introduces a new attention-based neural network architecture for extracting emotionally significant highlights from pop songs, demonstrating improved performance over traditional methods and providing open-source code for reproducibility.

Contribution

It proposes a non-recurrent attention-based model with fusion variants for music highlight extraction, advancing previous emotion-based approaches and extensive comparative evaluation.

Findings

The new model outperforms heuristic and structural methods.

Early-fusion attention variant yields better results.

Proposed methods are effective in identifying song highlights.

Abstract

The goal of music highlight extraction is to get a short consecutive segment of a piece of music that provides an effective representation of the whole piece. In a previous work, we introduced an attention-based convolutional recurrent neural network that uses music emotion classification as a surrogate task for music highlight extraction, for Pop songs. The rationale behind that approach is that the highlight of a song is usually the most emotional part. This paper extends our previous work in the following two aspects. First, methodology-wise we experiment with a new architecture that does not need any recurrent layers, making the training process faster. Moreover, we compare a late-fusion variant and an early-fusion variant to study which one better exploits the attention mechanism. Second, we conduct and report an extensive set of experiments comparing the proposed attention-based methods against a heuristic energy-based method, a structural repetition-based method, and a few other simple feature-based methods for this task. Due to the lack of public-domain labeled data for highlight extraction, following our previous work we use the RWC POP 100-song data set to evaluate how the detected highlights overlap with any chorus sections of the songs. The experiments demonstrate the effectiveness of our methods over competing methods. For reproducibility, we open source the code and pre-trained model at https://github.com/remyhuang/pop-music-highlighter/.