Music-to-Text Synaesthesia: Generating Descriptive Text from Music Recordings

TL;DR

This paper introduces a novel task called music-to-text synaesthesia, which generates descriptive texts from music recordings, supported by a new dataset and a model with a topology-preservation loss, demonstrating superior results.

Contribution

The paper presents the first dataset and model for music-to-text synaesthesia, advancing understanding of music content through descriptive text generation.

Findings

The proposed model outperforms five baseline methods.

The topology-preservation loss improves descriptive accuracy.

Qualitative analysis confirms the model's effectiveness.

Abstract

In this paper, we consider a novel research problem: music-to-text synaesthesia. Different from the classical music tagging problem that classifies a music recording into pre-defined categories, music-to-text synaesthesia aims to generate descriptive texts from music recordings with the same sentiment for further understanding. As existing music-related datasets do not contain the semantic descriptions on music recordings, we collect a new dataset that contains 1,955 aligned pairs of classical music recordings and text descriptions. Based on this, we build a computational model to generate sentences that can describe the content of the music recording. To tackle the highly non-discriminative classical music, we design a group topology-preservation loss, which considers more samples as a group reference and preserves the relative topology among different samples. Extensive experimental results qualitatively and quantitatively demonstrate the effectiveness of our proposed model over five heuristics or pre-trained competitive methods and their variants on our collected dataset.