MusicSem: A Semantically Rich Language--Audio Dataset of Natural Music Descriptions

TL;DR

MusicSem is a large, semantically rich dataset of natural language descriptions of music from social media, designed to improve multimodal music understanding and generation models by capturing nuanced human expressions.

Contribution

The paper introduces MusicSem, a novel dataset of 32,493 language-audio pairs with a semantic taxonomy, enhancing the training and evaluation of human-aligned multimodal music models.

Findings

MusicSem captures diverse and nuanced music descriptions from social media.

Multimodal models evaluated on MusicSem show the importance of fine-grained semantics.

The dataset supports improved retrieval and generation tasks in music information retrieval.

Abstract

Music representation learning is central to music information retrieval and generation. While recent advances in multimodal learning have improved alignment between text and audio for tasks such as cross-modal music retrieval, text-to-music generation, and music-to-text generation, existing models often struggle to capture users' expressed intent in natural language descriptions of music. This observation suggests that the datasets used to train and evaluate these models do not fully reflect the broader and more natural forms of human discourse through which music is described. In this paper, we introduce MusicSem, a dataset of 32,493 language-audio pairs derived from organic music-related discussions on the social media platform Reddit. Compared to existing datasets, MusicSem captures a broader spectrum of musical semantics, reflecting how listeners naturally describe music in nuanced and human-centered ways. To structure these expressions, we propose a taxonomy of five semantic categories: descriptive, atmospheric, situational, metadata-related, and contextual. In addition to the construction, analysis, and release of MusicSem, we use the dataset to evaluate a wide range of multimodal models for retrieval and generation, highlighting the importance of modeling fine-grained semantics. Overall, MusicSem serves as a novel semantics-aware resource to support future research on human-aligned multimodal music representation learning.