MusCaps: Generating Captions for Music Audio

TL;DR

This paper introduces MusCaps, the first deep learning model for generating natural language descriptions of music audio, outperforming non-music captioning baselines and emphasizing pre-training of audio encoders.

Contribution

It presents a novel audio captioning model for music, combining multimodal encoding and pre-training, shifting from classification to descriptive approaches in music information retrieval.

Findings

Pre-training of audio encoder significantly improves caption quality.

MusCaps outperforms non-music captioning baselines.

Design choices like modality fusion and attention have marginal impact.

Abstract

Content-based music information retrieval has seen rapid progress with the adoption of deep learning. Current approaches to high-level music description typically make use of classification models, such as in auto-tagging or genre and mood classification. In this work, we propose to address music description via audio captioning, defined as the task of generating a natural language description of music audio content in a human-like manner. To this end, we present the first music audio captioning model, MusCaps, consisting of an encoder-decoder with temporal attention. Our method combines convolutional and recurrent neural network architectures to jointly process audio-text inputs through a multimodal encoder and leverages pre-training on audio data to obtain representations that effectively capture and summarise musical features in the input. Evaluation of the generated captions through automatic metrics shows that our method outperforms a baseline designed for non-music audio captioning. Through an ablation study, we unveil that this performance boost can be mainly attributed to pre-training of the audio encoder, while other design choices - modality fusion, decoding strategy and the use of attention - contribute only marginally. Our model represents a shift away from classification-based music description and combines tasks requiring both auditory and linguistic understanding to bridge the semantic gap in music information retrieval.