MARBLE: Music Audio Representation Benchmark for Universal Evaluation

TL;DR

MARBLE introduces a comprehensive benchmark for evaluating music audio representations across multiple tasks, datasets, and models, fostering standardized assessment and advancement in music AI understanding.

Contribution

It provides the first universal, community-driven benchmark for music representations, including a taxonomy, evaluation protocol, and open-source toolkit for fair comparison.

Findings

Large-scale pre-trained musical models perform best on most tasks.

The benchmark reveals room for improvement in current music AI models.

MARBLE facilitates fair, reproducible evaluation of music representations.

Abstract

In the era of extensive intersection between art and Artificial Intelligence (AI), such as image generation and fiction co-creation, AI for music remains relatively nascent, particularly in music understanding. This is evident in the limited work on deep music representations, the scarcity of large-scale datasets, and the absence of a universal and community-driven benchmark. To address this issue, we introduce the Music Audio Representation Benchmark for universaL Evaluation, termed MARBLE. It aims to provide a benchmark for various Music Information Retrieval (MIR) tasks by defining a comprehensive taxonomy with four hierarchy levels, including acoustic, performance, score, and high-level description. We then establish a unified protocol based on 14 tasks on 8 public-available datasets, providing a fair and standard assessment of representations of all open-sourced pre-trained models developed on music recordings as baselines. Besides, MARBLE offers an easy-to-use, extendable, and reproducible suite for the community, with a clear statement on copyright issues on datasets. Results suggest recently proposed large-scale pre-trained musical language models perform the best in most tasks, with room for further improvement. The leaderboard and toolkit repository are published at https://marble-bm.shef.ac.uk to promote future music AI research.