IncompeBench: A Permissively Licensed, Fine-Grained Benchmark for Music Information Retrieval

TL;DR

IncompeBench is a new, high-quality, permissively licensed benchmark dataset designed for evaluating music information retrieval systems, featuring extensive annotations and relevance judgments to facilitate progress in the field.

Contribution

The paper introduces IncompeBench, a comprehensive, annotated benchmark dataset for music retrieval, addressing the lack of high-quality evaluation resources in MIR.

Findings

High agreement between human annotators

Large dataset with over 125,000 relevance judgments

Publicly available datasets for research use

Abstract

Multimodal Information Retrieval has made significant progress in recent years, leveraging the increasingly strong multimodal abilities of deep pre-trained models to represent information across modalities. Music Information Retrieval (MIR), in particular, has considerably increased in quality, with neural representations of music even making its way into everyday life products. However, there is a lack of high-quality benchmarks for evaluating music retrieval performance. To address this issue, we introduce \textbf{IncompeBench}, a carefully annotated benchmark comprising $1,574$ permissively licensed, high-quality music snippets, $500$ diverse queries, and over $125,000$ individual relevance judgements. These annotations were created through the use of a multi-stage pipeline, resulting in high agreement between human annotators and the generated data. The resulting datasets are publicly available at https://huggingface.co/datasets/mixedbread-ai/incompebench-strict and https://huggingface.co/datasets/mixedbread-ai/incompebench-lenient with the prompts available at https://github.com/mixedbread-ai/incompebench-programs.