HumMusQA: A Human-written Music Understanding QA Benchmark Dataset

TL;DR

This paper introduces HumMusQA, a carefully curated dataset of 320 expert-written music questions designed to evaluate Large Audio-Language Models' music understanding capabilities.

Contribution

It presents a new benchmark dataset with expert-curated questions and evaluates six state-of-the-art models on music comprehension tasks.

Findings

Models show varying performance on the dataset.

The dataset reveals models' robustness to uni-modal shortcuts.

Expert-curated questions improve evaluation quality.

Abstract

The evaluation of music understanding in Large Audio-Language Models (LALMs) requires a rigorously defined benchmark that truly tests whether models can perceive and interpret music, a standard that current data methodologies frequently fail to meet. This paper introduces a meticulously structured approach to music evaluation, proposing a new dataset of 320 hand-written questions curated and validated by experts with musical training, arguing that such focused, manual curation is superior for probing complex audio comprehension. To demonstrate the use of the dataset, we benchmark six state-of-the-art LALMs and additionally test their robustness to uni-modal shortcuts.