CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following

TL;DR

CMI-Bench introduces a comprehensive benchmark for evaluating music instruction-following capabilities of audio-text LLMs across diverse MIR tasks, highlighting current model limitations and biases.

Contribution

This paper presents CMI-Bench, a new standardized benchmark for assessing music instruction-following in audio-text LLMs across multiple MIR tasks, enabling fair comparison and driving progress.

Findings

Significant performance gaps between LLMs and supervised models.

Identification of cultural, chronological, and gender biases in models.

Benchmark supports multiple open-source audio-text LLMs.

Abstract

Recent advances in audio-text large language models (LLMs) have opened new possibilities for music understanding and generation. However, existing benchmarks are limited in scope, often relying on simplified tasks or multi-choice evaluations that fail to reflect the complexity of real-world music analysis. We reinterpret a broad range of traditional MIR annotations as instruction-following formats and introduce CMI-Bench, a comprehensive music instruction following benchmark designed to evaluate audio-text LLMs on a diverse set of music information retrieval (MIR) tasks. These include genre classification, emotion regression, emotion tagging, instrument classification, pitch estimation, key detection, lyrics transcription, melody extraction, vocal technique recognition, instrument performance technique detection, music tagging, music captioning, and (down)beat tracking: reflecting core challenges in MIR research. Unlike previous benchmarks, CMI-Bench adopts standardized evaluation metrics consistent with previous state-of-the-art MIR models, ensuring direct comparability with supervised approaches. We provide an evaluation toolkit supporting all open-source audio-textual LLMs, including LTU, Qwen-audio, SALMONN, MusiLingo, etc. Experiment results reveal significant performance gaps between LLMs and supervised models, along with their culture, chronological and gender bias, highlighting the potential and limitations of current models in addressing MIR tasks. CMI-Bench establishes a unified foundation for evaluating music instruction following, driving progress in music-aware LLMs.