Factual and Musical Evaluation Metrics for Music Language Models

TL;DR

This paper highlights the inadequacy of existing evaluation metrics for music language models and proposes new metrics and frameworks to better assess their factual correctness and performance.

Contribution

It introduces a domain-adapted evaluation metric and a factual correctness framework for Music LMs, addressing limitations of traditional linguistic metrics.

Findings

Existing metrics like BLEU and BERTScore do not measure factual correctness.

Proposed evaluation framework is modality-agnostic and generalizable.

Experiments on open datasets demonstrate the effectiveness of the new metrics.

Abstract

Music language models (Music LMs), like vision language models, leverage multimodal representations to answer natural language queries about musical audio recordings. Although Music LMs are reportedly improving, we find that current evaluations fail to capture whether their answers are correct. Specifically, for all Music LMs that we examine, widely-used evaluation metrics such as BLEU, METEOR, and BERTScore fail to measure anything beyond linguistic fluency of the model's responses. To measure the true performance of Music LMs, we propose (1) a better general-purpose evaluation metric for Music LMs adapted to the music domain and (2) a factual evaluation framework to quantify the correctness of a Music LM's responses. Our framework is agnostic to the modality of the question-answering model and could be generalized to quantify performance in other open-ended question-answering domains. We use open datasets in our experiments and will release all code on publication.