Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generation

TL;DR

This paper evaluates how well large language models understand and generate symbolic music, revealing their limitations in multi-step reasoning and highlighting the need for better integration of musical knowledge and reasoning capabilities.

Contribution

The study provides a comprehensive assessment of LLMs' performance in symbolic music understanding and generation, emphasizing their shortcomings in multi-step musical reasoning.

Findings

LLMs perform poorly on song-level multi-step reasoning tasks.

LLMs often fail to utilize learned musical knowledge in complex tasks.

Advanced musical reasoning is not inherently achieved by current LLMs.

Abstract

Symbolic Music, akin to language, can be encoded in discrete symbols. Recent research has extended the application of large language models (LLMs) such as GPT-4 and Llama2 to the symbolic music domain including understanding and generation. Yet scant research explores the details of how these LLMs perform on advanced music understanding and conditioned generation, especially from the multi-step reasoning perspective, which is a critical aspect in the conditioned, editable, and interactive human-computer co-creation process. This study conducts a thorough investigation of LLMs' capability and limitations in symbolic music processing. We identify that current LLMs exhibit poor performance in song-level multi-step music reasoning, and typically fail to leverage learned music knowledge when addressing complex musical tasks. An analysis of LLMs' responses highlights distinctly their pros and cons. Our findings suggest achieving advanced musical capability is not intrinsically obtained by LLMs, and future research should focus more on bridging the gap between music knowledge and reasoning, to improve the co-creation experience for musicians.