MIDI-LLaMA: An Instruction-Following Multimodal LLM for Symbolic Music Understanding

TL;DR

MIDI-LLaMA is a novel instruction-following multimodal large language model designed specifically for symbolic music understanding, demonstrating superior performance in music captioning, semantic alignment, and human-evaluated musical comprehension tasks.

Contribution

It introduces the first instruction-following MLLM for symbolic music, combining MusicBERT and Llama-3-8B through a two-stage training pipeline with a new MIDI-text dataset.

Findings

Outperforms baseline in music captioning and question answering

Human evaluation shows better music understanding and emotion recognition

Enhances LLM capabilities with symbolic music understanding

Abstract

Recent advances in multimodal large language models (MLLM) for audio music have demonstrated strong capabilities in music understanding, yet symbolic music, a fundamental representation of musical structure, remains unexplored. In this work, we introduce MIDI-LLaMA, the first instruction-following MLLM for symbolic music understanding. Our approach aligns the MIDI encoder MusicBERT and Llama-3-8B via a two-stage pipeline comprising feature alignment and instruction tuning. To support training, we design a scalable annotation pipeline that annotates GiantMIDI-Piano with fine-grained metadata, resulting in a MIDI-text dataset. Compared with the baseline trained on converting MIDI into ABC notation under the same instruction-tuning procedure, MIDI-LLaMA substantially outperforms in captioning and semantic alignment in question answering. Human evaluation further confirms the advantages of MIDI-LLaMA in music understanding, emotion recognition, creativity, and overall preference. These findings demonstrate that incorporating symbolic music into large language models enhances their capacity for musical understanding.