GaMMA: Towards Joint Global-Temporal Music Understanding in Large Multimodal Models

TL;DR

GaMMA is a large multimodal model that advances comprehensive music understanding by integrating audio and language processing, trained on extensive datasets, and evaluated with a new benchmark, MusicBench.

Contribution

This paper introduces GaMMA, a novel multimodal model that unifies temporal and non-temporal music understanding tasks within a single framework, and presents MusicBench, a large-scale music understanding benchmark.

Findings

GaMMA achieves state-of-the-art accuracy on multiple music understanding tasks.

GaMMA effectively unifies time-series and non-time-series music understanding.

MusicBench provides a comprehensive evaluation platform for music LMMs.

Abstract

In this paper, we propose GaMMA, a state-of-the-art (SoTA) large multimodal model (LMM) designed to achieve comprehensive musical content understanding. GaMMA inherits the streamlined encoder-decoder design of LLaVA, enabling effective cross-modal learning between music and language. By incorporating audio encoders in a mixture-of-experts manner, GaMMA effectively unifies both time-series and non-time-series music understanding tasks within one set of parameters. Our approach combines carefully curated datasets at scale with a progressive training pipeline, effectively pushing the boundaries of music understanding via pretraining, supervised fine-tuning (SFT), and reinforcement learning (RL). To comprehensively assess both temporal and non-temporal capability of music LMMs, we introduce MusicBench, the largest music-oriented benchmark, comprising 3,739 human-curated multiple-choice questions covering diverse aspects of musical understanding. Extensive experiments demonstrate that GaMMA establishes new SoTA in the music domain, achieving 79.1% accuracy on MuchoMusic, 79.3% on MusicBench-Temporal, and 81.3% on MusicBench-Global, consistently outperforming previous methods.