Music Arena: Live Evaluation for Text-to-Music

TL;DR

Music Arena is an open platform enabling scalable, live human preference evaluations for text-to-music models, fostering standardized comparison, data transparency, and domain-specific features to advance TTM research.

Contribution

It introduces a live evaluation platform with tailored features for TTM, including an LLM-based routing system and renewable preference data collection.

Findings

Established a scalable human preference evaluation protocol.

Created a renewable, privacy-preserving preference data resource.

Demonstrated the platform's utility in benchmarking TTM systems.

Abstract

We present Music Arena, an open platform for scalable human preference evaluation of text-to-music (TTM) models. Soliciting human preferences via listening studies is the gold standard for evaluation in TTM, but these studies are expensive to conduct and difficult to compare, as study protocols may differ across systems. Moreover, human preferences might help researchers align their TTM systems or improve automatic evaluation metrics, but an open and renewable source of preferences does not currently exist. We aim to fill these gaps by offering *live* evaluation for TTM. In Music Arena, real-world users input text prompts of their choosing and compare outputs from two TTM systems, and their preferences are used to compile a leaderboard. While Music Arena follows recent evaluation trends in other AI domains, we also design it with key features tailored to music: an LLM-based routing system to navigate the heterogeneous type signatures of TTM systems, and the collection of *detailed* preferences including listening data and natural language feedback. We also propose a rolling data release policy with user privacy guarantees, providing a renewable source of preference data and increasing platform transparency. Through its standardized evaluation protocol, transparent data access policies, and music-specific features, Music Arena not only addresses key challenges in the TTM ecosystem but also demonstrates how live evaluation can be thoughtfully adapted to unique characteristics of specific AI domains. Music Arena is available at: https://music-arena.org . Preference data is available at: https://huggingface.co/music-arena .