Voices of Civilizations: A Multilingual QA Benchmark for Global Music Understanding

TL;DR

This paper presents a multilingual question-answering benchmark for assessing audio language models' ability to understand cultural aspects of global music, highlighting current limitations and biases in model performance.

Contribution

It introduces Voices of Civilizations, a novel multilingual QA dataset for full-length music recordings, with automated question generation and manual verification, enabling cultural comprehension evaluation.

Findings

State-of-the-art models struggle with cultural nuances.

Models show systematic biases across different cultures.

Rich textual context improves model understanding.

Abstract

We introduce Voices of Civilizations, the first multilingual QA benchmark for evaluating audio LLMs' cultural comprehension on full-length music recordings. Covering 380 tracks across 38 languages, our automated pipeline yields 1,190 multiple-choice questions through four stages - each followed by manual verification: 1) compiling a representative music list; 2) generating cultural-background documents for each sample in the music list via LLMs; 3) extracting key attributes from those documents; and 4) constructing multiple-choice questions probing language, region associations, mood, and thematic content. We evaluate models under four conditions and report per-language accuracy. Our findings demonstrate that even state-of-the-art audio LLMs struggle to capture subtle cultural nuances without rich textual context and exhibit systematic biases in interpreting music from different cultural traditions. The dataset is publicly available on Hugging Face to foster culturally inclusive music understanding research.