CultureMERT: Continual Pre-Training for Cross-Cultural Music Representation Learning

TL;DR

CultureMERT-95M is a new multi-cultural music foundation model that uses continual pre-training to improve cross-cultural music understanding, outperforming previous models on non-Western music tasks while maintaining Western music performance.

Contribution

The paper introduces a novel two-stage continual pre-training strategy and a multi-cultural dataset to enhance cross-cultural music representation learning.

Findings

4.9% improvement in non-Western music auto-tagging

Stable adaptation with limited resources using re-warming and re-decaying

Multi-cultural model outperforms single-culture models overall

Abstract

Recent advances in music foundation models have improved audio representation learning, yet their effectiveness across diverse musical traditions remains limited. We introduce CultureMERT-95M, a multi-culturally adapted foundation model developed to enhance cross-cultural music representation learning and understanding. To achieve this, we propose a two-stage continual pre-training strategy that integrates learning rate re-warming and re-decaying, enabling stable adaptation even with limited computational resources. Training on a 650-hour multi-cultural data mix, comprising Greek, Turkish, and Indian music traditions, results in an average improvement of 4.9% in ROC-AUC and AP across diverse non-Western music auto-tagging tasks, surpassing prior state-of-the-art, with minimal forgetting on Western-centric benchmarks. We further investigate task arithmetic, an alternative approach to multi-cultural adaptation that merges single-culture adapted models in the weight space. Task arithmetic performs on par with our multi-culturally trained model on non-Western auto-tagging tasks and shows no regression on Western datasets. Cross-cultural evaluation reveals that single-culture models transfer with varying effectiveness across musical traditions, whereas the multi-culturally adapted model achieves the best overall performance. To support research on world music representation learning, we publicly release CultureMERT-95M and CultureMERT-TA-95M, fostering the development of more culturally aware music foundation models.