MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization

TL;DR

MuQ introduces a self-supervised music representation learning model utilizing Mel Residual Vector Quantization, outperforming previous models on various music understanding tasks with less data and demonstrating strong zero-shot capabilities.

Contribution

The paper proposes MuQ, a novel self-supervised music representation model using Mel-RVQ for improved stability and efficiency, and introduces MuQ-MuLan for state-of-the-art zero-shot music tagging.

Findings

MuQ outperforms previous models on multiple downstream tasks.

Scaling data and iterative training further improve performance.

MuQ-MuLan achieves state-of-the-art zero-shot music tagging results.

Abstract

Recent years have witnessed the success of foundation models pre-trained with self-supervised learning (SSL) in various music informatics understanding tasks, including music tagging, instrument classification, key detection, and more. In this paper, we propose a self-supervised music representation learning model for music understanding. Distinguished from previous studies adopting random projection or existing neural codec, the proposed model, named MuQ, is trained to predict tokens generated by Mel Residual Vector Quantization (Mel-RVQ). Our Mel-RVQ utilizes residual linear projection structure for Mel spectrum quantization to enhance the stability and efficiency of target extraction and lead to better performance. Experiments in a large variety of downstream tasks demonstrate that MuQ outperforms previous self-supervised music representation models with only 0.9K hours of open-source pre-training data. Scaling up the data to over 160K hours and adopting iterative training consistently improve the model performance. To further validate the strength of our model, we present MuQ-MuLan, a joint music-text embedding model based on contrastive learning, which achieves state-of-the-art performance in the zero-shot music tagging task on the MagnaTagATune dataset. Code and checkpoints are open source in https://github.com/tencent-ailab/MuQ.