Let Network Decide What to Learn: Symbolic Music Understanding Model Based on Large-scale Adversarial Pre-training

TL;DR

This paper introduces Adversarial-MidiBERT, a novel pre-training approach for symbolic music understanding that adaptively masks tokens to improve contextual learning and reduce bias, outperforming traditional methods.

Contribution

The paper proposes a new adversarial masking strategy for pre-training models in symbolic music understanding, enhancing contextual capture and reducing bias compared to standard MLM methods.

Findings

Achieves superior performance across four SMU tasks.

Effectively reduces bias associated with random masking.

Demonstrates the benefit of adaptive masking in music understanding models.

Abstract

As a crucial aspect of Music Information Retrieval (MIR), Symbolic Music Understanding (SMU) has garnered significant attention for its potential to assist both musicians and enthusiasts in learning and creating music. Recently, pre-trained language models have been widely adopted in SMU due to the substantial similarities between symbolic music and natural language, as well as the ability of these models to leverage limited music data effectively. However, some studies have shown the common pre-trained methods like Mask Language Model (MLM) may introduce bias issues like racism discrimination in Natural Language Process (NLP) and affects the performance of downstream tasks, which also happens in SMU. This bias often arises when masked tokens cannot be inferred from their context, forcing the model to overfit the training set instead of generalizing. To address this challenge, we propose Adversarial-MidiBERT for SMU, which adaptively determines what to mask during MLM via a masker network, rather than employing random masking. By avoiding the masking of tokens that are difficult to infer from context, our model is better equipped to capture contextual structures and relationships, rather than merely conforming to the training data distribution. We evaluate our method across four SMU tasks, and our approach demonstrates excellent performance in all cases. The code for our model is publicly available at https://github.com/RS2002/Adversarial-MidiBERT .