# Score Images as a Modality: Enhancing Symbolic Music Understanding through Large-Scale Multimodal Pre-Training

## TL;DR

This paper introduces a new method for understanding symbolic music by combining music score images with MIDI data using multimodal pre-training.

## Contribution

The novel SIM model and pre-training tasks that integrate score images and MIDI for symbolic music understanding.

## Key findings

- The SIM model effectively captures music structures and aligns visual and symbolic representations.
- The proposed pre-training tasks improve symbolic music understanding through experimental validation.

## Abstract

Symbolic music understanding is a critical challenge in artificial intelligence. While traditional symbolic music representations like MIDI capture essential musical elements, they often lack the nuanced expression in music scores. Leveraging the advancements in multimodal pre-training, particularly in visual-language pre-training, we propose a groundbreaking approach: the Score Images as a Modality (SIM) model. This model integrates music score images alongside MIDI data for enhanced symbolic music understanding. We also introduce novel pre-training tasks, including masked bar-attribute modeling and score-MIDI matching. These tasks enable the SIM model to capture music structures and align visual and symbolic representations effectively. Additionally, we present a meticulously curated dataset of matched score images and MIDI representations optimized for training the SIM model. Through experimental validation, we demonstrate the efficacy of our approach in advancing symbolic music understanding.