Fine-Tuning MIDI-to-Audio Alignment using a Neural Network on Piano Roll and CQT Representations

TL;DR

This paper introduces a neural network-based method for aligning MIDI files with audio recordings of piano performances, outperforming traditional techniques like DTW in accuracy and robustness.

Contribution

The paper presents a novel CRNN architecture trained on augmented datasets to improve MIDI-to-audio alignment accuracy beyond existing methods.

Findings

Achieves up to 20% higher alignment accuracy than DTW

Combining CRNN with DTW further improves robustness

Demonstrates neural networks' potential in MIDI-audio alignment

Abstract

In this paper, we present a neural network approach for synchronizing audio recordings of human piano performances with their corresponding loosely aligned MIDI files. The task is addressed using a Convolutional Recurrent Neural Network (CRNN) architecture, which effectively captures spectral and temporal features by processing an unaligned piano roll and a spectrogram as inputs to estimate the aligned piano roll. To train the network, we create a dataset of piano pieces with augmented MIDI files that simulate common human timing errors. The proposed model achieves up to 20% higher alignment accuracy than the industry-standard Dynamic Time Warping (DTW) method across various tolerance windows. Furthermore, integrating DTW with the CRNN yields additional improvements, offering enhanced robustness and consistency. These findings demonstrate the potential of neural networks in advancing state-of-the-art MIDI-to-audio alignment.