Quantifying the Corpus Bias Problem in Automatic Music Transcription Systems

TL;DR

This paper investigates how current automatic music transcription systems perform poorly on music genres and styles outside of classical piano, highlighting the corpus bias problem and its impact on generalization.

Contribution

It introduces new test sets to evaluate the effect of musical distribution shifts and quantifies the performance gap caused by corpus bias in AMT systems.

Findings

Significant performance drop on non-piano music genres

Corpus bias limits generalization of state-of-the-art AMT systems

Performance gap increases with greater musical distribution shift

Abstract

Automatic Music Transcription (AMT) is the task of recognizing notes in audio recordings of music. The State-of-the-Art (SotA) benchmarks have been dominated by deep learning systems. Due to the scarcity of high quality data, they are usually trained and evaluated exclusively or predominantly on classical piano music. Unfortunately, that hinders our ability to understand how they generalize to other music. Previous works have revealed several aspects of memorization and overfitting in these systems. We identify two primary sources of distribution shift: the music, and the sound. Complementing recent results on the sound axis (i.e. acoustics, timbre), we investigate the musical one (i.e. note combinations, dynamics, genre). We evaluate the performance of several SotA AMT systems on two new experimental test sets which we carefully construct to emulate different levels of musical distribution shift. Our results reveal a stark performance gap, shedding further light on the Corpus Bias problem, and the extent to which it continues to trouble these systems.