Melody transcription via generative pre-training

TL;DR

This paper introduces a novel melody transcription method leveraging generative pre-training with broad music representations and a new dataset, significantly improving accuracy and enabling direct transcription of lead sheets from audio.

Contribution

It presents a new approach combining generative pre-training and a large crowdsourced dataset to enhance melody transcription across diverse musical styles and instruments.

Findings

20% performance improvement over conventional methods

77% stronger performance with the new dataset

Enables direct transcription of lead sheets from audio

Abstract

Despite the central role that melody plays in music perception, it remains an open challenge in music information retrieval to reliably detect the notes of the melody present in an arbitrary music recording. A key challenge in melody transcription is building methods which can handle broad audio containing any number of instrument ensembles and musical styles - existing strategies work well for some melody instruments or styles but not all. To confront this challenge, we leverage representations from Jukebox (Dhariwal et al. 2020), a generative model of broad music audio, thereby improving performance on melody transcription by $20$% relative to conventional spectrogram features. Another obstacle in melody transcription is a lack of training data - we derive a new dataset containing $50$ hours of melody transcriptions from crowdsourced annotations of broad music. The combination of generative pre-training and a new dataset for this task results in $77$% stronger performance on melody transcription relative to the strongest available baseline. By pairing our new melody transcription approach with solutions for beat detection, key estimation, and chord recognition, we build Sheet Sage, a system capable of transcribing human-readable lead sheets directly from music audio. Audio examples can be found at https://chrisdonahue.com/sheetsage and code at https://github.com/chrisdonahue/sheetsage .