Piano Transcription by Hierarchical Language Modeling with Pretrained Roll-based Encoders

TL;DR

This paper introduces a hybrid hierarchical approach combining pre-trained roll-based encoders with a language model decoder for improved automatic music transcription, effectively handling long sequences and reducing computational costs.

Contribution

It presents a novel hybrid method with hierarchical prediction that leverages pre-trained encoders and a language model to enhance piano transcription accuracy.

Findings

Outperforms traditional piano-roll methods in F1 score by 0.01 and 0.022.

Reduces computational costs through hierarchical sequence processing.

Demonstrates potential as a plug-in for existing roll-based music transcription systems.

Abstract

Automatic Music Transcription (AMT), aiming to get musical notes from raw audio, typically uses frame-level systems with piano-roll outputs or language model (LM)-based systems with note-level predictions. However, frame-level systems require manual thresholding, while the LM-based systems struggle with long sequences. In this paper, we propose a hybrid method combining pre-trained roll-based encoders with an LM decoder to leverage the strengths of both methods. Besides, our approach employs a hierarchical prediction strategy, first predicting onset and pitch, then velocity, and finally offset. The hierarchical prediction strategy reduces computational costs by breaking down long sequences into different hierarchies. Evaluated on two benchmark roll-based encoders, our method outperforms traditional piano-roll outputs 0.01 and 0.022 in onset-offset-velocity F1 score, demonstrating its potential as a performance-enhancing plug-in for arbitrary roll-based music transcription encoder.