Detecting Music Performance Errors with Transformers

TL;DR

This paper introduces Polytune, a transformer-based model for detecting music performance errors directly from audio, overcoming alignment issues and data scarcity, achieving significant accuracy improvements across multiple instruments.

Contribution

The paper presents a novel transformer model and a synthetic data generation technique for music error detection, enabling end-to-end training and multi-instrument handling.

Findings

Achieved 64.1% Error Detection F1 score, a 40-point improvement over prior work.

Can handle multiple instruments, unlike previous methods.

Uses synthetic data to address data scarcity in training.

Abstract

Beginner musicians often struggle to identify specific errors in their performances, such as playing incorrect notes or rhythms. There are two limitations in existing tools for music error detection: (1) Existing approaches rely on automatic alignment; therefore, they are prone to errors caused by small deviations between alignment targets.; (2) There is a lack of sufficient data to train music error detection models, resulting in over-reliance on heuristics. To address (1), we propose a novel transformer model, Polytune, that takes audio inputs and outputs annotated music scores. This model can be trained end-to-end to implicitly align and compare performance audio with music scores through latent space representations. To address (2), we present a novel data generation technique capable of creating large-scale synthetic music error datasets. Our approach achieves a 64.1% average Error Detection F1 score, improving upon prior work by 40 percentage points across 14 instruments. Additionally, compared with existing transcription methods repurposed for music error detection, our model can handle multiple instruments. Our source code and datasets are available at https://github.com/ben2002chou/Polytune.