Towards Context-Aware Neural Performance-Score Synchronisation

TL;DR

This paper introduces a novel, data-driven, context-aware neural approach for music performance-score synchronization that overcomes limitations of traditional methods by handling structural differences and varying acoustic conditions.

Contribution

It proposes a neural, metric learning-based synchronization method that is adaptable, robust to structural differences, and eliminates the need for handcrafted features and dynamic programming.

Findings

Outperforms traditional methods in diverse acoustic environments

Effectively handles structural differences between performances and scores

Demonstrates robustness in data-scarce scenarios

Abstract

Music can be represented in multiple forms, such as in the audio form as a recording of a performance, in the symbolic form as a computer readable score, or in the image form as a scan of the sheet music. Music synchronisation provides a way to navigate among multiple representations of music in a unified manner by generating an accurate mapping between them, lending itself applicable to a myriad of domains like music education, performance analysis, automatic accompaniment and music editing. Traditional synchronisation methods compute alignment using knowledge-driven and stochastic approaches, typically employing handcrafted features. These methods are often unable to generalise well to different instruments, acoustic environments and recording conditions, and normally assume complete structural agreement between the performances and the scores. This PhD furthers the development of performance-score synchronisation research by proposing data-driven, context-aware alignment approaches, on three fronts: Firstly, I replace the handcrafted features by employing a metric learning based approach that is adaptable to different acoustic settings and performs well in data-scarce conditions. Secondly, I address the handling of structural differences between the performances and scores, which is a common limitation of standard alignment methods. Finally, I eschew the reliance on both feature engineering and dynamic programming, and propose a completely data-driven synchronisation method that computes alignments using a neural framework, whilst also being robust to structural differences between the performances and scores.