Musical Score Following and Audio Alignment

TL;DR

This paper reviews score following research, introduces new evaluation datasets, and presents three diverse score followers, notably a CQT-DTW method that outperforms traditional FFT-based approaches in robustness and accuracy.

Contribution

It introduces two open-source evaluation testbenches, a new dataset for qualitative benchmarking, and develops three score followers, including a novel CQT-DTW method that improves robustness in complex musical scenarios.

Findings

CQT-DTW score follower outperforms FFT-based methods.

Introduction of QualScofo dataset for qualitative benchmarking.

Development of diverse score followers demonstrating different applications.

Abstract

Real-time tracking of the position of a musical performance on a musical score, i.e. score following, can be useful in music practice, performance and production. Example applications of such technology include computer-aided accompaniment and automatic page turning. Score following is a challenging task, especially when considering deviations in performance data from the score stemming from mistakes or expressive choices. In this project, the extensive research present in the field is first explored before two open-source evaluation testbenches for score following--one quantitative and the other qualitative--are introduced. A new way of obtaining quantitative testbench data is proposed, and the QualScofo dataset for qualitative benchmarking is introduced. Subsequently, three different score followers, each of a different class, are implemented. First, a beat-based follower for an interactive conductor application--the TuneApp Conductor--is created to demonstrate an entertaining application of score following. Then, an Approximate String Matching (ASM) non-real-time follower is implemented to complement the quantitative testbench and provide more technical background details of score following. Finally, a Constant Q-Transform (CQT) Dynamic Time Warping (DTW) score follower robust against major challenges in score following (such as polyphonic music and performance deviations) is outlined and implemented; it is shown that this CQT-based approach consistently and significantly outperforms a commonly used FFT-based approach in extracting audio features for score following.