Improving Real-time Score Following in Opera by Combining Music with Lyrics Tracking

TL;DR

This paper introduces a novel opera tracking pipeline combining music and lyrics trackers to improve accuracy and robustness in real-time score following, addressing challenges posed by the genre's acoustic complexity.

Contribution

The paper presents a new combined tracking approach that integrates music and lyrics tracking to enhance real-time opera score following performance.

Findings

Improved accuracy in opera tracking demonstrated on Don Giovanni.

Enhanced robustness of tracking in complex acoustic environments.

Effective correction of music tracker errors using lyrics information.

Abstract

Fully automatic opera tracking is challenging because of the acoustic complexity of the genre, combining musical and linguistic information (singing, speech) in complex ways. In this paper, we propose a new pipeline for complete opera tracking. The pipeline is based on two trackers. A music tracker that has proven to be effective at tracking orchestral parts, will lead the tracking process. In addition, a lyrics tracker, that has recently been shown to reliably track the lyrics of opera songs, will correct the music tracker when tracking parts that have a text dominance over the music. We will demonstrate the efficiency of this method on the opera Don Giovanni, showing that this technique helps improving accuracy and robustness of a complete opera tracker.