Handling Structural Mismatches in Real-time Opera Tracking

TL;DR

This paper presents a dual-level real-time score following system for opera performances that effectively manages structural mismatches like jumps and repetitions, improving robustness over traditional linear assumptions.

Contribution

It introduces a novel two-level tracking approach combining high-resolution and low-resolution trackers to handle structural mismatches in real-time opera tracking.

Findings

Enhanced robustness in score following with structural mismatches

Effective handling of jumps and repetitions in live opera

Improved real-time tracking accuracy

Abstract

Algorithms for reliable real-time score following in live opera promise a lot of useful applications such as automatic subtitles display, or real-time video cutting in live streaming. Until now, such systems were based on the strong assumption that an opera performance follows the structure of the score linearly. However, this is rarely the case in practice, because of different opera versions and directors' cutting choices. In this paper, we propose a two-level solution to this problem. We introduce a real-time-capable, high-resolution (HR) tracker that can handle jumps or repetitions at specific locations provided to it. We then combine this with an additional low-resolution (LR) tracker that can handle all sorts of mismatches that can occur at any time, with some imprecision, and can re-direct the HR tracker if the latter is `lost' in the score. We show that the combination of the two improves tracking robustness in the presence of strong structural mismatches.