Real-Time Control of a Virtual Orchestra by Recognition of Conducting Gestures

TL;DR

This paper introduces a real-time virtual orchestra conducting system in a dome theater, using gesture recognition to control playback speed based on visitor gestures, enhancing museum interactive experiences.

Contribution

It presents a novel gesture-based control system for virtual orchestras using a hierarchical LSTM network trained on multiple conductors' gestures.

Findings

Quantitative evaluation shows accurate timing control.

User study indicates high realism and usability.

Field study demonstrates effective visitor engagement.

Abstract

We present a museum installation in a 180{\deg} dome theater, which gives the museum visitor the experience of conducting a symphony orchestra. We have pre-recorded a short music piece performed by a professional orchestra. This recording is played back in the dome with the visitor standing in the conductor's position. The visitor's gestures are captured with a vision-based skeleton tracker, steering the recording playback pace via a gesture recognition module that translates the gestures into a time control signal. This is sent to a playback module that plays the recording in the dome at the corresponding speed. The gesture recognition module is based on a hierarchical LSTM network, trained with recorded sequences of multiple conductors with different level of expertise conducting the same recording. The system is evaluated with a quantitative study of the estimated timing accuracy, a user study evaluating the musical realism and usability of the real-time control, and a field study to evaluate the performance of the entire system with real museum visitors.