Real-time auralization for performers on virtual stages

TL;DR

This paper introduces a real-time auralization system for virtual stage performance, emphasizing calibration, latency compensation, and experimental validation for enhanced psychophysical music performance studies.

Contribution

It presents a novel calibration and latency compensation approach tailored for interactive virtual stage acoustics experiments.

Findings

Calibration filters effectively account for microphone-instrument distances and directivity.

Latency compensation via direct sound bypass enables real-time interaction without cropping impulse responses.

Objective and subjective tests support the system's feasibility for psychophysical experiments.

Abstract

This article presents an interactive system for stage acoustics experimentation including considerations for hearing one's own and others' instruments. The quality of real-time auralization systems for psychophysical experiments on music performance depends on the system's calibration and latency, among other factors (e.g. visuals, simulation methods, haptics, etc). The presented system focuses on the acoustic considerations for laboratory implementations. The calibration is implemented as a set of filters accounting for the microphone-instrument distances and the directivity factors, as well as the transducers' frequency responses. Moreover, sources of errors are characterized using both state-of-the-art information and derivations from the mathematical definition of the calibration filter. In order to compensate for hardware latency without cropping parts of the simulated impulse responses, the virtual direct sound of musicians hearing themselves is skipped from the simulation and addressed by letting the actual direct sound reach the listener through open headphones. The required latency compensation of the interactive part (i.e. hearing others) meets the minimum distance requirement between musicians, which is 2 m for the implemented system. Finally, a proof of concept is provided that includes objective and subjective experiments, which give support to the feasibility of the proposed setup.