Wish You Were Here: Mental and Physiological Effects of Remote Music Collaboration in Mixed Reality

TL;DR

This study investigates how mixed reality technology influences musicians' mental states and physiological responses during remote collaboration, highlighting increased co-presence and potential indicators of flow.

Contribution

It provides empirical evidence on the effects of mixed reality on musical collaboration, including physiological markers and subjective experiences, filling a gap in remote music interaction research.

Findings

Increased co-presence in mixed reality condition

Positive affect increased after remote jamming regardless of condition

Heart rate and HF/LF features can classify flow states

Abstract

With face-to-face music collaboration being severely limited during the recent pandemic, mixed reality technologies and their potential to provide musicians a feeling of "being there" with their musical partner can offer tremendous opportunities. In order to assess this potential, we conducted a laboratory study in which musicians made music together in real-time while simultaneously seeing their jamming partner's mixed reality point cloud via a head-mounted display and compared mental effects such as flow, affect, and co-presence to an audio-only baseline. In addition, we tracked the musicians' physiological signals and evaluated their features during times of self-reported flow. For users jamming in mixed reality, we observed a significant increase in co-presence. Regardless of the condition (mixed reality or audio-only), we observed an increase in positive affect after jamming remotely. Furthermore, we identified heart rate and HF/LF as promising features for classifying the flow state musicians experienced while making music together.