Coupling the Heart to Musical Machines

TL;DR

This paper introduces a novel musical interface controlled by heart rate using PPG sensors, creating a biofeedback loop that influences sound perception and performance possibilities.

Contribution

It demonstrates a PPG-based NIME controller that uses heart rate as a real-time control parameter for musical expression, expanding biofeedback applications in music interfaces.

Findings

Real-time heart rate influences sound transformation via time scaling.

Biofeedback loop enhances control and expressive possibilities in musical performance.

Potential for extended control through generative latent spaces.

Abstract

Biofeedback is being used more recently as a general control paradigm for human-computer interfaces (HCIs). While biofeedback especially from breath has seen increasing uptake as a controller for novel musical interfaces, new interfaces for musical expression (NIMEs), the community has not given as much attention to the heart. The heart is just as intimate a part of music as breath and it is argued that the heart determines our perception of time and so indirectly our perception of music. Inspired by this I demonstrate a photoplethysmogram (PPG)-based NIME controller using heart rate as a 1D control parameter to transform the qualities of sounds in real-time over a Bluetooth wireless HCI. I apply time scaling to "warp" audio buffers inbound to the sound card, and play these transformed audio buffers back to the listener wearing the PPG sensor, creating a hypothetical perceptual biofeedback loop: changes in sound change heart rate to change PPG measurements to change sound. I discuss how a sound-heart-PPG biofeedback loop possibly affords greater control and/or variety of movements with a 1D controller, how controlling the space and/or time scale of sound playback with biofeedback makes for possibilities in performance ambience, and I briefly discuss generative latent spaces as a possible way to extend a 1D PPG control space.