TONUS: Neuromorphic human pose estimation for artistic sound co-creation

TL;DR

This paper introduces TONUS, a neuromorphic human pose estimation system designed for artistic sound co-creation, enabling seamless, non-intrusive interaction that enhances artistic expression and engagement.

Contribution

It presents a novel neuromorphic multihead human pose estimation neural sensor tailored for artistic sound installations, bridging advanced neural sensing with creative human-machine interaction.

Findings

Enables real-time soundscape shaping through body movements.

Provides a seamless, non-intrusive interaction interface.

Demonstrates effective neuromorphic pose estimation in artistic contexts.

Abstract

Human machine interaction is a huge source of inspiration in today's media art and digital design, as machines and humans merge together more and more. Its place in art reflects its growing applications in industry, such as robotics. However, those interactions often remains too technical and machine-driven for people to really engage into. On the artistic side, new technologies are often not explored in their full potential and lag a bit behind, so that state-of-the-art research does not make its way up to museums and exhibitions. Machines should support people's imagination and poetry in a seamless interface to their body or soul. We propose an artistic sound installation featuring neuromorphic body sensing to support a direct yet non intrusive interaction with the visitor with the purpose of creating sound scapes together with the machine. We design a neuromorphic multihead human pose estimation neural sensor that shapes sound scapes and visual output with fine body movement control. In particular, the feature extractor is a spiking neural network tailored for a dedicated neuromorphic chip. The visitor, immersed in a sound atmosphere and a neurally processed representation of themselves that they control, experience the dialogue with a machine that thinks neurally, similarly to them.