NeoLightning: A Modern Reimagination of Gesture-Based Sound Design

TL;DR

NeoLightning modernizes gesture-based sound design by integrating deep learning gesture recognition with real-time multimedia processing, making interactive musical instruments more accessible and expressive for contemporary users.

Contribution

It introduces a modern reinterpretation of the Buchla Lightning, utilizing MediaPipe and contemporary software for precise, low-latency gesture recognition and immersive 3D interaction.

Findings

Achieves precise, low-latency gesture recognition

Enables immersive 3D interaction for sound design

Expands expressive possibilities in gesture-based musical instruments

Abstract

This paper introduces NeoLightning, a modern reinterpretation of the Buchla Lightning. NeoLightning preserves the innovative spirit of Don Buchla's "Buchla Lightning" (introduced in the 1990s) while making its gesture-based interaction accessible to contemporary users. While the original Buchla Lightning and many other historical instruments were groundbreaking in their time, they are now largely unsupported, limiting user interaction to indirect experiences. To address this, NeoLightning leverages MediaPipe for deep learning-based gesture recognition and employs Max/MSP and Processing for real-time multimedia processing. The redesigned system offers precise, low-latency gesture recognition and immersive 3D interaction. By merging the creative spirit of the original Lightning with modern advancements, NeoLightning redefines gesture-based musical interaction, expanding possibilities for expressive performance and interactive sound design.