SnakeSynth: New Interactions for Generative Audio Synthesis

TL;DR

SnakeSynth is a browser-based interactive audio synthesizer that combines deep generative models with 2D gesture controls for real-time sound creation and modulation, enabling expressive musical interactions without extensive training.

Contribution

It introduces a lightweight, web-based interface integrating deep generative audio models with intuitive 2D gesture controls for real-time sound synthesis.

Findings

Real-time high-fidelity sound generation in browser

Interactive control of sound length and intensity

Adaptive sound interpolation during training

Abstract

I present "SnakeSynth," a web-based lightweight audio synthesizer that combines audio generated by a deep generative model and real-time continuous two-dimensional (2D) input to create and control variable-length generative sounds through 2D interaction gestures. Interaction gestures are touch and mobile-compatible with analogies to strummed, bowed, and plucked musical instrument controls. Point-and-click and drag-and-drop gestures directly control audio playback length and I show that sound length and intensity are modulated by interactions with a programmable 2D coordinate grid. Leveraging the speed and ubiquity of browser-based audio and hardware acceleration in Google's TensorFlow.js we generate time-varying high-fidelity sounds with real-time interactivity. SnakeSynth adaptively reproduces and interpolates between sounds encountered during model training, notably without long training times, and I briefly discuss possible futures for deep generative models as an interactive paradigm for musical expression.