Filtering Video Noise as Audio with Motion Detection to Form a Musical Instrument

TL;DR

This paper introduces a novel musical instrument that synthesizes harmonic sounds from video signals using motion detection, serving both as a sound source and a controller interface, creating a unique audiovisual experience.

Contribution

It proposes a new method for video-based sound synthesis and control, bridging visual motion detection with musical sound production in a creative way.

Findings

Produces pleasant and interesting sounds from video input

Enables video-based note selection and velocity control

Suitable for synth pad sound generation

Abstract

Even though they differ in the physical domain, digital video and audio share many characteristics. Both are temporal data streams often stored in buffers with 8-bit values. This paper investigates a method for creating harmonic sounds with a video signal as input. A musical instrument is proposed, that utilizes video in both a sound synthesis method, and in a controller interface for selecting musical notes at specific velocities. The resulting instrument was informally determined by the author to sound both pleasant and interesting, but hard to control, and therefore suited for synth pad sounds.