Filtering Video Noise as Audio with Motion Detection to Form a Musical Instrument
Carl Thom\'e

TL;DR
This paper introduces a novel musical instrument that synthesizes harmonic sounds from video signals using motion detection, serving both as a sound source and a controller interface, creating a unique audiovisual experience.
Contribution
It proposes a new method for video-based sound synthesis and control, bridging visual motion detection with musical sound production in a creative way.
Findings
Produces pleasant and interesting sounds from video input
Enables video-based note selection and velocity control
Suitable for synth pad sound generation
Abstract
Even though they differ in the physical domain, digital video and audio share many characteristics. Both are temporal data streams often stored in buffers with 8-bit values. This paper investigates a method for creating harmonic sounds with a video signal as input. A musical instrument is proposed, that utilizes video in both a sound synthesis method, and in a controller interface for selecting musical notes at specific velocities. The resulting instrument was informally determined by the author to sound both pleasant and interesting, but hard to control, and therefore suited for synth pad sounds.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTactile and Sensory Interactions · Multisensory perception and integration · Music Technology and Sound Studies
