An Artistic Visualization of Music Modeling a Synesthetic Experience
Matthew Joseph Adiletta, Oliver Thomas

TL;DR
This paper presents a visual simulation of synesthetic experiences of music, using digital signal processing and particle physics to create real-time visualizations that reflect auditory stimuli.
Contribution
It introduces a novel method combining audio analysis and particle simulation to visually represent synesthetic experiences of music.
Findings
Successfully visualizes music as colors and particles in real-time
Integrates digital signal processing with physics simulation for artistic visualization
Provides a new tool for exploring synesthetic perception
Abstract
This project brings music to sight. Music can be a visual masterpiece. Some people naturally experience a visualization of audio - a condition called synesthesia. The type of synesthesia explored is when sounds create colors in the 'mind's eye.' Project included interviews with people who experience synesthesia, examination of prior art, and topic research to inform project design. Audio input, digital signal processing (including Fast Fourier Transforms (FFTs)) and data manipulation produce arguments required for our visualization. Arguments are then applied to a physics particle simulator which is re-purposed to model a synesthetic experience. The result of the project is a simulator in MAX 8, which generates a visual performance using particles by varying each particle's position, velocity, and color based on parameters extracted via digital processing of input audio.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsColor perception and design · Multisensory perception and integration · Sensory Analysis and Statistical Methods
