An Artistic Visualization of Music Modeling a Synesthetic Experience

TL;DR

This paper presents a visual simulation of synesthetic experiences of music, using digital signal processing and particle physics to create real-time visualizations that reflect auditory stimuli.

Contribution

It introduces a novel method combining audio analysis and particle simulation to visually represent synesthetic experiences of music.

Findings

Successfully visualizes music as colors and particles in real-time

Integrates digital signal processing with physics simulation for artistic visualization

Provides a new tool for exploring synesthetic perception

Abstract

This project brings music to sight. Music can be a visual masterpiece. Some people naturally experience a visualization of audio - a condition called synesthesia. The type of synesthesia explored is when sounds create colors in the 'mind's eye.' Project included interviews with people who experience synesthesia, examination of prior art, and topic research to inform project design. Audio input, digital signal processing (including Fast Fourier Transforms (FFTs)) and data manipulation produce arguments required for our visualization. Arguments are then applied to a physics particle simulator which is re-purposed to model a synesthetic experience. The result of the project is a simulator in MAX 8, which generates a visual performance using particles by varying each particle's position, velocity, and color based on parameters extracted via digital processing of input audio.