From pixels to notes: a computational implementation of synaesthesia for cultural artefacts

TL;DR

This paper presents a Python-based simulation of synaesthesia that generates melodies from paintings, mimicking how some individuals perceive colors as sounds, based on Scriabin's definition.

Contribution

It introduces a deterministic computational model translating visual art into music, bridging sensory modalities in a novel way.

Findings

Successfully generates melodies from paintings

Implements a deterministic process based on Scriabin's ideas

Provides a tool for exploring synaesthetic perception

Abstract

Synaesthesia is a condition that enables people to sense information in the form of several senses at once. This work describes a Python implementation of a simulation of synaesthesia between listening to music and viewing a painting. Based on Scriabin's definition, we developed a deterministic process to produce a melody after processing a painting, mimicking the production of notes from colours in the field of view of persons experiencing synaesthesia.