Exploring Real-Time Music-to-Image Systems for Creative Inspiration in Music Creation

TL;DR

This study explores a real-time music-to-image system that uses AI to generate visual imagery from MIDI input, aiming to inspire musicians during their creative process.

Contribution

It introduces a novel real-time system that interprets MIDI input into visual images based on emotion and structure, supporting creative inspiration.

Findings

Most musicians found the generated images inspiring.

The system effectively supported improvisation and composition.

Generated images provided a new mechanism for creative stimulation.

Abstract

This paper presents a study on the use of a real-time music-to-image system as a mechanism to support and inspire musicians during their creative process. The system takes MIDI messages from a keyboard as input which are then interpreted and analysed using state-of-the-art generative AI models. Based on the perceived emotion and music structure, the system's interpretation is converted into visual imagery that is presented in real-time to musicians. We conducted a user study in which musicians improvised and composed using the system. Our findings show that most musicians found the generated images were a novel mechanism when playing, evidencing the potential of music-to-image systems to inspire and enhance their creative process.