Bridging Paintings and Music -- Exploring Emotion based Music Generation through Paintings

TL;DR

This paper presents a novel AI model that generates emotionally resonant music from paintings by converting visual art into descriptive text and then into musical compositions, supported by a new paired dataset.

Contribution

The study introduces a dual-stage framework and the Emotion Painting Music Dataset, enabling effective music generation from visual art with minimal data.

Findings

High alignment between generated music and emotional descriptions confirmed by CLAP

Effective music generation demonstrated using FAD, THD, IS, and KL metrics

Enhanced accessibility and multi-sensory experiences for visually impaired users

Abstract

Rapid advancements in artificial intelligence have significantly enhanced generative tasks involving music and images, employing both unimodal and multimodal approaches. This research develops a model capable of generating music that resonates with the emotions depicted in visual arts, integrating emotion labeling, image captioning, and language models to transform visual inputs into musical compositions. Addressing the scarcity of aligned art and music data, we curated the Emotion Painting Music Dataset, pairing paintings with corresponding music for effective training and evaluation. Our dual-stage framework converts images to text descriptions of emotional content and then transforms these descriptions into music, facilitating efficient learning with minimal data. Performance is evaluated using metrics such as Fr\'echet Audio Distance (FAD), Total Harmonic Distortion (THD), Inception Score (IS), and KL divergence, with audio-emotion text similarity confirmed by the pre-trained CLAP model to demonstrate high alignment between generated music and text. This synthesis tool bridges visual art and music, enhancing accessibility for the visually impaired and opening avenues in educational and therapeutic applications by providing enriched multi-sensory experiences.