Multimodal computational modeling of EEG and artistic painting for exploring the stress-relief mechanism of urban green spaces

TL;DR

This paper introduces ArtCLIP, a new framework that uses multimodal learning to improve dynamic freehand painting with real-time text-based control.

Contribution

ArtCLIP integrates CLIP with an attention fusion mechanism to enable responsive brushwork and style adaptation in digital art creation.

Findings

ArtCLIP outperforms baseline models in real-time artistic rendering tasks.

The framework shows enhanced adaptability to different artistic styles and prompt alignment.

It offers a more interpretable and interactive AI-assisted creative process.

Abstract

Multimodal learning has recently opened new possibilities for integrating semantic understanding into creative domains, and models like Contrastive Language-Image Pretraining (CLIP) provide a compelling foundation for bridging text-image relationships in artistic applications. However, while CLIP demonstrates exceptional capabilities in image-text alignment, its application in dynamic, creative tasks such as freehand brushwork painting remains underexplored. Traditional methods for generating artwork using neural networks often rely on static image generation techniques, which struggle to capture the fluidity and dynamism of brushstrokes in real-time creative processes. These approaches frequently lack the interpretive flexibility required to respond to real-time textual prompts with spontaneous, expressive outcomes.

To address this, we propose ArtCLIP, a novel framework that integrates CLIP with an attention fusion mechanism to facilitate dynamic freehand brushwork painting. Our method utilizes CLIP's ability to interpret textual descriptions and visual cues in tandem with an attention-based fusion model, which enables the system to modulate brushstrokes responsively and adjust painting styles dynamically based on evolving inputs.

We conduct extensive experiments demonstrating that ArtCLIP achieves significant improvements in real-time artistic rendering tasks compared to baseline models. The results show enhanced adaptability to varying artistic styles and better alignment with descriptive prompts, offering a promising avenue for digital art creation. By enabling semantically driven and stylistically controllable painting generation, our approach contributes to a more interpretable and interactive form of AI-assisted creativity.

Figures