Interactive Neural Painting

TL;DR

This paper introduces I-Paint, an interactive neural painting framework that suggests next strokes to assist users in creating artworks, utilizing a novel transformer-based VAE architecture and new datasets.

Contribution

It presents the first interactive neural painting approach with a transformer-based VAE, and introduces two datasets for evaluation.

Findings

Provides effective stroke suggestions for users

Outperforms existing methods in stroke prediction accuracy

Demonstrates the utility of the proposed framework in creative tasks

Abstract

In the last few years, Neural Painting (NP) techniques became capable of producing extremely realistic artworks. This paper advances the state of the art in this emerging research domain by proposing the first approach for Interactive NP. Considering a setting where a user looks at a scene and tries to reproduce it on a painting, our objective is to develop a computational framework to assist the users creativity by suggesting the next strokes to paint, that can be possibly used to complete the artwork. To accomplish such a task, we propose I-Paint, a novel method based on a conditional transformer Variational AutoEncoder (VAE) architecture with a two-stage decoder. To evaluate the proposed approach and stimulate research in this area, we also introduce two novel datasets. Our experiments show that our approach provides good stroke suggestions and compares favorably to the state of the art. Additional details, code and examples are available at https://helia95.github.io/inp-website.