ProcessPainter: Learn Painting Process from Sequence Data

TL;DR

ProcessPainter is a novel model that learns to generate detailed, step-by-step painting processes from text prompts, enabling more authentic art creation and educational tools.

Contribution

The paper introduces ProcessPainter, a text-to-video model trained on synthetic and artist data, capable of generating and controlling painting sequences from text prompts.

Findings

First model to generate painting processes from text prompts

Able to decompose images into painting sequences

Supports controlled generation and artwork completion

Abstract

The painting process of artists is inherently stepwise and varies significantly among different painters and styles. Generating detailed, step-by-step painting processes is essential for art education and research, yet remains largely underexplored. Traditional stroke-based rendering methods break down images into sequences of brushstrokes, yet they fall short of replicating the authentic processes of artists, with limitations confined to basic brushstroke modifications. Text-to-image models utilizing diffusion processes generate images through iterative denoising, also diverge substantially from artists' painting process. To address these challenges, we introduce ProcessPainter, a text-to-video model that is initially pre-trained on synthetic data and subsequently fine-tuned with a select set of artists' painting sequences using the LoRA model. This approach successfully generates painting processes from text prompts for the first time. Furthermore, we introduce an Artwork Replication Network capable of accepting arbitrary-frame input, which facilitates the controlled generation of painting processes, decomposing images into painting sequences, and completing semi-finished artworks. This paper offers new perspectives and tools for advancing art education and image generation technology.