Inverse Painting: Reconstructing The Painting Process

TL;DR

This paper introduces a novel autoregressive diffusion-based method to reconstruct a plausible painting process from a single image, leveraging learned instructions and region understanding, capable of generalizing across diverse artistic styles.

Contribution

It presents a new approach combining autoregressive modeling and diffusion rendering to reconstruct painting sequences from static images, extending beyond training styles.

Findings

Successfully reconstructs painting processes from static images.

Generalizes to various artistic styles and genres.

Produces plausible time-lapse videos of painting creation.

Abstract

Given an input painting, we reconstruct a time-lapse video of how it may have been painted. We formulate this as an autoregressive image generation problem, in which an initially blank "canvas" is iteratively updated. The model learns from real artists by training on many painting videos. Our approach incorporates text and region understanding to define a set of painting "instructions" and updates the canvas with a novel diffusion-based renderer. The method extrapolates beyond the limited, acrylic style paintings on which it has been trained, showing plausible results for a wide range of artistic styles and genres.