Neural Painters: A learned differentiable constraint for generating brushstroke paintings

TL;DR

This paper introduces neural painters, a differentiable model for brushstroke generation that accelerates training, enables human-like stroke reconstruction, visualizes image categories, and facilitates intrinsic style transfer in digital art.

Contribution

It presents a novel differentiable neural painter model learned from real painting programs, enhancing image synthesis, visualization, and artistic style transfer capabilities.

Findings

Faster convergence in training brushstroke-based image generation.

Ability to reconstruct digits with human-like strokes.

Visualization of ImageNet categories through optimized brushstrokes.

Abstract

We explore neural painters, a generative model for brushstrokes learned from a real non-differentiable and non-deterministic painting program. We show that when training an agent to "paint" images using brushstrokes, using a differentiable neural painter leads to much faster convergence. We propose a method for encouraging this agent to follow human-like strokes when reconstructing digits. We also explore the use of a neural painter as a differentiable image parameterization. By directly optimizing brushstrokes to activate neurons in a pre-trained convolutional network, we can directly visualize ImageNet categories and generate "ideal" paintings of each class. Finally, we present a new concept called intrinsic style transfer. By minimizing only the content loss from neural style transfer, we allow the artistic medium, in this case, brushstrokes, to naturally dictate the resulting style.