Birth of a Painting: Differentiable Brushstroke Reconstruction

TL;DR

This paper introduces a differentiable framework for reconstructing and stylizing brushstrokes in paintings, enabling realistic, expressive digital painting creation by jointly optimizing stroke geometry, color, and texture.

Contribution

It presents a novel differentiable stroke reconstruction method that unifies painting, stylized texturing, and smudging, capturing the human painting process more faithfully than prior models.

Findings

Produces realistic stroke reconstructions across various painting styles.

Generates smooth tonal transitions and richly stylized appearances.

Effectively models the human painting-smudging loop for expressive digital art.

Abstract

Painting embodies a unique form of visual storytelling, where the creation process is as significant as the final artwork. Although recent advances in generative models have enabled visually compelling painting synthesis, most existing methods focus solely on final image generation or patch-based process simulation, lacking explicit stroke structure and failing to produce smooth, realistic shading. In this work, we present a differentiable stroke reconstruction framework that unifies painting, stylized texturing, and smudging to faithfully reproduce the human painting-smudging loop. Given an input image, our framework first optimizes single- and dual-color Bezier strokes through a parallel differentiable paint renderer, followed by a style generation module that synthesizes geometry-conditioned textures across diverse painting styles. We further introduce a differentiable smudge operator to enable natural color blending and shading. Coupled with a coarse-to-fine optimization strategy, our method jointly optimizes stroke geometry, color, and texture under geometric and semantic guidance. Extensive experiments on oil, watercolor, ink, and digital paintings demonstrate that our approach produces realistic and expressive stroke reconstructions, smooth tonal transitions, and richly stylized appearances, offering a unified model for expressive digital painting creation. See our project page for more demos: https://yingjiang96.github.io/DiffPaintWebsite/.