PaintCopilot: Modeling Painting as Autonomous Artistic Continuation

TL;DR

PaintCopilot is a neural painting assistant that models artistic creation as an open-ended, autoregressive process, enabling fluid co-creative workflows without predefined target images.

Contribution

It introduces a novel framework with three models that predict and generate brushstrokes based on artistic dynamics, supporting interactive and autonomous painting workflows.

Findings

Enables fluid co-creative painting with professional artists.

Supports multiple interactive workflows like inpainting and stroke completion.

Predicts artist intent from partial canvas observations.

Abstract

We present PaintCopilot, a co-creative neural painting assistant that models painting as an open-ended autoregressive artistic behavior conditioned on evolving canvas states and prior brushstroke history, without requiring a target image. Unlike existing neural painting methods that frame painting as pixel reconstruction toward a predefined reference, PaintCopilot predicts future strokes directly from learned artistic dynamics, analogous to how large language models continue text sequences from prior context. The framework proposes three complementary models: a ViT-based Target Predictor that infers artist intent from partial canvas observations, an autoregressive Next Stroke Predictor that generates temporally coherent brushstrokes via flow matching, and a VAE-based Region Sampler that synthesizes semantically localized stroke sequences on demand. Built on three differentiable brush representations (Hard Round, Brush Tip, and 2D Gaussian), the system supports four interactive workflows: Optimize History, Stroke Completion, Region Inpainting, and Dynamic Brush. Through case studies with professional artists, we demonstrate that PaintCopilot enables fluid co-creative painting workflows in which artists and AI continuously alternate control throughout the creative process.