Data-Efficient Brushstroke Generation with Diffusion Models for Oil Painting

TL;DR

This paper introduces StrokeDiff, a diffusion-based framework that learns to generate human-like brushstrokes from limited data, enabling expressive oil painting synthesis with controllability and structural coherence.

Contribution

The work presents a novel data-efficient diffusion model with regularization for brushstroke generation, and integrates it into a complete painting pipeline with controllable primitives.

Findings

Produces diverse, coherent brushstrokes

Enables richer textured paintings

Validated by automatic metrics and human evaluation

Abstract

Many creative multimedia systems are built upon visual primitives such as strokes or textures, which are difficult to collect at scale and fundamentally different from natural image data. This data scarcity makes it challenging for modern generative models to learn expressive and controllable primitives, limiting their use in process-aware content creation. In this work, we study the problem of learning human-like brushstroke generation from a small set of hand-drawn samples (n=470) and propose StrokeDiff, a diffusion-based framework with Smooth Regularization (SmR). SmR injects stochastic visual priors during training, providing a simple mechanism to stabilize diffusion models under sparse supervision without altering the inference process. We further show how the learned primitives can be made controllable through a B\'ezier-based conditioning module and integrated into a complete stroke-based painting pipeline, including prediction, generation, ordering, and compositing. This demonstrates how data-efficient primitive modeling can support expressive and structured multimedia content creation. Experiments indicate that the proposed approach produces diverse and structurally coherent brushstrokes and enables paintings with richer texture and layering, validated by both automatic metrics and human evaluation.