VideoSketcher: Video Models Prior Enable Versatile Sequential Sketch Generation

TL;DR

This paper introduces a data-efficient method for generating sequential sketches by adapting pretrained text-to-video diffusion models, combining semantic stroke planning from language models with high-quality visual rendering.

Contribution

It presents a novel two-stage fine-tuning approach that leverages limited human sketch data and synthetic shapes to produce controllable, high-quality sequential sketches guided by text instructions.

Findings

High-quality sketches closely follow text-specified orderings

Method works with as few as seven human-drawn sketches

Extensions enable style conditioning and interactive drawing

Abstract

Sketching is inherently a sequential process, in which strokes are drawn in a meaningful order to explore and refine ideas. However, most generative models treat sketches as static images, overlooking the temporal structure that underlies creative drawing. We present a data-efficient approach for sequential sketch generation that adapts pretrained text-to-video diffusion models to generate sketching processes. Our key insight is that large language models and video diffusion models offer complementary strengths for this task: LLMs provide semantic planning and stroke ordering, while video diffusion models serve as strong renderers that produce high-quality, temporally coherent visuals. We leverage this by representing sketches as short videos in which strokes are progressively drawn on a blank canvas, guided by text-specified ordering instructions. We introduce a two-stage fine-tuning strategy that decouples the learning of stroke ordering from the learning of sketch appearance. Stroke ordering is learned using synthetic shape compositions with controlled temporal structure, while visual appearance is distilled from as few as seven manually authored sketching processes that capture both global drawing order and the continuous formation of individual strokes. Despite the extremely limited amount of human-drawn sketch data, our method generates high-quality sequential sketches that closely follow text-specified orderings while exhibiting rich visual detail. We further demonstrate the flexibility of our approach through extensions such as brush style conditioning and autoregressive sketch generation, enabling additional controllability and interactive, collaborative drawing.