Learning Deep Sketch Abstraction

TL;DR

This paper introduces a stroke-level sketch abstraction model using reinforcement learning to balance recognizability and stroke reduction, enhancing sketch analysis and synthesis tasks.

Contribution

It presents the first stroke-level abstraction model that predicts removable strokes, improving sketch recognition, synthesis, and retrieval without extensive paired data.

Findings

Effective stroke removal policy learned via reinforcement learning

Model improves sketch recognition and synthesis tasks

Enables training FG-SBIR models without photo-sketch pairs

Abstract

Human free-hand sketches have been studied in various contexts including sketch recognition, synthesis and fine-grained sketch-based image retrieval (FG-SBIR). A fundamental challenge for sketch analysis is to deal with drastically different human drawing styles, particularly in terms of abstraction level. In this work, we propose the first stroke-level sketch abstraction model based on the insight of sketch abstraction as a process of trading off between the recognizability of a sketch and the number of strokes used to draw it. Concretely, we train a model for abstract sketch generation through reinforcement learning of a stroke removal policy that learns to predict which strokes can be safely removed without affecting recognizability. We show that our abstraction model can be used for various sketch analysis tasks including: (1) modeling stroke saliency and understanding the decision of sketch recognition models, (2) synthesizing sketches of variable abstraction for a given category, or reference object instance in a photo, and (3) training a FG-SBIR model with photos only, bypassing the expensive photo-sketch pair collection step.