Stroke-based sketched symbol reconstruction and segmentation

TL;DR

This paper introduces a neural network-based framework for stroke-level segmentation of hand-drawn symbols, utilizing a novel stroke-based VAE encoder, achieving superior accuracy on multiple datasets and aiding in stylization and animation tasks.

Contribution

The paper presents a new stroke-based VAE encoder and a segmentation framework that generalizes across categories, improving accuracy over existing methods.

Findings

Surpasses existing segmentation methods on a small dataset.

Achieves significantly better accuracy on a large, newly annotated dataset.

Reuses a single encoder across multiple symbol categories with negligible accuracy loss.

Abstract

Hand-drawn objects usually consist of multiple semantically meaningful parts. For example, a stick figure consists of a head, a torso, and pairs of legs and arms. Efficient and accurate identification of these subparts promises to significantly improve algorithms for stylization, deformation, morphing and animation of 2D drawings. In this paper, we propose a neural network model that segments symbols into stroke-level components. Our segmentation framework has two main elements: a fixed feature extractor and a Multilayer Perceptron (MLP) network that identifies a component based on the feature. As the feature extractor we utilize an encoder of a stroke-rnn, which is our newly proposed generative Variational Auto-Encoder (VAE) model that reconstructs symbols on a stroke by stroke basis. Experiments show that a single encoder could be reused for segmenting multiple categories of sketched symbols with negligible effects on segmentation accuracies. Our segmentation scores surpass existing methodologies on an available small state of the art dataset. Moreover, extensive evaluations on our newly annotated big dataset demonstrate that our framework obtains significantly better accuracies as compared to baseline models. We release the dataset to the community.