Fast Sketch Segmentation and Labeling with Deep Learning

TL;DR

This paper introduces a deep learning approach for fast, accurate sketch segmentation and labeling that transfers knowledge from 3D models, enabling real-time applications like sketch-based 3D modeling.

Contribution

The method transfers segmentation knowledge from 3D models to sketches without extensive annotated data, improving speed and accuracy over previous techniques.

Findings

Outperforms state-of-the-art in accuracy

Significantly faster segmentation process

Enables real-time sketch-based modeling

Abstract

We present a simple and efficient method based on deep learning to automatically decompose sketched objects into semantically valid parts. We train a deep neural network to transfer existing segmentations and labelings from 3D models to freehand sketches without requiring numerous well-annotated sketches as training data. The network takes the binary image of a sketched object as input and produces a corresponding segmentation map with per-pixel labelings as output. A subsequent post-process procedure with multi-label graph cuts further refines the segmentation and labeling result. We validate our proposed method on two sketch datasets. Experiments show that our method outperforms the state-of-the-art method in terms of segmentation and labeling accuracy and is significantly faster, enabling further integration in interactive drawing systems. We demonstrate the efficiency of our method in a sketch-based modeling application that automatically transforms input sketches into 3D models by part assembly.