Deep Sketch-Based Modeling: Tips and Tricks

TL;DR

This paper explores the unique challenges of sketch-based modeling compared to image-based methods, identifying key differences and proposing solutions to improve deep learning approaches for sketch inputs.

Contribution

The work systematically analyzes the differences between sketch and image inputs and demonstrates how to adapt existing deep modeling methods to better handle sketches.

Findings

Sparsity leads to foreground-background prediction errors.

Style diversity affects model generalization.

Perspective mismatch complicates sketch modeling.

Abstract

Deep image-based modeling received lots of attention in recent years, yet the parallel problem of sketch-based modeling has only been briefly studied, often as a potential application. In this work, for the first time, we identify the main differences between sketch and image inputs: (i) style variance, (ii) imprecise perspective, and (iii) sparsity. We discuss why each of these differences can pose a challenge, and even make a certain class of image-based methods inapplicable. We study alternative solutions to address each of the difference. By doing so, we drive out a few important insights: (i) sparsity commonly results in an incorrect prediction of foreground versus background, (ii) diversity of human styles, if not taken into account, can lead to very poor generalization properties, and finally (iii) unless a dedicated sketching interface is used, one can not expect sketches to match a perspective of a fixed viewpoint. Finally, we compare a set of representative deep single-image modeling solutions and show how their performance can be improved to tackle sketch input by taking into consideration the identified critical differences.