Instance Segmentation of Scene Sketches Using Natural Image Priors

TL;DR

This paper presents InkLayer, a novel method for instance segmentation of scene sketches that adapts image segmentation models with style-robust fine-tuning and depth-based mask refinement, enabling advanced sketch editing.

Contribution

It introduces InkLayer, a new approach that adapts image segmentation models to sketches using class-agnostic fine-tuning and depth cues, and creates InkScenes, a diverse synthetic dataset for training and evaluation.

Findings

InkLayer effectively segments diverse scene sketches.

The synthetic InkScenes dataset enhances model robustness.

The method enables advanced sketch editing applications.

Abstract

Sketch segmentation involves grouping pixels within a sketch that belong to the same object or instance. It serves as a valuable tool for sketch editing tasks, such as moving, scaling, or removing specific components. While image segmentation models have demonstrated remarkable capabilities in recent years, sketches present unique challenges for these models due to their sparse nature and wide variation in styles. We introduce InkLayer, a method for instance segmentation of raster scene sketches. Our approach adapts state-of-the-art image segmentation and object detection models to the sketch domain by employing class-agnostic fine-tuning and refining segmentation masks using depth cues. Furthermore, our method organizes sketches into sorted layers, where occluded instances are inpainted, enabling advanced sketch editing applications. As existing datasets in this domain lack variation in sketch styles, we construct a synthetic scene sketch segmentation dataset, InkScenes, featuring sketches with diverse brush strokes and varying levels of detail. We use this dataset to demonstrate the robustness of our approach.