Instance Segmentation of Scene Sketches Using Natural Image Priors
Mia Tang, Yael Vinker, Chuan Yan, Lvmin Zhang, Maneesh Agrawala

TL;DR
This paper presents InkLayer, a novel method for instance segmentation of scene sketches that adapts image segmentation models with style-robust fine-tuning and depth-based mask refinement, enabling advanced sketch editing.
Contribution
It introduces InkLayer, a new approach that adapts image segmentation models to sketches using class-agnostic fine-tuning and depth cues, and creates InkScenes, a diverse synthetic dataset for training and evaluation.
Findings
InkLayer effectively segments diverse scene sketches.
The synthetic InkScenes dataset enhances model robustness.
The method enables advanced sketch editing applications.
Abstract
Sketch segmentation involves grouping pixels within a sketch that belong to the same object or instance. It serves as a valuable tool for sketch editing tasks, such as moving, scaling, or removing specific components. While image segmentation models have demonstrated remarkable capabilities in recent years, sketches present unique challenges for these models due to their sparse nature and wide variation in styles. We introduce InkLayer, a method for instance segmentation of raster scene sketches. Our approach adapts state-of-the-art image segmentation and object detection models to the sketch domain by employing class-agnostic fine-tuning and refining segmentation masks using depth cues. Furthermore, our method organizes sketches into sorted layers, where occluded instances are inpainted, enabling advanced sketch editing applications. As existing datasets in this domain lack variation…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Vision and Imaging · Computer Graphics and Visualization Techniques · Generative Adversarial Networks and Image Synthesis
