SketchyScene: Richly-Annotated Scene Sketches

TL;DR

SketchyScene is a large-scale, richly-annotated dataset of scene sketches designed to advance research in sketch understanding, enabling new models and applications such as segmentation, retrieval, and editing.

Contribution

The paper introduces SketchyScene, the first large-scale dataset of scene sketches with detailed annotations, created through a novel crowdsourcing pipeline.

Findings

Enabled training of new semantic segmentation models for scene sketches.

Facilitated applications like image retrieval, sketch colorization, editing, and captioning.

Demonstrated the dataset's scalability and extensibility.

Abstract

We contribute the first large-scale dataset of scene sketches, SketchyScene, with the goal of advancing research on sketch understanding at both the object and scene level. The dataset is created through a novel and carefully designed crowdsourcing pipeline, enabling users to efficiently generate large quantities of realistic and diverse scene sketches. SketchyScene contains more than 29,000 scene-level sketches, 7,000+ pairs of scene templates and photos, and 11,000+ object sketches. All objects in the scene sketches have ground-truth semantic and instance masks. The dataset is also highly scalable and extensible, easily allowing augmenting and/or changing scene composition. We demonstrate the potential impact of SketchyScene by training new computational models for semantic segmentation of scene sketches and showing how the new dataset enables several applications including image retrieval, sketch colorization, editing, and captioning, etc. The dataset and code can be found at https://github.com/SketchyScene/SketchyScene.