FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context

TL;DR

This paper introduces FS-COCO, a new dataset of 10,000 freehand scene sketches with descriptions, enabling research on scene understanding, image retrieval, and multimodal analysis with practical applications.

Contribution

The paper presents the first dataset of freehand scene sketches with annotations and explores novel tasks like fine-grained image retrieval and sketch-caption analysis.

Findings

Sketch strokes encode scene salience via temporal order.

Image retrieval performance varies between sketches and captions.

Combining sketches and captions improves retrieval accuracy.

Abstract

We advance sketch research to scenes with the first dataset of freehand scene sketches, FS-COCO. With practical applications in mind, we collect sketches that convey scene content well but can be sketched within a few minutes by a person with any sketching skills. Our dataset comprises 10,000 freehand scene vector sketches with per point space-time information by 100 non-expert individuals, offering both object- and scene-level abstraction. Each sketch is augmented with its text description. Using our dataset, we study for the first time the problem of fine-grained image retrieval from freehand scene sketches and sketch captions. We draw insights on: (i) Scene salience encoded in sketches using the strokes temporal order; (ii) Performance comparison of image retrieval from a scene sketch and an image caption; (iii) Complementarity of information in sketches and image captions, as well as the potential benefit of combining the two modalities. In addition, we extend a popular vector sketch LSTM-based encoder to handle sketches with larger complexity than was supported by previous work. Namely, we propose a hierarchical sketch decoder, which we leverage at a sketch-specific "pre-text" task. Our dataset enables for the first time research on freehand scene sketch understanding and its practical applications.