SketchTriplet: Self-Supervised Scenarized Sketch-Text-Image Triplet Generation

TL;DR

This paper introduces a self-supervised approach for scene sketch generation from single-object sketches, creating a large-scale dataset of text-sketch-image triplets to improve sketch-based tasks.

Contribution

It presents a novel self-supervised method for scene sketch generation and a large-scale dataset of triplets, advancing scene sketch understanding without relying on existing scene sketches.

Findings

Achieved state-of-the-art performance in zero-shot image-to-sketch tasks

Created a large-scale, semantically consistent scene sketch dataset

Enhanced sketch-based image retrieval and synthesis capabilities

Abstract

The scarcity of free-hand sketch presents a challenging problem. Despite the emergence of some large-scale sketch datasets, these datasets primarily consist of sketches at the single-object level. There continues to be a lack of large-scale paired datasets for scene sketches. In this paper, we propose a self-supervised method for scene sketch generation that does not rely on any existing scene sketch, enabling the transformation of single-object sketches into scene sketches. To accomplish this, we introduce a method for vector sketch captioning and sketch semantic expansion. Additionally, we design a sketch generation network that incorporates a fusion of multi-modal perceptual constraints, suitable for application in zero-shot image-to-sketch downstream task, demonstrating state-of-the-art performance through experimental validation. Finally, leveraging our proposed sketch-to-sketch generation method, we contribute a large-scale dataset centered around scene sketches, comprising highly semantically consistent "text-sketch-image" triplets. Our research confirms that this dataset can significantly enhance the capabilities of existing models in sketch-based image retrieval and sketch-controlled image synthesis tasks. We will make our dataset and code publicly available.