CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing

TL;DR

CustomSketching introduces a novel framework for extracting sketch concepts from image-sketch pairs, enabling fine-grained, sketch-based image synthesis and editing, thus providing more control than traditional text-based personalization methods.

Contribution

The paper proposes a dual-sketch representation and a two-stage framework for effective sketch concept extraction, enhancing control over image synthesis and editing.

Findings

Outperforms adapted baselines in experiments

Effective in fine-grained image editing

User study confirms superiority

Abstract

Personalization techniques for large text-to-image (T2I) models allow users to incorporate new concepts from reference images. However, existing methods primarily rely on textual descriptions, leading to limited control over customized images and failing to support fine-grained and local editing (e.g., shape, pose, and details). In this paper, we identify sketches as an intuitive and versatile representation that can facilitate such control, e.g., contour lines capturing shape information and flow lines representing texture. This motivates us to explore a novel task of sketch concept extraction: given one or more sketch-image pairs, we aim to extract a special sketch concept that bridges the correspondence between the images and sketches, thus enabling sketch-based image synthesis and editing at a fine-grained level. To accomplish this, we introduce CustomSketching, a two-stage framework for extracting novel sketch concepts. Considering that an object can often be depicted by a contour for general shapes and additional strokes for internal details, we introduce a dual-sketch representation to reduce the inherent ambiguity in sketch depiction. We employ a shape loss and a regularization loss to balance fidelity and editability during optimization. Through extensive experiments, a user study, and several applications, we show our method is effective and superior to the adapted baselines.