One-shot Face Sketch Synthesis in the Wild via Generative Diffusion Prior and Instruction Tuning

TL;DR

This paper introduces a one-shot face sketch synthesis method using diffusion models and instruction tuning, capable of generating realistic sketches from a single photo without extensive training data.

Contribution

The authors propose a novel one-shot face sketch synthesis approach leveraging diffusion models and instruction tuning, along with a new benchmark dataset for evaluation.

Findings

Effective one-shot face sketch synthesis demonstrated

Outperforms existing methods in realism and consistency

New benchmark dataset facilitates comprehensive evaluation

Abstract

Face sketch synthesis is a technique aimed at converting face photos into sketches. Existing face sketch synthesis research mainly relies on training with numerous photo-sketch sample pairs from existing datasets. However, these large-scale discriminative learning methods will have to face problems such as data scarcity and high human labor costs. Once the training data becomes scarce, their generative performance significantly degrades. In this paper, we propose a one-shot face sketch synthesis method based on diffusion models. We optimize text instructions on a diffusion model using face photo-sketch image pairs. Then, the instructions derived through gradient-based optimization are used for inference. To simulate real-world scenarios more accurately and evaluate method effectiveness more comprehensively, we introduce a new benchmark named One-shot Face Sketch Dataset (OS-Sketch). The benchmark consists of 400 pairs of face photo-sketch images, including sketches with different styles and photos with different backgrounds, ages, sexes, expressions, illumination, etc. For a solid out-of-distribution evaluation, we select only one pair of images for training at each time, with the rest used for inference. Extensive experiments demonstrate that the proposed method can convert various photos into realistic and highly consistent sketches in a one-shot context. Compared to other methods, our approach offers greater convenience and broader applicability. The dataset will be available at: https://github.com/HanWu3125/OS-Sketch