DiffSketching: Sketch Control Image Synthesis with Diffusion Models

TL;DR

This paper introduces DiffSketching, a diffusion model-based approach for sketch-to-image synthesis that accurately translates abstract sketches into diverse, high-quality images without large datasets, outperforming GANs.

Contribution

The paper proposes a novel diffusion model framework for sketch-to-image synthesis that does not require large-scale datasets and improves accuracy and diversity over existing methods.

Findings

Outperforms GAN-based methods in quality and human evaluation

Maintains diversity and imagination in generated images

Effective in image editing and interpolation applications

Abstract

Creative sketch is a universal way of visual expression, but translating images from an abstract sketch is very challenging. Traditionally, creating a deep learning model for sketch-to-image synthesis needs to overcome the distorted input sketch without visual details, and requires to collect large-scale sketch-image datasets. We first study this task by using diffusion models. Our model matches sketches through the cross domain constraints, and uses a classifier to guide the image synthesis more accurately. Extensive experiments confirmed that our method can not only be faithful to user's input sketches, but also maintain the diversity and imagination of synthetic image results. Our model can beat GAN-based method in terms of generation quality and human evaluation, and does not rely on massive sketch-image datasets. Additionally, we present applications of our method in image editing and interpolation.