MaskSketch: Unpaired Structure-guided Masked Image Generation

TL;DR

MaskSketch is a novel image generation method that enables spatial control over generated images using sketches as guidance, leveraging a pre-trained transformer without additional training.

Contribution

It introduces a structure-guided sampling technique utilizing self-attention maps in a pre-trained transformer, allowing unpaired, sketch-based image generation with high fidelity.

Findings

Outperforms state-of-the-art sketch-to-image translation methods

Achieves high realism and structural fidelity in generated images

Operates without additional training or paired supervision

Abstract

Recent conditional image generation methods produce images of remarkable diversity, fidelity and realism. However, the majority of these methods allow conditioning only on labels or text prompts, which limits their level of control over the generation result. In this paper, we introduce MaskSketch, an image generation method that allows spatial conditioning of the generation result using a guiding sketch as an extra conditioning signal during sampling. MaskSketch utilizes a pre-trained masked generative transformer, requiring no model training or paired supervision, and works with input sketches of different levels of abstraction. We show that intermediate self-attention maps of a masked generative transformer encode important structural information of the input image, such as scene layout and object shape, and we propose a novel sampling method based on this observation to enable structure-guided generation. Our results show that MaskSketch achieves high image realism and fidelity to the guiding structure. Evaluated on standard benchmark datasets, MaskSketch outperforms state-of-the-art methods for sketch-to-image translation, as well as unpaired image-to-image translation approaches.