Sketch-Guided Scene Image Generation

TL;DR

This paper introduces a novel sketch-guided scene image generation framework that decomposes scene creation into object-level and scene-level processes, leveraging pre-trained diffusion models to produce detailed, conceptually faithful images from sketches.

Contribution

The proposed method uniquely combines object-level diffusion-based generation with scene-level composition using identity embeddings and layout prompts, advancing sketch-guided scene image synthesis.

Findings

Outperforms state-of-the-art methods in qualitative assessments

Achieves higher fidelity and diversity in generated images

Demonstrates robustness across various sketch inputs

Abstract

Text-to-image models are showcasing the impressive ability to create high-quality and diverse generative images. Nevertheless, the transition from freehand sketches to complex scene images remains challenging using diffusion models. In this study, we propose a novel sketch-guided scene image generation framework, decomposing the task of scene image scene generation from sketch inputs into object-level cross-domain generation and scene-level image construction. We employ pre-trained diffusion models to convert each single object drawing into an image of the object, inferring additional details while maintaining the sparse sketch structure. In order to maintain the conceptual fidelity of the foreground during scene generation, we invert the visual features of object images into identity embeddings for scene generation. In scene-level image construction, we generate the latent representation of the scene image using the separated background prompts, and then blend the generated foreground objects according to the layout of the sketch input. To ensure the foreground objects' details remain unchanged while naturally composing the scene image, we infer the scene image on the blended latent representation using a global prompt that includes the trained identity tokens. Through qualitative and quantitative experiments, we demonstrate the ability of the proposed approach to generate scene images from hand-drawn sketches surpasses the state-of-the-art approaches.