Sketch-Guided Scene Image Generation
Tianyu Zhang, Xiaoxuan Xie, Xusheng Du, Haoran Xie

TL;DR
This paper introduces a novel sketch-guided scene image generation framework that decomposes scene creation into object-level and scene-level processes, leveraging pre-trained diffusion models to produce detailed, conceptually faithful images from sketches.
Contribution
The proposed method uniquely combines object-level diffusion-based generation with scene-level composition using identity embeddings and layout prompts, advancing sketch-guided scene image synthesis.
Findings
Outperforms state-of-the-art methods in qualitative assessments
Achieves higher fidelity and diversity in generated images
Demonstrates robustness across various sketch inputs
Abstract
Text-to-image models are showcasing the impressive ability to create high-quality and diverse generative images. Nevertheless, the transition from freehand sketches to complex scene images remains challenging using diffusion models. In this study, we propose a novel sketch-guided scene image generation framework, decomposing the task of scene image scene generation from sketch inputs into object-level cross-domain generation and scene-level image construction. We employ pre-trained diffusion models to convert each single object drawing into an image of the object, inferring additional details while maintaining the sparse sketch structure. In order to maintain the conceptual fidelity of the foreground during scene generation, we invert the visual features of object images into identity embeddings for scene generation. In scene-level image construction, we generate the latent…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsComputer Graphics and Visualization Techniques · Generative Adversarial Networks and Image Synthesis · 3D Shape Modeling and Analysis
MethodsDiffusion
