SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesis

TL;DR

SceneGenie introduces a novel scene graph-guided diffusion approach that enhances text-to-image synthesis by incorporating semantic and geometric constraints, achieving state-of-the-art results without additional training data.

Contribution

The paper presents a new guidance method for diffusion models using scene graph structured prompts and CLIP embeddings, improving accuracy in complex scene generation.

Findings

Achieves state-of-the-art performance on scene graph benchmarks.

Outperforms existing scene graph to image and text-based diffusion models.

Effectively incorporates bounding box and segmentation guidance during sampling.

Abstract

Text-conditioned image generation has made significant progress in recent years with generative adversarial networks and more recently, diffusion models. While diffusion models conditioned on text prompts have produced impressive and high-quality images, accurately representing complex text prompts such as the number of instances of a specific object remains challenging. To address this limitation, we propose a novel guidance approach for the sampling process in the diffusion model that leverages bounding box and segmentation map information at inference time without additional training data. Through a novel loss in the sampling process, our approach guides the model with semantic features from CLIP embeddings and enforces geometric constraints, leading to high-resolution images that accurately represent the scene. To obtain bounding box and segmentation map information, we structure the text prompt as a scene graph and enrich the nodes with CLIP embeddings. Our proposed model achieves state-of-the-art performance on two public benchmarks for image generation from scene graphs, surpassing both scene graph to image and text-based diffusion models in various metrics. Our results demonstrate the effectiveness of incorporating bounding box and segmentation map guidance in the diffusion model sampling process for more accurate text-to-image generation.