What Makes a Scene ? Scene Graph-based Evaluation and Feedback for Controllable Generation

TL;DR

This paper introduces Scene-Bench, a new benchmark with a large dataset and a novel evaluation metric, SGScore, to improve factual accuracy in scene graph-based image generation, along with a feedback pipeline for iterative refinement.

Contribution

The paper presents Scene-Bench, a large-scale dataset, SGScore for better evaluation, and a feedback method to enhance controllable scene image generation.

Findings

Scene-Bench outperforms existing benchmarks in evaluating complex scenes.

SGScore provides more accurate assessment of object and relationship accuracy.

Feedback pipeline improves factual consistency of generated images.

Abstract

While text-to-image generation has been extensively studied, generating images from scene graphs remains relatively underexplored, primarily due to challenges in accurately modeling spatial relationships and object interactions. To fill this gap, we introduce Scene-Bench, a comprehensive benchmark designed to evaluate and enhance the factual consistency in generating natural scenes. Scene-Bench comprises MegaSG, a large-scale dataset of one million images annotated with scene graphs, facilitating the training and fair comparison of models across diverse and complex scenes. Additionally, we propose SGScore, a novel evaluation metric that leverages chain-of-thought reasoning capabilities of multimodal large language models (LLMs) to assess both object presence and relationship accuracy, offering a more effective measure of factual consistency than traditional metrics like FID and CLIPScore. Building upon this evaluation framework, we develop a scene graph feedback pipeline that iteratively refines generated images by identifying and correcting discrepancies between the scene graph and the image. Extensive experiments demonstrate that Scene-Bench provides a more comprehensive and effective evaluation framework compared to existing benchmarks, particularly for complex scene generation. Furthermore, our feedback strategy significantly enhances the factual consistency of image generation models, advancing the field of controllable image generation.