Complex Scene Image Editing by Scene Graph Comprehension

TL;DR

This paper introduces SGC-Net, a two-stage scene graph-based image editing method that improves object recognition and editing accuracy in complex scenes, outperforming prior approaches on benchmark datasets.

Contribution

The paper presents a novel two-stage approach combining scene graph comprehension with diffusion-based editing, enabling more accurate manipulation of complex scenes.

Findings

8-point SSIM improvement on CLEVR

9-33% human preference over prior work on Visual Genome

Effective recognition of target objects via scene graph understanding

Abstract

Conditional diffusion models have demonstrated impressive performance on various tasks like text-guided semantic image editing. Prior work requires image regions to be identified manually by human users or use an object detector that only perform well for object-centric manipulations. For example, if an input image contains multiple objects with the same semantic meaning (such as a group of birds), object detectors may struggle to recognize and localize the target object, let alone accurately manipulate it. To address these challenges, we propose a two-stage method for achieving complex scene image editing by Scene Graph Comprehension (SGC-Net). In the first stage, we train a Region of Interest (RoI) prediction network that uses scene graphs and predict the locations of the target objects. Unlike object detection methods based solely on object category, our method can accurately recognize the target object by comprehending the objects and their semantic relationships within a complex scene. The second stage uses a conditional diffusion model to edit the image based on our RoI predictions. We evaluate the effectiveness of our approach on the CLEVR and Visual Genome datasets. We report an 8 point improvement in SSIM on CLEVR and our edited images were preferred by human users by 9-33% over prior work on Visual Genome, validating the effectiveness of our proposed method. Code is available at github.com/Zhongping-Zhang/SGC_Net.