SGEdit: Bridging LLM with Text2Image Generative Model for Scene Graph-based Image Editing

TL;DR

This paper presents SGEdit, a novel framework that combines large language models with Text2Image generative models to enable precise, flexible, and scene-aware image editing using scene graphs.

Contribution

It introduces a new scene graph-based image editing approach that leverages LLMs for scene parsing and editing control, enhancing editing accuracy and scene coherence.

Findings

Outperforms existing methods in editing precision

Achieves higher scene aesthetic quality

Enables object-level modifications with fine-grained control

Abstract

Scene graphs offer a structured, hierarchical representation of images, with nodes and edges symbolizing objects and the relationships among them. It can serve as a natural interface for image editing, dramatically improving precision and flexibility. Leveraging this benefit, we introduce a new framework that integrates large language model (LLM) with Text2Image generative model for scene graph-based image editing. This integration enables precise modifications at the object level and creative recomposition of scenes without compromising overall image integrity. Our approach involves two primary stages: 1) Utilizing a LLM-driven scene parser, we construct an image's scene graph, capturing key objects and their interrelationships, as well as parsing fine-grained attributes such as object masks and descriptions. These annotations facilitate concept learning with a fine-tuned diffusion model, representing each object with an optimized token and detailed description prompt. 2) During the image editing phase, a LLM editing controller guides the edits towards specific areas. These edits are then implemented by an attention-modulated diffusion editor, utilizing the fine-tuned model to perform object additions, deletions, replacements, and adjustments. Through extensive experiments, we demonstrate that our framework significantly outperforms existing image editing methods in terms of editing precision and scene aesthetics.