Instruction-based Image Editing with Planning, Reasoning, and Generation

TL;DR

This paper introduces a multi-modality model that enhances instruction-based image editing by integrating planning, reasoning, and generation, leading to improved handling of complex real-world images.

Contribution

It proposes a novel multi-modality chain of thought framework that combines reasoning and generation for more effective instruction-based image editing.

Findings

Achieves competitive editing performance on complex images.

Effectively integrates reasoning and generation in image editing.

Outperforms prior models in handling real-world editing tasks.

Abstract

Editing images via instruction provides a natural way to generate interactive content, but it is a big challenge due to the higher requirement of scene understanding and generation. Prior work utilizes a chain of large language models, object segmentation models, and editing models for this task. However, the understanding models provide only a single modality ability, restricting the editing quality. We aim to bridge understanding and generation via a new multi-modality model that provides the intelligent abilities to instruction-based image editing models for more complex cases. To achieve this goal, we individually separate the instruction editing task with the multi-modality chain of thought prompts, i.e., Chain-of-Thought (CoT) planning, editing region reasoning, and editing. For Chain-of-Thought planning, the large language model could reason the appropriate sub-prompts considering the instruction provided and the ability of the editing network. For editing region reasoning, we train an instruction-based editing region generation network with a multi-modal large language model. Finally, a hint-guided instruction-based editing network is proposed for editing image generations based on the sizeable text-to-image diffusion model to accept the hints for generation. Extensive experiments demonstrate that our method has competitive editing abilities on complex real-world images.