Text-to-image Editing by Image Information Removal

TL;DR

This paper introduces a novel text-to-image editing method using an Image Information Removal module to selectively erase irrelevant details, improving content preservation and editability in diffusion-based image editing.

Contribution

It proposes a new IIR module that enhances text-guided image editing by removing unnecessary information, addressing overfitting and content leakage issues of existing methods.

Findings

Achieves the best editability-fidelity trade-off on multiple datasets.

User study shows 35% preference for our edited images over prior methods.

Effectively preserves non-text-related content during editing.

Abstract

Diffusion models have demonstrated impressive performance in text-guided image generation. Current methods that leverage the knowledge of these models for image editing either fine-tune them using the input image (e.g., Imagic) or incorporate structure information as additional constraints (e.g., ControlNet). However, fine-tuning large-scale diffusion models on a single image can lead to severe overfitting issues and lengthy inference time. Information leakage from pretrained models also make it challenging to preserve image content not related to the text input. Additionally, methods that incorporate structural guidance (e.g., edge maps, semantic maps, keypoints) find retaining attributes like colors and textures difficult. Using the input image as a control could mitigate these issues, but since these models are trained via reconstruction, a model can simply hide information about the original image when encoding it to perfectly reconstruct the image without learning the editing task. To address these challenges, we propose a text-to-image editing model with an Image Information Removal module (IIR) that selectively erases color-related and texture-related information from the original image, allowing us to better preserve the text-irrelevant content and avoid issues arising from information hiding. Our experiments on CUB, Outdoor Scenes, and COCO reports our approach achieves the best editability-fidelity trade-off results. In addition, a user study on COCO shows that our edited images are preferred 35% more often than prior work.